Simple is Better: Training an End-to-end Contract Bridge Bidding Agent without Human Knowledge

1 INTRODUCTION . Games have long been recognized as a testbed for reinforcement learning . Recent technology advancements have outperformed top level experts in perfect information games like Chess ( Campbell et al. , 2002 ) and Go ( Silver et al. , 2016 ; 2017 ) , through human supervision and selfplay . During recent years researchers have also steered towards imperfection information games , such as Poker ( Brown & Sandholm , 2018 ; Moravčík et al. , 2017 ) , Dota 2 1 , and real-time strategy games ( Arulkumaran et al. , 2019 ; Tian et al. , 2017 ) . There are multiple programs which focus specifically in card games . Libratus ( Brown & Sandholm , 2018 ) and DeepStack ( Moravčík et al. , 2017 ) outperforms human experts in two-player Texas Holdem . Bayesian Action Decoder ( Foerster et al. , 2018b ) is able to achieve near optimal performance in multi-player collaborative games like Hanabi . Contract Bridge , or simply Bridge , is a trick-taking card game with 2 teams , each with 2 players . There are 52 cards ( 4 suits , each with 13 cards ) . Each player is dealt with 13 cards . The game has two phases : bidding and playing . In the bidding phase , each player can only see their own card and negotiate in turns via proposing contract , which sets an explicit goal to aim at during the playing stage . High contracts override low ones . Players with stronger cards aim at high contracts for high reward ; while failing to reach the contract , the opponent team receives rewards . Therefore , players utilize the bidding phase to reason about their teammate and opponents ’ cards for a better final contract . In the playing phase , one player reveals their cards publicly . In each round , each player plays one card in turn and the player with best card wins the round . The score is simply how many rounds each team can win . We introduce the game in more detail in Appendix A . Historically AI programs can handle the playing phase well . Back in 1999 , the GIB program ( Ginsberg , 1999 ) placed 12th among 34 human experts partnership , in a competition without the bidding phase . In more recent years , Jack 2 and Wbridge5 3 , champions of computer bridge tournament , has demonstrated strong performances against top level professional humans . 1https : //openai.com/blog/openai-five/ 2http : //www.jackbridge.com/eindex.htm 3http : //www.wbridge5.com/ On the other hand , the bidding phase is very challenging for computer programs . During the bidding phase a player can only access his own 13 cards ( private information ) and the bidding history ( public information ) . They need to exchange information with their partners and try to interfere opponents from doing so through a sequences of non-decreasing bids . Moreover these bids also carry the meaning of suggesting a contract . If the bid surpasses the highest contract they can make , they will get negative score and risk of being doubled . Thus , the amount of information exchange is constrained and dependent on the actual hands . Nevertheless the state space is very large . A player can hold 6.35 × 1011 unique hands and there are 1047 possible bidding sequences . Human has designed a lot of hand-crafted rules and heuristics to cover these cases , called bidding system , and designated a meaning to many common bidding sequences . However , due to large state space , the meaning of these sequences are sometimes ambiguous or conflicting . The bidding system itself also has room for improvement . The award winning programs often implement a subset of some specified human bidding system . Recently , there are also attempts to learn such a bidding system automatically through reinforcement learning . These methods either focus on bidding in the collaborative only setting , where both opponents will bid PASS throughout ( Tian et al. , 2018 ; Yeh & Lin , 2016 ) , or heavily used human expert data for extra supervision ( Rong et al. , 2019 ) . In this work , we propose a system that is the state-of-the-art in competitive bridge bidding . It allows end-to-end training without any human knowledge through selfplay . We propose a novel bidding history representation , and remove any explicit modeling of belief in other agent ’ s state , which are shown to be critical in previous works ( Rong et al. , 2019 ; Tian et al. , 2018 ) . We show that selfplay schedule and details are critical in learning imperfect information games . We use a much smaller model ( about 1/70 in total parameters compared with previous state-of-the-art ( Rong et al. , 2019 ) ) , and reach better performance than the baselines ( Rong et al. , 2019 ; Yeh & Lin , 2016 ) . Furthermore , we outperform world computer bridge championship Wbridge5 by 0.41 IMPs per board over a tournament of 64 boards . Finally , we show an interpretation of the trained system , and will open source the code , model , and experimental data we use . 2 RELATED WORK . Imperfect information games , especially card games , have drawn multiple researchers ’ attention . Prior works on two-player Texas Holdem mainly focus on finding the Nash Equilibrium through variations of counterfactual regret minimization ( Zinkevich et al. , 2008 ) . Libratus ( Brown & Sandholm , 2018 ) utilizes nested safe subgame solving and handles off-tree actions by real time computing . It also has a built-in self improver to enhance the background blueprint strategy . DeepStack ( Moravčík et al. , 2017 ) proposed to use a value network to approximate the value function of the state . They both outperform top human experts in the field . Bayesian Action Decoder ( BAD ) ( Foerster et al. , 2018b ) proposes to model public belief and private belief separately , and sample policy based on an evolving deterministic communication protocol . This protocol is then improved through Bayesian updates . BAD is able to reach near optimal results in two-player Hanabi , outperforming previous methods by a significant margin . In recent years there are also multiple works specifically focusing on contract bridge . Yeh and Lin ( Yeh & Lin , 2016 ) uses deep reinforcement learning to train a bidding model in the collaborative setting . It proposes Penetrative Bellman ’ s Equation ( PBE ) to make the Q-function updates more efficient . The limitation is that PBE can only handle fixed number of bids , which are not realistic in a normal bridge game setting . We refer to this approach as baseline16 . Tian et al ( Tian et al. , 2018 ) proposes Policy Belief Learning ( PBL ) to alternate training between policy learning and belief learning over the whole selfplay process . PBL also only works on the collaborative setting . Rong et al ( Rong et al. , 2019 ) proposes two networks , Estimation Neural Network ( ENN ) and Policy Neural Network ( PNN ) to train a competitive bridge model . ENN is first trained supervisedly from human expert data , and PNN is then learned based on ENN . After learning PNN and ENN from human expert data , the two network are further trained jointly through reinforcement learning and selfplay . PBE claims to be better than Wbridge5 in the collaborative setting , while PNN and ENN outperforms Wbridge5 in the competitive setting . We refer to this approach as baseline19 . Selfplay methods have been proposed for a long time . Back in 1951 , Brown et al ( Brown , 1951 ) proposes fictitious play in imperfect information games to find the Nash Equilibrium . This is a classic selfplay algorithm in game theory and inspires many extensions and applications ( Brown & Sandholm , 2018 ; Heinrich et al. , 2015 ; Heinrich & Silver , 2016 ; Moravčík et al. , 2017 ) . Large scale selfplay algorithms do not emerge until recent years , partially due to computation constraint . AlphaGo ( Silver et al. , 2016 ) uses selfplay to train a value network to defeat the human Go champion Lee Sedol 4:1 . AlphaGoZero ( Silver et al. , 2017 ) and AlphaZero ( Silver et al. , 2018 ) completely discard human knowledge and train superhuman models from scratch . In Dota 2 and StarCraft , selfplay is also used extensively to train models to outperform professional players . Belief modeling is also very critical in previous works about imperfect information games . Besides the previous mentioned card game agents ( Foerster et al. , 2018b ; Rong et al. , 2019 ; Tian et al. , 2018 ) , LOLA agents ( Foerster et al. , 2018a ) are trained with anticipated learning of other agents . StarCraft Defogger ( Synnaeve et al. , 2018 ) also tries to reason about states of unknown territory in real time strategy games . 3 METHOD . 3.1 PROBLEM SETUP . We focus on the bidding part of the bridge game . Double Dummy Solver ( DDS ) 4 computes the maximum tricks each side can get during the playing phase if all the plays are optimal . Previous works show that DDS is a good approximate to human expert real plays ( Rong et al. , 2019 ) , so we directly use the results of DDS at the end of bidding phase to assign reward to each side . The training dataset contains randomly generated 2.5 million hands along with their precomputed DDS results . The evaluation dataset contains 100k such hands . We will open source this data for the community and future work . Inspired by the format of duplicate bridge tournament , during training and evaluation , each hand is played twice , where a specific partnership sits North-South in one game , and East-West in another . The difference in the results of the two tables is the final reward . In this way , the impact of randomness in the hands is reduced to minimum and model ’ s true strength can be better evaluated . The difference in scores is then converted to IMPs scale , and then normalized to [ -1 , 1 ] . 3.2 INPUT REPRESENTATION . We encode the state of a bridge game to a 267 bit vector . The first 52 bits indicate that if the current player holds a specific card . The next 175 bits encodes the bidding history , which consists of 5 segments of 35 bits each . These 35 bit segments correspond to 35 contract bids . The first segment indicates if the current player has made a corresponding bid in the bidding history . Similarly , the next 3 segments encodes the contract bid history of the current player ’ s partner , left opponent and right opponent . The last segment indicates that if a corresponding contract bid has been doubled or redoubled . Since the bidding sequence can only be non-decreasing , the order of these bids are implicitly conveyed . The next 2 bits encode the current vulnerability of the game , corresponding to the vulnerability of North-South and East-West respectively . Finally , the last 38 bits indicates whether an action is legal , given the current bidding history . We emphasize that this encoding is quite general and there is not much domain-specific information . baseline19 presents a novel bidding history representation using positions in the maximal possible bidding sequence , which is highly specific to the contract bridge game . 4https : //github.com/dds-bridge/dds

Summary: The authors propose a deep learning agent for automatic bidding in the bridge game. The agent is trained with a standard A3C reinforcement learning model with self-play, and the internal neural network only takes a rather succinct representation of the bidding history as the input. Experiment results demonstrate state-of-the-art performance with a simpler model. The authors discuss some findings with the proposed agent, such as the lack of need to explicitly model the belief and the possibility to self-train with different variants of opponents. Some visualization is also provided to understand how the trained agent behaves.