Simple is Better: Training an End-to-end Contract Bridge Bidding Agent without Human Knowledge

1 INTRODUCTION . Games have long been recognized as a testbed for reinforcement learning . Recent technology advancements have outperformed top level experts in perfect information games like Chess ( Campbell et al. , 2002 ) and Go ( Silver et al. , 2016 ; 2017 ) , through human supervision and selfplay . During recent years researchers have also steered towards imperfection information games , such as Poker ( Brown & Sandholm , 2018 ; Moravčík et al. , 2017 ) , Dota 2 1 , and real-time strategy games ( Arulkumaran et al. , 2019 ; Tian et al. , 2017 ) . There are multiple programs which focus specifically in card games . Libratus ( Brown & Sandholm , 2018 ) and DeepStack ( Moravčík et al. , 2017 ) outperforms human experts in two-player Texas Holdem . Bayesian Action Decoder ( Foerster et al. , 2018b ) is able to achieve near optimal performance in multi-player collaborative games like Hanabi . Contract Bridge , or simply Bridge , is a trick-taking card game with 2 teams , each with 2 players . There are 52 cards ( 4 suits , each with 13 cards ) . Each player is dealt with 13 cards . The game has two phases : bidding and playing . In the bidding phase , each player can only see their own card and negotiate in turns via proposing contract , which sets an explicit goal to aim at during the playing stage . High contracts override low ones . Players with stronger cards aim at high contracts for high reward ; while failing to reach the contract , the opponent team receives rewards . Therefore , players utilize the bidding phase to reason about their teammate and opponents ’ cards for a better final contract . In the playing phase , one player reveals their cards publicly . In each round , each player plays one card in turn and the player with best card wins the round . The score is simply how many rounds each team can win . We introduce the game in more detail in Appendix A . Historically AI programs can handle the playing phase well . Back in 1999 , the GIB program ( Ginsberg , 1999 ) placed 12th among 34 human experts partnership , in a competition without the bidding phase . In more recent years , Jack 2 and Wbridge5 3 , champions of computer bridge tournament , has demonstrated strong performances against top level professional humans . 1https : //openai.com/blog/openai-five/ 2http : //www.jackbridge.com/eindex.htm 3http : //www.wbridge5.com/ On the other hand , the bidding phase is very challenging for computer programs . During the bidding phase a player can only access his own 13 cards ( private information ) and the bidding history ( public information ) . They need to exchange information with their partners and try to interfere opponents from doing so through a sequences of non-decreasing bids . Moreover these bids also carry the meaning of suggesting a contract . If the bid surpasses the highest contract they can make , they will get negative score and risk of being doubled . Thus , the amount of information exchange is constrained and dependent on the actual hands . Nevertheless the state space is very large . A player can hold 6.35 × 1011 unique hands and there are 1047 possible bidding sequences . Human has designed a lot of hand-crafted rules and heuristics to cover these cases , called bidding system , and designated a meaning to many common bidding sequences . However , due to large state space , the meaning of these sequences are sometimes ambiguous or conflicting . The bidding system itself also has room for improvement . The award winning programs often implement a subset of some specified human bidding system . Recently , there are also attempts to learn such a bidding system automatically through reinforcement learning . These methods either focus on bidding in the collaborative only setting , where both opponents will bid PASS throughout ( Tian et al. , 2018 ; Yeh & Lin , 2016 ) , or heavily used human expert data for extra supervision ( Rong et al. , 2019 ) . In this work , we propose a system that is the state-of-the-art in competitive bridge bidding . It allows end-to-end training without any human knowledge through selfplay . We propose a novel bidding history representation , and remove any explicit modeling of belief in other agent ’ s state , which are shown to be critical in previous works ( Rong et al. , 2019 ; Tian et al. , 2018 ) . We show that selfplay schedule and details are critical in learning imperfect information games . We use a much smaller model ( about 1/70 in total parameters compared with previous state-of-the-art ( Rong et al. , 2019 ) ) , and reach better performance than the baselines ( Rong et al. , 2019 ; Yeh & Lin , 2016 ) . Furthermore , we outperform world computer bridge championship Wbridge5 by 0.41 IMPs per board over a tournament of 64 boards . Finally , we show an interpretation of the trained system , and will open source the code , model , and experimental data we use . 2 RELATED WORK . Imperfect information games , especially card games , have drawn multiple researchers ’ attention . Prior works on two-player Texas Holdem mainly focus on finding the Nash Equilibrium through variations of counterfactual regret minimization ( Zinkevich et al. , 2008 ) . Libratus ( Brown & Sandholm , 2018 ) utilizes nested safe subgame solving and handles off-tree actions by real time computing . It also has a built-in self improver to enhance the background blueprint strategy . DeepStack ( Moravčík et al. , 2017 ) proposed to use a value network to approximate the value function of the state . They both outperform top human experts in the field . Bayesian Action Decoder ( BAD ) ( Foerster et al. , 2018b ) proposes to model public belief and private belief separately , and sample policy based on an evolving deterministic communication protocol . This protocol is then improved through Bayesian updates . BAD is able to reach near optimal results in two-player Hanabi , outperforming previous methods by a significant margin . In recent years there are also multiple works specifically focusing on contract bridge . Yeh and Lin ( Yeh & Lin , 2016 ) uses deep reinforcement learning to train a bidding model in the collaborative setting . It proposes Penetrative Bellman ’ s Equation ( PBE ) to make the Q-function updates more efficient . The limitation is that PBE can only handle fixed number of bids , which are not realistic in a normal bridge game setting . We refer to this approach as baseline16 . Tian et al ( Tian et al. , 2018 ) proposes Policy Belief Learning ( PBL ) to alternate training between policy learning and belief learning over the whole selfplay process . PBL also only works on the collaborative setting . Rong et al ( Rong et al. , 2019 ) proposes two networks , Estimation Neural Network ( ENN ) and Policy Neural Network ( PNN ) to train a competitive bridge model . ENN is first trained supervisedly from human expert data , and PNN is then learned based on ENN . After learning PNN and ENN from human expert data , the two network are further trained jointly through reinforcement learning and selfplay . PBE claims to be better than Wbridge5 in the collaborative setting , while PNN and ENN outperforms Wbridge5 in the competitive setting . We refer to this approach as baseline19 . Selfplay methods have been proposed for a long time . Back in 1951 , Brown et al ( Brown , 1951 ) proposes fictitious play in imperfect information games to find the Nash Equilibrium . This is a classic selfplay algorithm in game theory and inspires many extensions and applications ( Brown & Sandholm , 2018 ; Heinrich et al. , 2015 ; Heinrich & Silver , 2016 ; Moravčík et al. , 2017 ) . Large scale selfplay algorithms do not emerge until recent years , partially due to computation constraint . AlphaGo ( Silver et al. , 2016 ) uses selfplay to train a value network to defeat the human Go champion Lee Sedol 4:1 . AlphaGoZero ( Silver et al. , 2017 ) and AlphaZero ( Silver et al. , 2018 ) completely discard human knowledge and train superhuman models from scratch . In Dota 2 and StarCraft , selfplay is also used extensively to train models to outperform professional players . Belief modeling is also very critical in previous works about imperfect information games . Besides the previous mentioned card game agents ( Foerster et al. , 2018b ; Rong et al. , 2019 ; Tian et al. , 2018 ) , LOLA agents ( Foerster et al. , 2018a ) are trained with anticipated learning of other agents . StarCraft Defogger ( Synnaeve et al. , 2018 ) also tries to reason about states of unknown territory in real time strategy games . 3 METHOD . 3.1 PROBLEM SETUP . We focus on the bidding part of the bridge game . Double Dummy Solver ( DDS ) 4 computes the maximum tricks each side can get during the playing phase if all the plays are optimal . Previous works show that DDS is a good approximate to human expert real plays ( Rong et al. , 2019 ) , so we directly use the results of DDS at the end of bidding phase to assign reward to each side . The training dataset contains randomly generated 2.5 million hands along with their precomputed DDS results . The evaluation dataset contains 100k such hands . We will open source this data for the community and future work . Inspired by the format of duplicate bridge tournament , during training and evaluation , each hand is played twice , where a specific partnership sits North-South in one game , and East-West in another . The difference in the results of the two tables is the final reward . In this way , the impact of randomness in the hands is reduced to minimum and model ’ s true strength can be better evaluated . The difference in scores is then converted to IMPs scale , and then normalized to [ -1 , 1 ] . 3.2 INPUT REPRESENTATION . We encode the state of a bridge game to a 267 bit vector . The first 52 bits indicate that if the current player holds a specific card . The next 175 bits encodes the bidding history , which consists of 5 segments of 35 bits each . These 35 bit segments correspond to 35 contract bids . The first segment indicates if the current player has made a corresponding bid in the bidding history . Similarly , the next 3 segments encodes the contract bid history of the current player ’ s partner , left opponent and right opponent . The last segment indicates that if a corresponding contract bid has been doubled or redoubled . Since the bidding sequence can only be non-decreasing , the order of these bids are implicitly conveyed . The next 2 bits encode the current vulnerability of the game , corresponding to the vulnerability of North-South and East-West respectively . Finally , the last 38 bits indicates whether an action is legal , given the current bidding history . We emphasize that this encoding is quite general and there is not much domain-specific information . baseline19 presents a novel bidding history representation using positions in the maximal possible bidding sequence , which is highly specific to the contract bridge game . 4https : //github.com/dds-bridge/dds

Summary: This paper develops a method to train agents to bid competitively in the game of Bridge. The authors focus on the bidding phase of the game and develop a model to predict the best bid to make at each turn of the phase. The difficulty in the bidding lies in understanding the signals provided by your own teammate as well as the opponent team in order to estimate the state of the game, which is partially observable since each player cannot see the hands of the three other players. The authors show that explicitly modeling the belief of other agents is not necessary and that competitive performance can be achieved with self-play.