Simple is Better: Training an End-to-end Contract Bridge Bidding Agent without Human Knowledge
Authors: Qucheng Gong
Yu Jiang
Yuandong Tian
Venue:  (2020)
Keywords: Contract Bridge
Bidding
Selfplay
AlphaZero

Abstract: Contract bridge is a multi-player imperfect-information game where one partnership collaborate with each other to compete against the other partnership. The game consists of two phases: bidding and playing. While playing is relatively easy for modern software, bidding is challenging and requires agents to learn a communication protocol to reach the optimal contract jointly, with their own private information. The agents need to exchange information to their partners, and interfere opponents, through a sequence of actions. In this work, we train a strong agent to bid competitive bridge purely through selfplay, outperforming WBridge5, a championship-winning software. Furthermore, we show that explicitly modeling belief is not necessary in boosting the performance. To our knowledge, this is the first competitive bridge agent that is trained with no domain knowledge. It outperforms previous state-of-the-art that use human replays with 70x fewer number of parameters.

Under review as a conference paper at ICLR 2020
SIMPLE IS BETTER: TRAINING AN END-TO-END CON- TRACT BRIDGE BIDDING AGENT WITHOUT HUMAN KNOWLEDGE
Anonymous authors Paper under double-blind review
ABSTRACT
Contract bridge is a multi-player imperfect-informatio game where one partner- ship collaborate with each other to compete against the other partnership. The game consists of two phases: bidding and playing. While playing is relatively easy for modern software, bidding is challenging and requires agents to learn a communication protocol to reach the optimal contract jointly, with their own private information. The agents need exchange information to their partners, and interfere opponents, through a sequence of actions. In this work, we train a strong agent to bid competitive bridge purely through selfplay, outperforming WBridge5, a thampionship-winae winning software. Furthermore, we show that explicitly modeling belief is not necessary boosting the performance. To our knowledge, this is the first competitive bridge agent that is trained with no domain knowledge. It outper- forms previous state-of-the-ee that use human replays with 70x fewer number of parameters.
INTRODUCTION
Games have long been recognized as a testbed for reinforcement learning. Recent technology advancements have outperformed top level experts in perfect information games like Chess (Campbell al. 2002) and Go (Silver et (2116;2017) through human supervision and selfplay. During recent years researchers have also steered towards imperfection information games, such as Poker (Brown & Sandholm, 2018 Moravčík et al. 2017). Dota _ and real-time strategy games (Arulkumaran [019 lian al. 2017). There multiple programs which focus specifically card games. Libratus (Brown & Sandholm 2018) and DeepStack (Moravčík et al. 2017) outperforms human experts in two-player Texas Holdem. Bayesian Action Decoder (Foerster et al. 2018b) able to achieve near optimal performance in multi-player collaborative games like Hanabi.
al.
Contract Bridge, or simply Bridge, trick-taking card game with 2 teams, each with 2 players. There are 52 cards (4 suits, each with 13 cards). Each player is dealt with 13 cards. The game has two phases: bidding and playing. the bidding phase, each player can only see their own card and negotiate in turns via proposing contract, which sets an explicit goal to aim at during the playing stage. High contracts override low ones. Players with stronger cards aim at high contracts for high reward; while failing to reach the contract, the opponent team receives rewards. Therefore, players utilize the bidding phase to reason about their teammate and opponents' cards for a better final contract. In the playing phase, turn and the player with best card wins the round. The score is simply how many rounds each team win. We introduce the game in more detail in Appendix A.
player reveals their cards publicly. In each round, each player plays card in
Historically Al programs can handle the playing phase well. Back in 1999, the GIB program @Gis
berg 1999) placed 12th among 34 human experts partnership. in competition without the bidding phase. In more recent years, Jack and Wbridge5 5 champions of computer bridge tournament, has demonstrated strong performances against top level professional humans.
http://wwww..cckrrideeee... ndeex htm http:/www.wbridge.t

---

Under review as a conference paper at ICLR 2020
On the other hand, the bidding phase is very challenging for computer programs. During the bidding phase a player can only access his own 13 cards (private information) and the bidding history (public information). They need to exchange information with their partners and try to interfere opponents from doing so through a sequences of non-decreasing bids. Moreover these bids also carry the meaning of suggesting contract. If the bid surpasses the highest contract they make, they will get negative score and risk of being doubled. Thus, the amount of information exchange is constrained and dependent on the actual hands. Nevertheless the state space very large. player can hold 6.35 x 10"l unique hands and there are 10*7 possible bidding sequences. Human has designed lot of hand-crafted rules and heuristics cover these cases, called bidding system, and designated a meaning to many common bidding sequences. However, due to large state space, the meaning of these sequences are sometimes ambiguous or conflicting. The bidding system itself also has room for improvement. The award winning programs often implement a subset of some specified human bidding system. Recently, there are also attempts to learn such a bidding system automatically through reinforcement learning. These methods either focus on bidding in the collaborative only setting, where both opponents will bid PASS throughout (Tian et al., 2018: Yeh & Lin 2016). or heavily used human expert data for extra supervision (Rong al. 2019).
In this work, we propose a system that is the state-of of-te-of-the-art in competitive bridge bidding. It allows end-to-end training without any human knowledge through selfplay. We propose novel bidding history representation, and remove any explicit modeling of belief in other agent's state, which are shown to be critical in previous works (Rong et al. 2019 Tian et al.) 2018). We show that selfplay schedule and details are critical in learning imperfect information games. We use a much smaller model (about 1/70 in total parameters compared with previous state-of-the-art (Rong al. 2019))
and reach better performance than the baselines (Rong et al. 2019 Yeh & Lin 2016). Furthermore, we outperform world computer bridge championship Wbridge5 by 0.41 IMPs per board over a tournament of 64 boards. Finally, we show an interpretation of the trained system, and will open source the code, model, and experimental data we use.
RELATED WORK
Imperfect information games, especially card games, have drawn multiple researchers' attention. Prior works on two-player Texas Holdem mainly focus on finding the Nash Equilibrium through variations of counterfactual regret minimization (Zinkevich et al.] 22088. Libratus (Brown & Sandholm (018) utilizes nested safe subgame solving and handles off tre- actions by real time computing. It also has a built-in self improver to enhance the background blueprint strategy. DeepStack (Moravčík et al.. 2017) proposed to use value network to approximate the value function of the state. They both outperform top human experts in the field. Bayesian Action Decoder (BAD) (Foerster et al., |20 1b)) proposes to model public belief and private belief separately, and sample policy based on an evolving deterministic communication protocol. This protocol is then improved through Bayesian updates. BAD is able to reach near optimal results in two-player Hanabi, outperforming previous methods by a significant margin.
In recent years there are also multiple works specifically focusing contract bridge. Yeh and Lin (Yeh & Lin 2016) uses deep reinforcement learning to train a bidding model in the collaborative setting. proposes Penetrative Bellman's Equation (PBE) to make the Q-function updates more efficient. The limitation is that PBE can only handle fixed number of bids, which are not realistic in normal bridge game setting. We refer to this approach as basel eile16.... 6. Tian et al (Tian et al. 2018) proposes Policy Belief Learning (PBL) to alternate training between policy learning and belief learning over the whole selfplay process. PBL also only works on the collaborative setting. Rong et al (Rong et al., 2019) proposes two networks, Estimation Neural Network (ENN) and Policy Neural Network (PNN) to train a competitive bridge model. ENN is first trained supervisedly from human expert data, and PNN is then learned based ENN. After learning PNN and ENN from human expert data, the two network are further trained jointly through reinforcement learning and selfplay. PBE claims to be better than Wbridge5 in the collaborative setting, while PNN and ENN outperforms Wbridge5 in the competitive setting. We refer to this approach as baselinell
Selfplay methods have been proposed for a long time. Back in 1951, Brown et al (Brown 1951 proposes fictitious play in imperfect information games to find the Nash Equilibrium. This is a classic selfplay algorithm in game theory and inspires many extensions and applications (Brown
2

---

Under review as a conference paper at ICLR 2020
& Sandholm 2018 Heinrich al. 2015 Heinrich & Silver 2016 Moravčík et al. 2017), Large
scale selfplay algorithms do not emerge until recent years, partially due to computation constraint. AlphaGo (Silver et al. 2016) uses selfplay to train value network to defeat the human Go champion
Lee Sedol 4:1. AlphaCioZero (Silver et al. 2017) and AlphaZero (Silver et al., 2018) completely discard human knowledge and train superhuman models from scratch. In Dota 2 and StarCraft, selfplay is also used extensively to train models to outperform professional players.
Belief modeling is also very critical previous works about imperfect information games. Besides the previous mentioned card game agents (Foerster et al. 2018b Rong et al. 2019) TTan et al. (2018), LOLA agents (Foerster al., 2018a) are trained with anticipated learning of other agents. StarCraft Defogger (Synnaeve et al. 2018) also tries to reason about states of unknown territory in real time strategy games.
METHOD
3.1 PROBLEM SETUP
We focus on the bidding part of the bridge game. Double Dummy Solver (DDS) computes the maximum tricks each side can get during the playing phase if all the plays are optimal. Previous works show that DDS is good approximate to human expert real plays (Rong et al. 2019). so we directly use the results of DDS at the end of bidding phase to assign reward each side. The training dataset contains randomly generated 2.5 million hands along with their precomputed DDS results. The evaluation dataset contains 100k such hands. We will open source this data for the community and future work.
Inspired by the format of duplicate bridge tournament, during training and evaluation, each hand is played twice, where a specific partnership sits North-South in one game, and Fast- West another. The difference the results of the two tables is the final reward. In this way, the impact of randomness in the hands is reduced to minimum and model's true strength can be better evaluated. The difference scores then converted IMPs scale, and then normalized [-1, 1].
3.2 INPUT REPRESENTATION
We encode the state of bridge game to 267 bit vector. The first 52 bits indicate that if the current player holds a specific card. The next 175 bits encodes the bidding history, which consists segments of 35 bits each. These 35 bit segments correspond to 35 contract bids. The first segment indicates if the current player has made a corresponding bid the bidding history. Similarly, the next 3 segments encodes the contract bid history of the current player's 's partner, left opponent and right opponent. The last segment indicates that if a corresponding contract bid has been doubled or redoubled. Since the bidding sequence can only be non-decreasing, the order of these bids implicitly conveyed. The next 2 bits encode the current vulnerability of the game, corresponding and East- West respectively. Finally, the last 38 bits indicates
the vulnerability of whether an action is legal, given the current bidding history.
We emphasize that this encoding is quite general and there is not much comain-speciffe information. basel inne 9 presents a novel bidding history representation using positions in the maximal possible bidding sequence, which is highly specific to the contract bridge game.
Own Cards
history, multi-hot contract bids
Vul ner- ability
Available Action
14
Own
Partner
Left Opp
Right Opo
Double Indicator
1e 24
52
35
35
35
35
35
Figure 1: Input representation. With the decision point shown in the example, South will mark the following bits the bidding history encoding: 1 in "Own" segment, 1 and 24 in "Partner" segment, 1 in "Left Opp" segment, and & in "Double Indicator" segment.
https://gi com.:/d.coom///a/ cww/dds dds
3

---

Under review as a conference paper at ICLR 2020
Residual FC Block (RFC)
Partner Hand
BCELoss
Input
RFC
RFC
Policy
RFC
RFC
Input
Value
Figure 2:
Network Architecture. Supervision from partner's hand is unused in the main results, and used the ablation studies. BCELoss stands for Binary Cross Entropy Loss.
3.3 NETWORK
We use similar network structure that is used in basel inel 9, As show in Figure the network consists of an initial fully connected layer, then fully connected layer with skip connections added every 2 layers get a latent representation. We use 200 neurons at each hidden layer, so it is much smaller (about 1/70 parameter size compared with basel inel 9). The full network architecture is shown In Figure However, during training do not partner's 's information to further supervise the belief training. We investigate the impact training belief separately, and find that our model cannot benefit from extra supervision from partner's 's information.
From the latent representation, one branch is to a policy head. is a fully connected layer to 38 output neurons, masking out illegal actions provided in the input, and then normalizes log policy. The other branch is a value head, which just fully connected layer to neuron.
3.4 TRAINING DETAILS
RL Method and Platform Implementation. We use selfplay on random data train our models. The model is trained with A3C (Mnih et al. (016) using improved ELF framework (Tian et al. (017). ELF supports off- off-policy training with importance factor correction, and has inherent parallelization implementations to make training fast. We implement contract bridge game logic and feature extraction logic C++. Each game thread has 2 agent groups, namely training agent and opponent agent. Each agent group has corresponding batcher. Once agent needs action, the current state and feature sent to the batcher in ELF. ELF uses pybind to communicate between C++ and python. If batcher gathers enough data from different game threads for specific actor group, the batch is forwarded to python for a pytorch model to evaluate. The results are then returned to the C++ game thread through pybind. ELF supports zero-copy during this process. During the selfplay training, the model of training agent actor group keeps updating, while the model of opponent agent actor group updates only when opponent update frequency condition is met. We implement an additional interface to track the full game trajectory. Once the game finishes, the interface receives terminal flag with reward signal. fills all the history state action pairs with the correct discounted rewards. This whole episode is then sent to the ELF train batcher to perform the actor critic update. The whole training process takes roughly 4-5 hours to converge on single GPU.
During training we run 500 games in parallel and use batch size of 100. We use an entropy ratio 0.01 in A3C training. The discount factor set to to encourage longer sequences of information exchange, and since the bidding is non-decreasing, it will not cause convergence issue. We train the model using R.MSProp with learning rate of 1e-3. We fine tune our model by dropping learning rate to le-4 at 50k training minibatches and further dropping it to 1e-5 at 70k minibatches. During training we use multinominal exploration to get the action from a policy distribution, and during evaluation pick the greedy action from the model.
Training Parameters.
4

---

Under review as a conference paper at ICLR 2020
Table 1: Performance Comparison. The left table compares performance when giving different weights to the belief loss and the performance when using the same history encoding as (Rong et al. 2019). The right table shows performance under different level of diversity of opponent models, by updating the opponent model at different frequency or sample opponent model randomly or using Nash Equilibrium.
B
A
Update frequency
imps + std
Ratio
imps std
1
2.26 + 0.10
2.31 0.15
50
2.14 + 0.20
0.01
1.90 + 0.29
100
2.08 + 0.07
0.1
1.63 0.27
200
2.31 + 0.15
1
1.22 + 0.22
Opponent Diversity
Hist encoding
Randomly sample
2.09 + 0.04
basel inee hist
1.27 + 0.22
Nash averaging
2.18 + 0.20
2
2
200
400
400
600
400
600
400
600
epoch
epoch
epoch
epoch
Figure 3: Training curves for different update frequency, From left to right, the opponent model updated every 1, 50, 100, 200 minibatches. Epoch is defined 200 minibatches.
Baselines. As suggested by the authors of baselines 6, we modify their pretrained model to bid competitively, by bidding PASS if the cost of all bids are greater than 0.2. We implement this and further fix its weakness that the model sometimes behaves randomly in competitive setting if the scenario can never occur in collaborative setting. We benchmark against them at each episode. We could not fully reproduce the results in base linel
cannot directly compare against them.
EXPERIMENTS
4.1
MAIN RESULTS
We train a competitive bridge bidding model through selfplay. We perform grid search on hyper- parameters such as discount factor, exploring rate, learning schedules and find the best combination. The training curve against baselinell is shown in Figure As can be seen, we significantly beat baselinell 6 2.31 IMPs per board. We manually run 64 board tournament against Wbridge5, and outperforms it by 0.41 IMPs per board. The standard error over these 64 boards are 0.27 IMPs per board, which translate 93.6% win probability in standard match. This also surpasses the previous state-of-the toef-theann art basel ine 1 9, which outperforms Wbridge5 by 0.25 IMPs per board. is shown in previous work that a margin of 0.1 IMPs per board significant (Rong al. 2019).
We outperform basel. nel6 6 with a large margin partially due to basel inel cannot adapt well to competitive bidding setting. It can also only handle fixed length of bids. We outperform basel iine9 mainly due to a better history encoding and not to model belief explicitly. These results are shown in the ablation studies.
4.2 ABLATION STUDIES
Prior works focus on explicitly modeling belief, either by adding an auxiliary loss to train jointly (Rong et al. 2019), alternating stages between training policy and belief (Tian et al.) 2018). However, training belief using supervision from partner's 's hand does not help in our model. We set the final loss T. TTeetief LAACC- where is hyper-parameted" to control the weight the auxiliary
5

---

Under review as a conference paper at ICLR 2020
Bidding length distribution
heatmap
Final contract
0.00
1NT
INT
0.775
2NT
0150
NNT
0.25
0.00
ANT
ANT
0075
SNT
SNT
GNT
GNT
0025
7NT
7NT
10
15
Seiddnn length
(a)
(b)
(c)
Figure 4: Statistical visualization. (a) Bidding length histogram. (b) Heatmap for all actions during the bidding. (c) Heatmap for final contracts reached.
lask, As shown in Table I when r = 0, the model reaches the best performance and the performance decreases as r' increase. This demonstrates that focusing on the main task can achieve better results.
Bidding history encoding plays critical role in model architecture. basel lieell 9 proposed a novel representation of sequenced bidding history, which listed all possible actions in sequence and then labeled what has been used. We compared our representation to theirs. As shown in Table [ our encoding can reach a better performance. The potential reason why our encoding performs better that the intrinsic order of bridge bidding is already kept by the action itself, so there is no need to specify the sequence, and our encoding captures the owner of each action.
In imperfect information games, one common strategy is to the experiences. We also investigate this strategy in bridge. To increase the diversity, we set two ways: First, we maintain model-zoo with 20 most recent models and then randomly sample the opponent model from this zoo; Second, we save the 20 models with best performance and sample using the Nash Averaging strategy (Balduzzi al. 2018). We find self-play with opponent using the most recent model works best in terms of performance comparing to baseline models as shown in Table [ One possible explanation is that bridge a game with both competition and collaborations. Mixed strategy can mislead both opponents and partners, so a Nash Averaging strategy will not work well enough. Hence, using the most recent model is more suitable for such training.
pool of opponents to add diversity
Besides the strategy to choose opponent model, we also study the impact of opponent model update frequency. As can be seen from Table [ the final performances are similar. However, the training curve Figure [hows different patterns. Using the exact the same model for selfplay opponent during the training shows the most stable results, especially at the early stage of the training. It is possibly due to the fast model progression during the early stage of the training. If selfplay opponent does not update frequent enough cannot learn new knowledge.
INTERPRETATION
5.1 VISUALIZATION
It is interesting to visualize what the model has learned, and understand some rational behind the learned conventions. In Figure we show the bidding length distribution and frequency of each bid used, as well as the distribution of final contracts. The results are averaged from our best 3 models. We can see that typically agents exchanges 6-10 rounds of information to reach the final contract. The agent uses low level bids more frequently and puts an emphasis on V and contracts. The final contract is mostly part scores and game contracts, particularly often 3NT, 40, very few slam contracts. This is because part scores and game contracts are optimal based on DDS for 87% of hand 5 The model does not optimize to bid slam contracts, because it needs to hold a firm belief after longer rounds of information exchange to bid slam contract, and the risk of not making the contract is very high.
and we observe
https://1ajall http:////aa..a.oooooo..ooo/oa/// 1ameS5 amGrand. htm

---

Under review as a conference paper at ICLR 2020
1NT
INT
2NT
2NT
3NT
3
3NT
3NT
ANT
ANT
4NT
SNT
5
SNT
6
6
GNT
GNT
GNT
7
7
7NT
7NT
7NT
P
X
XX
XX
P
XX
(a)
(b)
(c)
Figure 5: Action heatmaps for checkpoint models. (a) Early model. (b) Intermediate model. (c) Final model.
Table 2: Opening table comparisons. "bal" is abbreviation for balanced distribution for each suit.
opening bids
ours
SAYC
8-20 HCP 8-18 HCP, 4+
12+ HCP, 3+
1
12+ HCP, 3+O)
10
8-18 HCP, 4-6
12+ HCP, 5+
1
7-16 HCP, 4-6
12+ HCP, 5+
INT
14-18 HCP, bal
15-17 HCP, bal
24
8-13 HCP, 5+4
22+ HCP
20
7-11 HCP, 5+1
5-11 HCP, 6+0
20
7-11 HCP, 5+
5-11 HCP, 6+0
20
7-11 HCP, 5+
5-11 HCP, 6+
2NT
14+ HCP. 4+8 4+Q
20-21 HCP, bal
5.2 BIDDING PATTERN EVOLUTION
It is important to be understand how the model evolves during the selfplay training. We pick three different checkpoint models along single training trajectory, and check the frequency of each bid used. The result is shown in Figure S and bids are non-dccr,,s,ng, most contracts end at 6 or 7-level. This is clearly a very naive model. (b) is intermediate model after about 10k minibatches training. The model learns that high level contracts are unlikely to make, and gradually starts to explore low level contracts that can make with the right hands. (c) is the final model which learns prioritize NT and major contracts through information exchange and optimized categorization of various hands.
(a) is early model. Since it behaves mostly randomly,
5.3 OPENING TABLE
There are two mainstream bidding system human experts use. One is called natural, where opening and subsequent bids usually shows length in the nominated suit, e.g. the opening bid 1 usually shows or more with a decent strength. The other is called precision, which heavily relies on relays of bids to partition the state space, either in suit lengths hand strengths. e.g. opening bid of & usually shows 16 or more High Card Points (HCP/)) and a subsequent 1 can show or more To further understand the bidding system the model learns, it is interesting establish opening table of the model, defined by the meaning of each opening bid. We select one the best models, and check the length of each suit and HCP associated with each opening bid. From the opening table, it appears that the model learns a semi-natural bidding system with very aggressive openings.
5.4 BIDDING EXAMPLES
We check a few interesting hands from the tournament between our model and Wbridge5. We present the following 5 examples in Figure 6
'Hihh Card Points heuristic evaluate hand strength, which counts A=4, K=3, Q=2, J=1
7

---

Under review as a conference paper at ICLR 2020
Nort
W M
Narh
W
North
W
14:
P 19
KQ19
AJ1004
FK102
SNT
2+
Y KJJ7
K82
JJ4
24 3+
A62
AJ11554
P
A.119872
West
East
West
East
West
East
Y...5
AQ10854
0.-105
K88
KK88765
1-0083
+ J9754
-X987
x003
+ .944
South
A1009 Q82
North
South
North
N8: EW
No: a EW:
+5
NS: BWW:
(b)
(c)
North
North
A10754
J8763
09
Q6S
West
East
East
8543
*AQ.14
AAK10852
AJ108722
South
1097
JB3
Q97
Q982
56 Veai
K3703
104
West
JB4
NS: EWW
(d)
(e)
Figure 6: Bidding examples. D marks the dealer, and red seats indicate vulnerable side.
(a) This is a purely collaborative auction where our agents sit artificial short & suit. North shows his V suit, and South raises his partner in return. With strong hold North re-raise to 4V, a game contract. The best contract determined by double dummy solver is 60, but it is due to the favorable position of missing honor cards, it is not worth the risk to bid it.
South chooses to open an
(b) This is a competitive auction where our agents sit East-West. The first few bids are natural. East, holding 5 cards in both V and 4 takes action to double opponent's 's INT bid. While partner is silent East balances with 2 bid again. This successfully pushes opponents into non-making 3 contract.
(c) Preemptive bids play important role in bridge bidding. Historically level opening indicates a very strong hand, but modern bidding system bids it with a relative weak hand with a long suit (called weak 2). It is due to this hand type is much more frequent, and can disrupt opponent's bidding by taking away the bidding space. In a standard system weak 2 usually promises 6 cards in the nominated suit, but from the opening table we can see that our agents do it more aggressively with routinely 5 cards. In this hand our agent opens weak 27, and as a result North-South fails to find their best contract in and bids a non-making 3 instead, due to lack of information exchange.
(d) Double is critical bid, unique to competitive bidding. The original meaning is that the doubler is confident to beat opponents' contract. However it can also be used to transmit other information. In this hand, our agent sitting North doubles opponent's 1 with short & suit and support for other suits, and a strong hand. This aligns well with the meaning a modern "takeout double".
(e) The model jumps to 4 too quickly. While the contract is making, it fails to investigate slam opportunities on this hand.
6 CONCLUSION AND FUTURE WORK
In conclusion, we provide a strong baseline that is the state-of-the-art in bridge bidding, with a significantly smaller model. We offer insights through ablation studies to rethink about the training process and belief modeling in imperfect information games. We also interpret the learned bidding system through statistical visualizations and bidding examples. Our code, model and experimental data will be publicly available. We believe this addition is beneficial to the bridge community and imperfect information game researchers, to push forward further research in this direction. It remains a challenging problem to correctly model belief, to reason counterfactually. and to communicate efficiently in multi-agent imperfect information games. We leave this future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective. CoRR, abs/1902.01724. 2019. URL Bttttt/://arae org/ wttp:bbb//9...01222/222
David Balduzzi, Karl Tuyls. Julien Pérolat, and Thore Graepel. Re-evaluating evaluation. CoRR.
1as/18806022444 2018. URL http: arxiv. org/ abs wwtp://ggg/abb/bsbeeeesee
02643
GW Brown. Iterative solution of games by fictitious play, in "activity analysis of production and allocation" koopmans, ed.), 1951.
Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(637)) 424, 2018. ISSN 0036-8075, doi: 10. 10.1126/sieece.aaa0 a73. www.eeeeeeecccc..eeeeeee URL htt.://pw....o.eeeee...ec//o/o/r 33t.////5//337///// 6374/4374444
Murray Camphell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134
(1-2):57-83, 2002. 2002.
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaming awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 122-130. International Foundation for Autonomous Agents and Multiagent Systems, 2018a.
Jakob N. Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent rein- forcement learning. CoRR, abs/1811
oww//org//........ 11111111145 .01458, 2018b. httl:t/t:///aaa http:////a.w.o../ab/a/////
01458
Matthew L Ginsberg. Gib: Steps toward an expert-level bridge-playing program. In IJCAI, pp. 584-593, Citeseer, 1999.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games. arXiv preprint arXiv:1603.01121, 2016.
Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International Conference Machine Learning, pp. 805-813, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver. and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1933 2016.
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisy, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337) 2017. ISSN 0036-8075. doi: 10.1 10.1126/science..amm amm.6666........m660.. URL
6337/500
Jiang Rong, Tao Qin, and Bo An. Competitive bridge bidding with deep neural networks. CoRR, 00900
abs/1903.00900, 2019. URL htt.htttpp//arx.voor////b/s
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiesser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
David Silver, Julian Schrittweeeer, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of without human knowledge. Nature, 550(7676):354, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 36206419) 2018. ISSN 0036-8075. doi: 10. 2wt////iecce.arbb..... aar6404. URL httt::/ssciene swcincc........cc...e.... sci eeeeenneaag...mmmomm wwt.cootteg///eeeeee/eoee 641 9/1140

---

Under review as a conference paper at ICLR 2020
Gabriel Synnaeve, Zeming Lin, Jonas Gehring, Dan Gant, Vegard Mella, Vasil Khalidov, Nicolas Carion, and Nicolas Usumier. Forward modeling for partial observation strategy games-a starcraft defogger. In Advances in Neural Information Processing Systems, pp. 10738-10748, 2018.
Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C. Lawrence Zitnick. ELF: an extensive, lightweight and flexible research platform for real-time strategy games. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA. USA, pp. 2656-2666, 2017.
Zheng Tian, Shihao Zou, Tim Warr, Lisheng Wu, and Jun Wang. Learning multi-agent implicit communication through actions: A case study in contract bridge. a collaborative imperfect- information game. CoRR, abs/1810.04444, 2018. URL http: btttp://arxae wwreiiiorgg// wwwpaaa/////// abs 1810 04444
Chih-Kuan Yeh and Hsuan- -ien Lin. Automatic bridge bidding using deep reinforcement learning. CoRR,
2016. URL
607. 03290
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in neural information processing systems, pp. 1729-1736, 2008.
A THE BRIDGE GAME
Bridge is played with standard 52-card deck, and each player is dealt with 13 cards. There are two phases during the game, namely bidding and playing. After the game, scoring done based on the bidding and playing. An example of contract bridge bidding and playing in shown in Figure 7
W
N
E
S
1
24
3
4
4
P
P
P
Figure 7: (a) A bidding example. North-South prevail and will declare the contract 4 During the bidding, assuming natural bidding system, the bid 1 24, 4 and 4 are natural bids, which shows lengths in the nominated suit. The bid 3 is an artificial bid, which shows a good hand with A support for partner, and shows nothing about the & suit. To make the contract, North-South needs to take 10 tricks during the playing phase. (b) A playing example. Currently shown is the 2nd round of the playing phase. The dummy's card is visible to all players, and controlled by his partner, declarer. In the current round North player wins with K, and will lead the next round.
Bidding phase. During the bidding phase, each player takes turns to bid from 38 available actions. The sequence of bids form an auction. There are 35 contract bids, which consists a level and strain, ranging from an ordered set {1+ 14, 19, 1NT, ...NT where NT stands for No-Trump. The level determines the number of tricks needed to make the contract, and the strain determines the trump suit if the player wins the contract. Each contract bid must be either higher in level or higher in strain than the previous contract bids. There are also 3 special bids. Pass (P) is always available when a player is not willing to make a contract bid. Three consecutive passes will terminate the auction (Four if the consecutive passes happen in the beginning of the auction), and the last contract bid becomes the final contract, with their side winning the contract. Double (X) can used when either opponent has made a contract bid. It will increase both the contract score and the penalty score for not making the contract. Originally this is used when a player has high confidence that opponent's contract cannot be made, but it can also be used communicate other information. Finally, Redouble (XX) can be used to further amplify the risk and reward of a contract, if the contract is doubled. Similarly, this bid can also be used to convey other information.
10

---

Under review as a conference paper at ICLR 2020
Playing phase. After the bidding phase is over, the contract is determined, and the owner of the final contract is the declarer. His partner becomes dummy. The other partnership is the defending side. During the playing phase, there are 13 rounds and each rounds the player plays card. The first round starts with the defending side, and then dummy immediately lays down his cards, and the declarer control both him and dummy. The trump suit is designated by the strain of the final contract (Or None if the strain is NT). Each round, every player has to follow suit. If a player is out of a certain suit, he can play trump card beat it. Discarding other suits is always losing in this round. The player with the best card wins a trick, and will play first in the next round. The required number of tricks to make the contract is contract level enough tricks, they make the contract. Tricks in addition to the required tricks are called over-tricks. they fail to make the contract, the tricks short are called under-tricks.
At the end of the game, the declaring side wins
Scoring. if the contract is made, the declaring side will receive contract score as reward, plus small bonuses for over-tricks. Otherwise they will receive negative score determined by under-tricks. Contracts below 40 (except 3NT) are called part score contracts, with relatively low contract scores. Contracts 40 and higher, along with 3NT, are called game contracts with a large bonus score. Finally, Contract with level 6 and are called small slams and grand slams respectively, each with huge bonus score made. To introduce more variance, a vulnerability to randomly assigned to each board to increase both these bonuses and penalties for failed contracts. After raw scores assigned the difference is usually converted to IMPs scale in a tournament match setting, which is roughly proportional the square root of the raw score, and ranges from 0 to 24.


Reviews:
[{"review_id": "r1xKJ15TFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In recent years machine learning methods enjoy great popularity in the field of game theory. The paper presents a bidding agent for the card game contract bridge trained through selfplay. Contract bridge is an imperfect information game which has a bidding and a playing phase. \nThe bidding phase is challenging for computer agents because there are 1047 possible sequences and through the bidding process, the player has to exchange information with his game partner. The agent is trained using deep reinforcement learning without modeling of belief in other agent’s state or using human expert data.\nThe paper is easy to follow, it is almost free of errors and very well structured. The good readability of the text makes it easy to understand the key aspects of the work. The authors give a clear review of the related literature and work done on this field and use the approaches by Yeh & Lin (2016) and Rong et al. (2019) as baselines for their own method. The presented bidding agent is tested against Wbridge5 and outperforms the state-of-the-art approach by Rong et al. (2019). For a better comparison the authors also test the strategy of modelling belief but prove (controlling it via a hyper-parameter) that it doesn't help in training their model. \nA strength of the paper is the interpretation of the training process and its statistical visualization. By investigating the bidding action in the training process the authors can show the learned strategy of the agent which is compared to the mainstream bidding systems of human experts.\nThe game of contract bridge is explained in the appendix which is a big plus so readers who didn't know this card game can follow the shown examples of bidding actions.\nThe authors plan to publish their code, the trained model and the experimental data so the results will be reproduceable for future work. \n\nOverall, I have troubles in assessing the novelty of the system, since it is not my area of expertise. A concern is that there might be better fitting venues for this topic than ICLR.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJl0Je-1cr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Simple is Better\n--------------\n\nThis paper develops a method to train agents to bid competitively in the game of Bridge. The authors focus on the bidding phase of the game and develop a model to predict the best bid to make at each turn of the phase. The difficulty in the bidding lies in understanding the signals provided by your own teammate as well as the opponent team in order to estimate the state of the game, which is partially observable since each player cannot see the hands of the three other players. The authors show that explicitly modeling the belief of other agents is not necessary and that competitive performance can be achieved with self-play. \n\nWhile the application is interesting, there is not much novelty in the method and factoring that, the empirical study is lacking as well.\n\nPros:\n1. Interesting application of self-play to a multi-agent setting with limited communication!\n\nCons:\n1. There doesn't seem to be much novelty method-wise other than the input representation of the game state. Given that there is an oracle which provides information on how the game would progress assuming optimal agents, could this even be turned into a standard classification problem?\n2. It is not clear what the baselines being compared to are (baseline16 and baseline19). Also, the training curves are missing error bars. On the same lines, how many runs are the 'std' numbers in Table 1 reported over?\n\n\nOther comments:\n1. The authors emphasize that the method is not 'domain-specific'. Given that they build the representation using knowledge of bridge, I'm not sure if this is an appropriate statement to make. \n2. The results section is a bit unclear. First, what are the numbers in table 1? The authors mention 'performance' but are these the number of tricks won or the difference between what the team bid and the actual number of tricks won? The baselines are unclear as well, so it is hard to evaluate the results.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bklt_hRCYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a deep learning agent for automatic bidding in the bridge game. The agent is trained with a standard A3C reinforcement learning model with self-play, and the internal neural network only takes a rather succinct representation of the bidding history as the input. Experiment results demonstrate state-of-the-art performance with a simpler model. The authors discuss some findings with the proposed agent, such as the lack of need to explicitly model the belief and the possibility to self-train with different variants of opponents. Some visualization is also provided to understand how the trained agent behaves.\n\nIt is recommended to weak-reject the paper. There is an obvious contribution of using a simple model to reach state-of-the-art performance. The only concern is whether such a contribution is sufficient to warrant acceptance for a top conference. While setting a new milestone for the community is important (and having open-source code and data in the future will lead to a huge impact for the community), the current paper does not appear to have introduced sufficient \"other contributions.\" In particular,\n\n(1) The ablation study is very shallow. Section 4.2 shows four things: [a] training with an auxiliary belief task does not help the proposed agent; [b] simple representation is sufficiently good; [c] using a pool of opponents won't do better than using the most recent model as opponent; [d] frequency of updating the opponents does not matter much.\n\nFor [a], it is certainly interesting. But one could get to a deeper understanding by, for instance, zooming in to a ratio of 10^-3 or less, to see if there are sweet points of the choice of r. One could also get to a deeper understanding by checking how well BCELoss is doing, and how well the value loss is doing, to understand the trade-off. It is not even clear what the ranges of BCEloss and the value loss are, making it uncertain on whether r is properly chosen.\n\nFor [b], deeper study could be taken for analyzing \"why\", especially given that the more complicated encoding is significantly worse. The hand-waving explanation of \"The potential reason why our encoding performs better is that the intrinsic order of bridge bidding is already kept by the action itself ...\" at best suggests that the information between the baseline19 encoding and the author's are of the same information amount, but does not say why baseline19 is significantly worse. The authors leave a big question mark here that does not seem to match the grand title of \"simple is better\" of this paper.\n\nFor [c], deeper study could be taken for analyzing \"what if some worse opponents are included\" or \"what if some targeted opponents are included.\" For instance, what if the authors include early-stage models to increase diversity? What is the trade-off between opponent capability/diversity and performance? What if using baseline16 as opponents---would the agent then beat baseline16 more easily? Note that baseline16 and the proposed agent basically learned a semi-natural bidding system. What if a precision system is set as a potential opponent---would the learned policy be very different?\n\nFor [d], getting a result that the update frequency leads to similar performance can hardly match the claim \"We show that selfplay schedule and details are critical in learning imperfect information games.\" Also, the big error bar and the inconclusive trend in Table 1(B) does not answer the question on what frequency should best be used. Deeper discussions shall be included. \n\n(2) Besides the ablation study, there is very little information about the design choices. For instance, why does the authors choose the specific RFC block? Would a full FC block do better or worse? Would a shallower network do better or worse? Actually, the title \"simple is better\" may be misleading, as the authors did not study even simpler choices. For RL algorithm, why A3C but not other algorithms?\n\n(3) The authors' finding that the agent is conservative and does not bid high appears interesting. Nevertheless, it perhaps suggests that the authors have not compared with strong bidding systems (like precision) that aimed for bidding high to win more. That is, the claimed advantages are over semi-natural bidding systems. More study on whether the agent should bid high could greatly enrich this paper.\n\n===\n\nI've read the rebuttal. I thank the authors for the discussion on some possibilities in deepening the paper. I suggest the authors to avoid some hand-waving claims such as \"We argue it is easier to learn the connections between them.\" In general, some of the claims (such as \"simple is better\" in the title) lacks sufficient scientific support, and I believe that this paper will have more impacts if it focuses more on supporting the claims with science evidence, not with human descriptions.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new training method for an end-to-end contract bridge bidding agent. Reviewers R2 and R3 raised concerns regarding limited novelty and also experimental results not being convincing. R2's main objection is that the paper has "strong SOTA performance with a simple model, but empirical study are rather shallow."

Based on their recommendations, I recommend to reject this paper.

Decision: Reject