Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Simple is Better
--------------

This paper develops a method to train agents to bid competitively in the game of Bridge. The authors focus on the bidding phase of the game and develop a model to predict the best bid to make at each turn of the phase. The difficulty in the bidding lies in understanding the signals provided by your own teammate as well as the opponent team in order to estimate the state of the game, which is partially observable since each player cannot see the hands of the three other players. The authors show that explicitly modeling the belief of other agents is not necessary and that competitive performance can be achieved with self-play. 

While the application is interesting, there is not much novelty in the method and factoring that, the empirical study is lacking as well.

Pros:
1. Interesting application of self-play to a multi-agent setting with limited communication!

Cons:
1. There doesn't seem to be much novelty method-wise other than the input representation of the game state. Given that there is an oracle which provides information on how the game would progress assuming optimal agents, could this even be turned into a standard classification problem?
2. It is not clear what the baselines being compared to are (baseline16 and baseline19). Also, the training curves are missing error bars. On the same lines, how many runs are the 'std' numbers in Table 1 reported over?


Other comments:
1. The authors emphasize that the method is not 'domain-specific'. Given that they build the representation using knowledge of bridge, I'm not sure if this is an appropriate statement to make. 
2. The results section is a bit unclear. First, what are the numbers in table 1? The authors mention 'performance' but are these the number of tricks won or the difference between what the team bid and the actual number of tricks won? The baselines are unclear as well, so it is hard to evaluate the results.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose a deep learning agent for automatic bidding in the bridge game. The agent is trained with a standard A3C reinforcement learning model with self-play, and the internal neural network only takes a rather succinct representation of the bidding history as the input. Experiment results demonstrate state-of-the-art performance with a simpler model. The authors discuss some findings with the proposed agent, such as the lack of need to explicitly model the belief and the possibility to self-train with different variants of opponents. Some visualization is also provided to understand how the trained agent behaves.

It is recommended to weak-reject the paper. There is an obvious contribution of using a simple model to reach state-of-the-art performance. The only concern is whether such a contribution is sufficient to warrant acceptance for a top conference. While setting a new milestone for the community is important (and having open-source code and data in the future will lead to a huge impact for the community), the current paper does not appear to have introduced sufficient "other contributions." In particular,

(1) The ablation study is very shallow. Section 4.2 shows four things: [a] training with an auxiliary belief task does not help the proposed agent; [b] simple representation is sufficiently good; [c] using a pool of opponents won't do better than using the most recent model as opponent; [d] frequency of updating the opponents does not matter much.

For [a], it is certainly interesting. But one could get to a deeper understanding by, for instance, zooming in to a ratio of 10^-3 or less, to see if there are sweet points of the choice of r. One could also get to a deeper understanding by checking how well BCELoss is doing, and how well the value loss is doing, to understand the trade-off. It is not even clear what the ranges of BCEloss and the value loss are, making it uncertain on whether r is properly chosen.

For [b], deeper study could be taken for analyzing "why", especially given that the more complicated encoding is significantly worse. The hand-waving explanation of "The potential reason why our encoding performs better is that the intrinsic order of bridge bidding is already kept by the action itself ..." at best suggests that the information between the baseline19 encoding and the author's are of the same information amount, but does not say why baseline19 is significantly worse. The authors leave a big question mark here that does not seem to match the grand title of "simple is better" of this paper.

For [c], deeper study could be taken for analyzing "what if some worse opponents are included" or "what if some targeted opponents are included." For instance, what if the authors include early-stage models to increase diversity? What is the trade-off between opponent capability/diversity and performance? What if using baseline16 as opponents---would the agent then beat baseline16 more easily? Note that baseline16 and the proposed agent basically learned a semi-natural bidding system. What if a precision system is set as a potential opponent---would the learned policy be very different?

For [d], getting a result that the update frequency leads to similar performance can hardly match the claim "We show that selfplay schedule and details are critical in learning imperfect information games." Also, the big error bar and the inconclusive trend in Table 1(B) does not answer the question on what frequency should best be used. Deeper discussions shall be included. 

(2) Besides the ablation study, there is very little information about the design choices. For instance, why does the authors choose the specific RFC block? Would a full FC block do better or worse? Would a shallower network do better or worse? Actually, the title "simple is better" may be misleading, as the authors did not study even simpler choices. For RL algorithm, why A3C but not other algorithms?

(3) The authors' finding that the agent is conservative and does not bid high appears interesting. Nevertheless, it perhaps suggests that the authors have not compared with strong bidding systems (like precision) that aimed for bidding high to win more. That is, the claimed advantages are over semi-natural bidding systems. More study on whether the agent should bid high could greatly enrich this paper.

===

I've read the rebuttal. I thank the authors for the discussion on some possibilities in deepening the paper. I suggest the authors to avoid some hand-waving claims such as "We argue it is easier to learn the connections between them." In general, some of the claims (such as "simple is better" in the title) lacks sufficient scientific support, and I believe that this paper will have more impacts if it focuses more on supporting the claims with science evidence, not with human descriptions.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In recent years machine learning methods enjoy great popularity in the field of game theory. The paper presents a bidding agent for the card game contract bridge trained through selfplay. Contract bridge is an imperfect information game which has a bidding and a playing phase. 
The bidding phase is challenging for computer agents because there are 1047 possible sequences and through the bidding process, the player has to exchange information with his game partner. The agent is trained using deep reinforcement learning without modeling of belief in other agent’s state or using human expert data.
The paper is easy to follow, it is almost free of errors and very well structured. The good readability of the text makes it easy to understand the key aspects of the work. The authors give a clear review of the related literature and work done on this field and use the approaches by Yeh & Lin (2016) and Rong et al. (2019) as baselines for their own method. The presented bidding agent is tested against Wbridge5 and outperforms the state-of-the-art approach by Rong et al. (2019). For a better comparison the authors also test the strategy of modelling belief but prove (controlling it via a hyper-parameter) that it doesn't help in training their model. 
A strength of the paper is the interpretation of the training process and its statistical visualization. By investigating the bidding action in the training process the authors can show the learned strategy of the agent which is compared to the mainstream bidding systems of human experts.
The game of contract bridge is explained in the appendix which is a big plus so readers who didn't know this card game can follow the shown examples of bidding actions.
The authors plan to publish their code, the trained model and the experimental data so the results will be reproduceable for future work. 

Overall, I have troubles in assessing the novelty of the system, since it is not my area of expertise. A concern is that there might be better fitting venues for this topic than ICLR.
review_title: Official Blind Review #1

Meta-review: This paper proposes a new training method for an end-to-end contract bridge bidding agent. Reviewers R2 and R3 raised concerns regarding limited novelty and also experimental results not being convincing. R2's main objection is that the paper has "strong SOTA performance with a simple model, but empirical study are rather shallow."

Based on their recommendations, I recommend to reject this paper.

Decision: Reject