Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes an approach to planning in imperfect information games called DSMCP. The algorithm is the basis for a system that won the 2020 reconnaissance blind chess competition. The algorithm views the game as a POMDP by assuming a fixed policy for the opponent. Infostates are represented compactly using a particle filter where each particle represents a set of world states with a maximum size. These are further compressed as "synopses", which are conjunctions or disjunctions of Boolean features over all the states in the set. The synposes are then input to a neural network that has been trained to imitate one of a set of possible policies including several other RBC programs. Policy action choices are mixed with UCB action choices to create a search tree. Saliency of different synopses is found to be somewhat correlated with decreased performance when removing that feature.
main_review: ## Originality

The builds on existing tree search algorithms for imperfect-information games, and algorithms that combine tree search and trained policies. The main original contribution seems to be the formulation of the synopses. There are also numerous design choices made to make the problem tractable that, taken collectively, result in an original system. Related work is cited extensively.

## Quality

The techniques and specific design choices employed are mainly heuristics, but they are well-motivated by the goal of tractability. The key experimental results are the Elo scores from the RBC tournament, which demonstrate that the method achieves a higher playing strength than its competitors. There is a lack of ablation-type results that drill down into the impact of specific choices -- e.g., selection of particular synopsis features, choice of set sizes for approximations, etc. (at least in the main paper). Some results are also presented with little in the way of interpretation -- specifically, Section 7 and Figure 6. What do the results on saliency tell us about the synopsis features, or how could they guide the selection of features?

## Clarity

The paper is generally well-written but is extremely information-dense. It feels like there was a struggle to fit in all the content, and the presentation is fairly terse.

## Significance

Solution methods for imperfect information games are of broad interest. This work does not make any ground-breaking advances, but demonstrates a number of practical techniques for making larger problems tractable. Since the agent won an organized RBC competition, presumably the approach will have some impact on the design of other RBC agents.

## Notes:

Alg. 3: What is $\nu$?

Alg. 4: Should ChooseAction take (N, Q) as arguments?

159-160: Is it correct that $|X| = F$ (the size of the state set is the same as the number of features)? Should it be $|X| = \ell$?
limitations_and_societal_impact: I see little potential for negative societal impact of this work
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents Deep Synoptic Monte Carlo Planning, the technique behind the 2020 champion Reconnaissance Blind Chess (RBC) program Penumbra.  This method samples possible world states to approximate information sets, which it then abstracts uses synopses.  These abstracted information sets are then used monte-carlo planning to determine actions to take in the game.  The approach, with some RBC specific tweaks, is evaluated in the RBC domain and shown to be very successful.

main_review: The core idea of abstracting information sets using the stochastic synopses is, to my knowledge, a novel one and appears to be very successful.  The problem is well-motivated and the results significant for this challenging problem.  On the other hand, I found the paper fairly dense, and the individual contributions not granted enough space for their explanations to be clear.  The notation was not clearly introduced, especially as it related to individual parameters.  In my opinion, this is important work, but the current presentation could be significantly improved to make the contributions more clear and understandable.  

Comments
- I found the algorithm references to be slightly confusing, as the algorithms aren't named, (except for Alg. 1, 2, etc) but then are referred to in other algorithms.  I was able to figure out what was going on, but this made the description a bit harder to follow. 
- Many of the parameters and variables that are used to define the algorithms are not described (that I could find), anywhere in the text or algorithm descriptions.  For example, an $m$ value is referred to multiple times in the text (page 7 line 193, page 8 line 204-205) but the only place it appears is in Algorithm 1.  I would have appreciated some discussion in the text of what this $m$ value controls or influences in the operation of the algorithm.  It was unwieldy to have to flip back from page 8 to the definition of Alg. 1 and then try to figure out how the versions of Penumbra that were evaluated differ.  
- In a similar fashion I could see no discussion or explanation of $b$, $d$, $n_{vl}$, and $\nu$, which are referred to in Alg. 3.  
- Towards the end of Alg 3. "Bandit" is mispelled "Bandi" and the final parameter passed to it is a 0, where it should be false to match other usages. 
- I found the explication of the Synopsis method, a key contribution of the work, hard to follow. In particular, I failed to see the connection between formula 1 and the contents of Figure 4.  Specifically, the example following formula 1 leads me to see each $x_j$ as a different world states possible (in the abstraction set), while the $g_i$ function used refers to moving to a specific square on the board.  Combined with the AND function, this gives a single binary value that tells whether or not it is guaranteed that the knight can move to that square.  In Figure 4, we see bitboards, the same size as the chessboard, which are described as summarizing things like all the places an enemy pawn could appear. The connection between these two was not immediate to me.  Would the bitboard for the example feature (knight movement) have a black bit everywhere that the knight could definitely move, and a white bit where it might not be able to move?  
- How were the 14 different headsets of the network used in the planning?  This was an interesting detail to me that I would have loved to have more of an explanation of.  The text merely says that the policy heads and the All value head were used for planning, but how this occurs was left undescribed. 
- Why does having a smaller l for the opponent lead to more cautious play?  My first thought is that the fewer states that are considered in each opponent infoset the more chance there is that we miss the truth and thus it would seem less cautious and more risky to have fewer.  I would have loved more discussion about this. 
- What did it mean for an opponent to be "recognized as" vs. "unrecognized" in the experimental results? I couldn't find a discussion of what this meant or entailed in the paper anywhere. 
- In the discussion of Saliency (Section 7) I was expecting some conclusions about which type of synopsis features were found to be most impactful.  Am I correct in assuming that the features shown in Figure 6 were also found to be salient in other training runs?  
- As far as the saliency features, how were these determined?  Were these hand-designed, or learned somehow? 

In summary, I really like this work, the problem it addresses, and the methods it uses.  But I was left with many questions and confusions as I tried to parse the contributions and what had actually been done from the paper itself.  

Addendum: I appreciated the author's responses and am more confident that with the changes they describe the work will be clear enough. 
limitations_and_societal_impact: Discussion was adequate
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper describes a Reconnaissance blind chess (RBC) program Penumbra that won the 2020 RBC rating competition. Penumbra’s method is called “deep synoptic Monte Carlo planning” which is a set of bandit, belief state summarization, imitation learning techniques that together act on an underlying POMDP model. 

main_review: Originality: This combination of techniques that leads to an effective algorithm for playing RBC, in my opinion, is novel and interesting. The most original part of the methodology seems to be the synoptic function (summary function) that takes as input approximate infostates and outputs a fixed size summary through the use of logical operators that “scalarize” across the set of possible world states (= a belief state).

Quality: 
The approach to summarizing approximate infostates is clever and makes sense. However, as a reader, it would be nice to see a clearer discussion regarding the related work with respect to the synopsis function.
Can the authors discuss what happens when there is no training data for an opponent? How well does Penumbra handle playing against a new opponent? Is there a possibility of catastrophic outcomes if Penumbra uses the model of one opponent but plays against another? These would be useful to discuss in a limitations section.


Clarity: The paper is generally well-written with intuitive illustrations. It was a pleasure to read. I have a few points below --- it would help to make some notation more precise or give explanations when there is possible ambiguity.
Line 106: Clarify the statement “equivalently, the set of all possible action histories from p’s perspective”. 
Line 104: Why is the action a function defined on a subset of world states?
The discussion about n_particles and the unweighted particle filter deserves more clarity -- I was a bit lost in reading this part as few details are given.
Typo in Algorithm 3: Bandi -> Bandit
How are SoonWin, SoonLose, and PieceCount used?

Significance: Given the performance of Penumbra in the RBC competition, my judgment is that this constitutes a significant contribution to gameplay AI. The method of summarizing infostates seems generalizable to other settings and perhaps could be studied theoretically: how much are we losing by collapsing infostates in this manner? Could the authors give a discussion on this point?
limitations_and_societal_impact: The authors could discuss further limitations given some of the questions posed above. They do point out that they are not trying to compute a Nash equilibrium.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper describes Penumbra, a top reconnaissance blind chess program, with focus on the state abstraction and the Monte Carlo planning algorithm. 
main_review: The main algorithmic contribution is probably the deep synoptic Monte Carlo planning algorithm, but there is little evaluation of it, other than the game program being strong (which can be due to other factors as well). It would be useful to have a more in-depth comparison with other planning algorithms used by the top competitors.

The paper includes a detailed description of the network architecture and other implementation details, but I am not sure that some of these details are too important for the planning algorithm.

Reconnaissance blind chess is a complex domain with many interesting properties for the NIPS audience. Therefore, I regard it as a very good benchmark. However, the particular algorithmic ideas need to be properly evaluated and compared directly with possible alternatives, and not just indirectly through the performance of a game program that happens to include it. As it is, the paper seems a solid paper for a game conference, but I am not sure that it presents sufficient interest for the NIPS audience.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The reviewers had several concerns about this work, but after discussion these were generally agreed to be fixable issues - with one exception. One reviewer felt very strongly that RBC would not be sufficient to demonstrate the generality of the contribution. I suspect that some of the disagreement here may be an issue of different sub communities having different expectations. While RBC may be a familiar and accepted benchmark in one sub community, it may not be familiar in others - even some that aren't that far away.

The potential narrowness of the appeal of this paper indicates that a poster may be the more appropriate route.

I would also encourage the authors to find some way to make their contribution appeal more broadly if is is practical to (convincingly) make the case that it has broader application than to a single benchmark.
Decision: Accept (Poster)