Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a new method for calculating distributions over states in Markov problems in which the true state is not directly observable. The method relies on using an existing model and then fine tuning the results online at each time step. Experiments presented show this gives a more accurate distribution, and that this more accurate distribution leads to statistically significant improvements in performance.
main_review: The abstract, introduction and background is very clearly written. It does a good job of introducing the problem to be solved and outlining the method to be used at a high level.

The method itself is rather mathematically simple. It is based on simple mathematical techniques in the area and effectively amounts to sampling from known distributions.

While the authors state that they present an algorithm, it might be better to view this as a framework, as the method involves an externally supplied model as an input, and the fine tuning method needs to be tailored to this model. The experiments performed use only a single model, which limits the conclusions that can be drawn from the results.

The experimental results are only provided for a single domain, albeit for a HMM, POMDP and FOSG setting in the same model. The choice of domain seems rather poorly selected, as the basic models used as a comparison in the domain already appear to be almost optimal, meaning there is little improvement from the new technique - while the results may be statistically significant, they appear to be a very minor improvements in effect size.

It would be very useful to see some mention of time in the results section. The procedure doesn't appear particularly onerous but in a game playing setting, there may be time limits and any improvement in quality would need to be seen after X seconds, not after X iterations.

These criticisms aside, the experiments appear to be fairly conducted and reported, and look at sensible aspects of the problem, with a good choice of baselines/competitors.


summary_of_the_review: The authors present an interesting idea well. However, the experimentation is very limited (even for a conference paper) and the results given show a very small effect size. More data is needed to convince that the technique is useful.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper seeks to understand whether finetuning a learned belief state model can enable improvements in performance in partially observable problems, in particular in the Hanabi domain. The general intuition is that while the original parametric model is trained on a wide distribution of belief distributions / input histories, during deployment, the model need only be good on the sequence that is actually realized. The authors demonstrate that indeed such techniques lead to improved performance over a non-adaptive belief state method, in particular when there is significant stochasticity in the transitions.

main_review: 
The problem of learning and optimizing over belief states is an important problem in reinforcement learning, and while the ideas in this paper are not particularly novel, the core idea is implemented and tested well. I thought that the lack of details made certain section rocky, but in general found the paper a pleasure to read. 

Strengths:

- Simplicity is perhaps the main strength of this paper: the method is very closely inspired by a wide body of existing work on belief filtering, and the method looks like fine-tuning a learned model on augmented data. Empirical details withstanding, the generic idea of re-allocating the "capacity" of a neural network at test-time in this way seems well-justified and simple to implement.

- The paper is easy to read, and exposes the main concepts well.

Weaknesses:

- The paper is a little light on exact algorithmic details (and I did not see more in the appendix). In particular, I think it would be useful to exactly detail out the algorithm being proposed in a more fleshed-out manner somewhere in the main text or appendix. It would also be a good idea within Section 2 to more explicitly what the goal of the belief state modelling problem is, and what the belief state model will be used downstream for (as this dictates what approximations are useful / worthwhile). I found the latter point difficult to understand in the current treatment.
- Along the same lines, I think it would be useful to more explicitly outline in the appendix exactly how BFT is used for down-stream decision-time planning (e.g. with RL Search). To generate an action at a time-step, is BFT (for 10k timesteps) run only once, or is it necessary to run multiple times within the policy search loop? 
- I found no details for exactly how the particular choice of 10,000 gradient steps for BFT were chosen. How sensitive is BFT to the number of gradient steps that are taken to fine-tune the learned model? How much does this depend on the quality of the original model: e.g. with a worse original model, does running BFT degrade the quality of the belief since BFT bootstraps off of the original model? 
- It seems that this problem assumes access to the "true" underlying state space and the transition dynamics / emission distributions in this state space, in which case the main purpose of the neural network is simply amortization of an exact search problem. It would be useful to make these assumptions more explicit earlier (perhaps even in the introduction). As a question to the authors, how would these techniques transfer to the setting where these assumptions do not hold (in particular, where the underlying state space $\mathcal{X}$ is not known).
- I think the point that the method works because it "refocuses" the capacity of the parametric belief state model is interesting, but I don't think it is supported very strongly by the provided data. It would be interesting to see whether these effects diminish as the neural network capacity increases (or equivalently, if the effects amplify as it decreases), since presumably, the better the original neural network is at the belief modelling problem, the less impact a fine-tuning step should have. Along these lines, I think it would be interesting to have some discussion as to whether fine-tuning procedures can simply be supplanted by training bigger networks.
summary_of_the_review: (Copied from above) The problem of learning and optimizing over belief states is an important problem in reinforcement learning, and while the ideas in this paper are not particularly novel, the core idea is implemented and tested well. I thought that the lack of details made certain section rocky, but in general found the paper a pleasure to read.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a strategy for improving a trained parametric model for belief-state update, by generating and training on new data, online.  This strategy is particularly helpful for dealing with problems of covariate shift (when, e.g, the system has moved outside the training regime) and dependence on unobservable factors (e.g. the agent's policies in a game setting).   This fine-tuning strategy is shown both to improve the fidelity of belief-state updates and to have an ultimate impact in the quality of play in Hanabi, a game with a substantial amount of private information.
main_review: This is an interesting paper that is fairly novel, as far as I know.  However, there were some problems in the exposition that left me feeling that I might have misunderstood some aspects of it.  I do think it will be important to clarify some of these points in any published version of the paper.  Many of my confusions were about the types or APIs of various components.  I will start with larger questions, and then list some minor ones.

In section 3.1, I immediately was confused by the type of b^{t+1}_{Y^{t+1}}.   Because Y is capitalized, it seems to be a random variable?  But that doesn't make a lot of sense to me.  Here are some possible interpretations of b^{t+1}_{Y^{t+1}}:
- Given a particular observation Y_{t+1}, then it's just the next belief state (but why is Y capitalized?)
- Not given an observation, it's a random variable over next beliefs b (where the distribution over b is introduced by distribution over observations)
- Not given an observation, it's the next belief state we would get if we made no observation.
- Not given an observation, it's some sort of expectation over the next belief (this is tricky, I think.)

The discussion of the sampling procedure (in that same section) says it produces samples of X^{t+1}---but it seems to be important that we are getting samples of the Y_{t+i}, as well, so we can use them to fine-tune our estimator.

I'm not sure the analogy with approximate dynamic programming is helpful;  or, at least, it would help to clarify it.  In particular, is it critical to actually fine-tune \theta?   An alternative would be to stick with the particle view, and use the sampled X's to represent the posterior belief, or use some combination of the belief produced by the parameteric model \theta and the particles more explicitly, rather than actually changing the \theta.   (To me, it would feel much more like ADP if you were training some function b_\theta to represent the belief-state directly, on each iteration, which would be yet another possible approach I suppose, except that representing belief states is notoriously difficult.)

I really wanted Figure 2 (which is called table 2) clarify things for me, but it really didn't help me at all.   I think something more like a data-flow diagram that illustrates how \theta gets changed over time (and makes very clear that \theta doesn't parameterize B (nothing parameterizes or really explicitly represents B itself), but instead parametrizes the state-estimator "box".) would be more helpful.

The standard error values seem quite small (given the amount of potential variance in the different training processes).  Please state very clearly exactly what sources of variance are being captured here:  initializations / batches /data for training the initial \theta, randomness due to evolution of the game trajectory at runtime (due to card shuffling, randomness in agents' play, etc.), variance of sampling to generate the fine-tuning data, variance of the fine-tuning training (potentially due to batching I guess).   What were the various "N" values?

The performance improvements seem to be real, and interesting, but the gains aren't enormous.  This is fine, but I'm not sure I'd claim that performance is *greatly* improved.

More minor:
- I appreciate the generality of the approach and the desire to be agnostic to the form and training details of f_\theta, but I had to go digging, a bit, into other papers to try to understand the actual approach you used.  A bit more detail about that would have been helpful in understanding your approach.
- In early parts of the paper you talked about going from step t to step t+1;  later it was from step t-1 to step t.
- "which is naively from independence assumptions"
- "covariate distribution should not impact model performance" (missing "shift")
summary_of_the_review: This paper has interesting ideas that seem to be useful.  Clarity could be improved but interesting enough to publish.  I wanted to give it a 7.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper concerns the idea of directly learning a model of a belief state MDP from samples from the ground truth dynamics and observation models. One could then use that model for decision-time planning (e.g. some form of tree search). In particular, the paper aims to improve upon this idea by fine-tuning the belief state model before search using samples from local state region. Experiments are performed in Hanabi, showing that fine-tuning does improve belief accuracy and enables search through belief space to improve control performance in problems where exact inference is impractical.
main_review: -- After Author Response --

I am leaving my original review below for transparency, but my most significant concerns have been addressed. In particular:
- The RLSearch paper has been accepted for presentation at NeurIPS 2021.
- The authors have provided the sample sizes for their averages and they are sufficient to support their claims.

I still think the authors should aim to make the paper more self-contained by providing more algorithmic details about RLSearch (which is a new approach), how it interacts with BFT, and why it was selected for these experiments instead of a more well-established search method.

-- Original Review -- 

Belief state inference is an important step toward developing agents that can make sound decisions in partially observable environments. A great deal of effort has been spent studying exact inference or approximate inference with access to ground truth state distributions. This paper's focus on bringing this tool to more complex domains where representing distributions and performing exact inference are both a challenge is worthwhile and is bound to be of interest to the ICLR community. The results also fit into a larger story about neural network fine-tuning/meta-learning, which is certainly of interest.

Technically, the approach is only modestly novel, mainly applying an idea that has been studied elsewhere in a context where it hasn't been applied yet. That said, in my opinion, the empirical results are both novel and significant. In particular, the idea of directly learning a belief state model has not gained a lot of traction because it, frankly, doesn't work all that well! We can see in the 7-card Hanabi results that performing search on such a model yields essentially no benefit (we are lucky that it didn't cause harm!). The fact that the fine-tuning enabled the approximate belief model to yield a planning benefit is notable and a promising signpost toward future work, which might enable belief state inference to scale to larger, more interesting partially observable problems.

All that said, I do have a significant concern about this paper. The empirical results rely heavily on an existing algorithm called RLSearch. As far as I can tell, the paper that describes RLSearch has not been peer reviewed and has only been available on arXiv for about a month at time of reviewing. The paper makes a very brief attempt to explain RLSearch at a high level, and how that algorithm is altered for this paper, but this is not sufficient. From this paper, I do not know what algorithm is being performed in these experiments or why RLSearch is even a sound foundation to build upon, let alone why it is the right base algorithm to combine with BFT in order to answer these empirical questions. Can BFT be applied to other search methods? Is there some special synergy with RLSearch since they both appear to be fine-tuning methods? If the answers to these questions is no, then I would recommend using a baseline algorithm that is more established and better studied. If the answer is yes, then that needs to be made far more explicit and clear. If the answer is "we don't know" then I think that's a major missing piece of this work.

The fact is that the paper is not sufficiently self-contained and the parts that it doesn't contain haven't been peer reviewed. That makes me deeply uncomfortable and makes it difficult for me to confidently assess the technical quality and the significance of the findings.

Another concern I have is that I don't know how many independent trials are represented in the empirical results. Maybe I just missed it, but it seems like a very important detail to state. If the number is not sufficiently high, that would raise questions about the strength of the support for the conclusions.

A couple of more minor issues:
- The references section is, frankly, a mess. There is no consistency in the formatting and content of the references and some don't even list a venue, just authors, a title, and a year.
- p. 2: "At the time of writing, there has been no successful demonstration of this." It wasn't clear to me at this point what "this" refers to. Even now I'm not totally sure. It would be good to have a more clear expression of the open problem here.
summary_of_the_review: -- After Author Response --

The paper is well-written and considers an important and challenging problem. The empirical results seem to offer a path to progress in a direction that hasn't shown much promise. The paper could be improved by offering more details about RLSearch, a recently introduced approach upon which the experiments heavily rely.

-- Original Summary -- 

The paper is well-written and considers an important and challenging problem. The empirical results seem to offer a path to progress in a direction that hasn't shown much promise. However, the algorithmic results make heavy use of an algorithm that is not sufficiently described in this paper and that has not been peer-reviewed. Though I would really like to have these results in the literature, that makes me uncomfortable with recommending acceptance.
review_title: null

Meta-review: The paper proposes to fine tune the belief states of a MDP, for later using the learned model for decision-time planning, e.g. via search.
The contribution is well-presented, motivated and focused to a specific scenario, which is generally considered challenging in the literature. This scenario is exemplified by the cooperative card game Hanabi, which takes the role of the benchmarking setting for the empirical evaluation of the fine-tuning procedure.

The major concern raised in the review and discussion phases are about the limited evaluation, which is centered around only Hanabi, as well as the magnitudes of the improvements over previous baselines. However, three knowledgeable reviewers agreed that since the setting has been historically challenging, the reported improvements are in fact significant and potentially inspiring future works in this direction. 

The paper is accepted provided that the authors include and polish in the camera-ready the additional experiments over the parameter sensitivities, the ablation tests and the discussions highlighted by the reviewers in the comments.
Decision: Accept (Poster)