Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new planning method, aiming at improving performance in a large and realistic environment. The proposed method performs as a planning module and can be integrated with different types of online learning agents. Different from tabular search algorithms, the method formulates the planning step as a reinforcement learning problem and updates the policy or action-value function by bootstrapping. Thus, the method reduces the horizon of search and gains learning efficiency in large search spaces and continuous control problems.
main_review: The proposed method improves search efficiency and is applicable in large search spaces. However, the discussion on how key parameters affect the performance is missing. It is also not clear if the hyperparameter setting in the submitted codebase is the same as the setting used for generating the result.

More details are provided below.

The paper discusses different cases separately in detail, including policy gradient methods and action value methods, as well as single-agent and multi-agent settings. This suggests the method can be integrated with different types of reinforcement learning agents. 

The reported performance in Table 1 is averaged over 2000 games, which gives a reliable averaged value. The experiment reported in Figure 1 tests 5 seeds. Testing more seeds would be better, though 5 is acceptable.

The idea of truncating the target is similar to n-step bootstrapping (Chapter7 in [1]). I understand this paper focuses on planning instead of value function learning, though adding references about n-step bootstrapping might help the reader to understand the main idea.

The trajectory length H seems like a key parameter in the algorithm. If H equals the episode length, the update would be the same as MCTS, which means the search goes until the end of the episode. I assume this parameter needs to be tuned carefully in the experiment. It would be nice if there is an experiment discussing how performance changes regarding different search horizons. If the length of the horizon can be fixed and no need to tune, it will also be good to explain how this number should be chosen. This will make it easier for readers to apply this method to other problems.

It may be worth listing hyperparameter settings in the paper, such as network structure and learning rate for both the proposed method and baselines, for reproducibility. I understand that the code base is provided and there are default settings in the code, but it is not clear to me if the default setting is the same setting used for generating the results reported in the paper.

Small Things:
In Algorithm 1, the symbol used in output ($\tilde{\theta}$) is inconsistent with the return ($\theta_N$)

[1] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to replace a Monte-Carlo tree search style search method by instead fine-tuning a neural network by running reinforcement learning on trajectories generated by a perfect model of the environment starting from the current state at inference time. They apply this idea in a cooperative multi-agent planning in the game Hanabi, as well as in single-agent planning in the game Ms. Pacman where the agent has is given access to the simulator. Experimental results suggest that the fine-tuning procedure is able to perform similarly to or better than a strong tabular search method (SPARTA) in Hanabi. In Atari, they demonstrate that the procedure seems to outperform MCTS, particularly as the total search budget is made larger for each algorithm.
main_review: I feel the approach considered in this work is quite interesting and important. It makes sense that ultimately we will want to replace MCTS like procedures with something that generalizes more naturally during the search procedure itself. The experiments presented here seem like a good proof of concept for such methods and the fact that they include results for both a single agent game and a multi agent benchmark is a plus.

My main uncertainty is that the experimental details included are a bit lacking (though at least the code is made available). In particular, see 'Questions about Experimental Details' heading below.

Also, the paper could be made more self-contained in terms of ensuring that all the variables that are used are clearly defined (see Minor Comments and suggestions).

I also am not very knowledgeable of the relevant multi-agent search literature to evaluate the multi-agent portion of the work. But from what I can tell it seems generally well done.

Questions about Experimental Details
====================================

Some details of the experimental methodology such as various hyper-parameters and how they were selected seem to be missing:

Line 131: 'the transitions used to fine-tune the Q-network are sampled with probability p from the global buffer replay' what is this p and how is it chosen? Also, why is the global replay buffer and how is it generated?

Line 236: 'We obtain the best performance with c = 5 and $\beta$ = 0.1' what other values were searched over here?

What hyper-parameters were used for the blueprint Q-learning and PPO agents in Hanabi and Ms. Pacman respectively and how were they chosen?

Minor Comments and Suggestions
==============================

Line 33: 'non-tabular planning algorithms improve performance'->'non-tabular planning algorithms to improve performance'

Between Line 59-60 (line numbers missing here for some reason): is $\Omega^i(\tau_0)$ defined somewhere?

Line 137: Is $B(\tau_t^i)$ explicitly defined somewhere? From looking at the cited SPARTA paper it seems like this should maybe be $B^i(\tau_t)$ if the notation is supposed to be consistent with that work.

Algorithm 1: I think t is kind of overloaded here in that $s_t$ is the state at the current time but $\delta_t$ is $\forall t\in[0,H-2]$ using different variable names would make this clearer.

Line 260: 'like MCTS do'->'like MCTS does'

For the Ms. Pacman results, it would be interesting to see how MCTS and RL-fine-Tuning compare in terms of total number of states expanded in addition to average search time.

Update
======
I thank the reviewers for their clarifications. My empirical concerns are addressed and I will maintain my score. The additional experiments are also welcome.
limitations_and_societal_impact: Limitations are reasonably well covered. I don't see much potential for negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to replace tabular search methods like MCTS with fine tuning of neural policies for online planning. 

The motivation is that fine tuning the neural policies online may be more efficient than doing MCTS which has no generalization between states during the search. 

The key algorithmic contribution is the biased RL objective used for more efficient online fine tuning of the neural policy. 
main_review: Originality: I find the idea of replacing MCTS with online fine tuning of neural policies interesting. However, I don't think overall the paper has enough algorithmic contributions. 

Quality: The method seems correct. Limitations see below. 

Clarity: The paper is easy to follow. 

Significance: As mentioned, I think this is an interesting direction. Also, I think the general problem setup of this paper, which is how to make good online decisions with a blueprint policy and a model important to study. 
limitations_and_societal_impact: My main concern is that this paper does not have enough algorithmic contributions.

My second concern is that this method may only work well when the given blueprint policy is well trained. This is because the key potential advantage of this method over tabular search methods is the effective generalization between states and actions. This is not possible with a poorly trained neural network. In the Atari experiments, a PPO agent is trained to convergence. I would be very interested to see how would this method perform when given blueprint policies that are not so well trained. I am also wondering why there is still a performance gain from online fine tuning given that the PPO agent has already been trained to convergence? Does this imply that the training of the PPO agent is not optimal or perhaps a larger network can be used? Nevertheless, I think requiring a good blueprint policy to be fine tuned over should be a major limitation of this method. 

My third concern is that the method has only been tested and compared on Hanabi and one Atari game. More experimental results from other domains, for example other Atari games, will make the results more convincing.
review_title: null

Meta-review: The paper proposes to replace tabular MCTS-like search methods with finetuning a pretrained policy for the states encountered online. The the high-level method is a good contribution despite being straightforward. The experiments are fairly limited though, for what is largely an empirical paper. Even with the addition of a few more Atari games as a result of the post-rebuttal discussion, the evaluation is weak by the standards of empirical RL papers. I'd expect at least 20-30 more Atari games, or the Atari games on which the authors already have the results and the Procgen suite on top of that. I'm not discounting Hanabi -- it's a hard benchmark, but it's still just one benchmark. Running experiments on more games/environments is especially important since, according to the additional results mentioned in the discussion, in low-data and weak-blueprint regimes RL-Fine-Tune's advantage over MCTS is far from apparent.

On balance, I cautiously recommend acceptance since RL-Fine-Tune seems novel and a strong baseline, and due to its simplicity it's likely to be experimented with and built on by others.
Decision: Accept (Poster)