Scalable Online Planning via Reinforcement Learning Fine-Tuning
Authors: Arnaud Fickinger
Hengyuan Hu
Brandon Amos
Stuart Russell
Noam Brown
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Search
Neural Network Fine-Tuning
Hanabi
Atari

Abstract: Lookahead search has been a critical component of recent AI successes, such as in the games of chess, go, and poker. However, the search methods used in these games, and in many other settings, are tabular. Tabular search methods do not scale well with the size of the search space, and this problem is exacerbated by stochasticity and partial observability. In this work we replace tabular search with online model-based fine-tuning of a policy neural network via reinforcement learning, and show that this approach outperforms state-of-the-art search algorithms in benchmark settings. In particular, we use our search algorithm to achieve a new state-of-the-art result in self-play Hanabi, and show the generality of our algorithm by also showing that it outperforms tabular search in the Atari game Ms. Pacman.

Scalable Online Planning via Reinforcement Learning Fine-Tuning
Arnaud Fickinger" Facebook AI Research arnaudfi www.iirofninnr.c.mmmom com
Hengyuan Hu' Facebook Al Research hengyuan@fb. com
Brandon Amos Facebook Al Research bda@fb com
Stuart Russell
Noam Brown
UC Berkeley
Facebook AI Research noambrown@fb.cm com
russell8berkeley.com
Abstract
Lookahead search has been critical component of recent successes, such as in the games of chess, go, and poker. However, the search methods used in these games, and in many other settings, are tabular. Tabular search methods do not scale well with the size of the search space, and this problem is exacerbated by stochasticity and partial observability. In this work we replace tabular search with online model-basee based fine-tuning of policy neural network via reinforcement learn- ing, and show that this approach outperforms state-of-thee benchmark settings. In particular, state-of-the-arr result in self-play Hanabi, and show the generality of our algorithm by also showing that it outperforms tabular search in the Atari game Ms. Pacman.
search algorithms use search algorithm achieve new
1 Introduction
Lookahead search has been key component successful AI systems in sequential decision-making problems. For example, in order achieve superhuman performance in go, chess and shogi, Alp- haZero leveraged Monte Carlo tree search (MCTS) [B8], MuZero extended this even further to Atari games, again using MCTS [32]] Without MCTS, AlphaZero performs below top human level, and more generally no superhuman Go bot has yet been developed that does not use some form of MCTS. Similarly, search algorithms
critical component of AI successes in backgammon [[5], chess [[τС], poker (27, 7 8), and Hanabi [22]]. However, even though different search algorithms were used in each domain, all of them were tabular search algorithms, i.e., distinct policy was computed for each state encountered during search, without any function approximation to generalize between similar states.
While tabular search has achieved great success, particularly in perfect-information deterministic environments, its applicability is clearly limited. For example, in the popular partially observ- able stochastic AI benchmark game Hanabi [S], one-step lookahead search involves a search over about 500 possible next states. However, searching search over 2020 states, which clearly intractable for tabular search. Additionally, unlike perfect- information deterministic games where it is only necessary search over tiny fraction of the next several moves, partial observability and stochasticity make impossible to limit the search to a tiny subset of all possible states. Fortunately, many of these states are extremely similar, so search algorithm can in theory benefit by generalizing between similar states. This is the motivation for our non-tabular search algorithm.
all two-step joint policies would require
Equaa Contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper we take inspiration from related research in continuous control environments that use non-tabular planning algorithms to improve performance at inference time [ 25, M.. These meth- ods leverage finite-horizon model-based rollouts. Specifically, we replace tabular search with fine- tuning of the policy network at inference time. We show that with this approach we are able to achieve state-of-the-art performance in Hanabi. Specifically, our method is able to search multiple moves ahead and discover joint deviations, which in general is intractable using tabular search. We also show the generality of our approach by showing that it outperforms Monte Carlo tree search in deterministic and stochastic versions of the Atari game Ms. Pacman.
Background
2.1 MDPs, POMDPs, and Dec-POMDPss
We consider Markov decision processes (MDPs) [42], where the agent observes the state 8 E S at time i, performs action according their policy a (T(s)) and receive reward ri r(st, 0t). The environment transit to the next state following transition probability $141 ~ P(s
POMDPs [3 extend MDPs to the partially observable setting where the agent cannot observe the true underlying state but instead receive information through a observation function or = ()())). Due to the partial observability, the agent policy π often needs to take into account the entire action- observation history (AOH) denoted as oy¹ = foo, ao. oz) with тę = {5o, ao. To 8, repre- senting the true underlying trajectory.
Dec-POMDPs [ extend POMDPs the cooperative multi-agent setting. At each time step f, each agent i receives their individual observation from the state of [2' (st) and selects ac- tion a². The joint action of N player is a tuple a (a), a.. a,²² and can be observed by all players. We denote the individual policy for each player as π. and joint policy as т. We use T!
{of: ao, ro,
o,} to represent the AOH from the perspective of agent
2.2 Tabular Search in Dec-POMDPs
SPARTA [ is tabular search algorithm previously applied in Dec-POMDPs that is proven to never hurt performance and in practice greatly improves performance. Specifically, SPARTA as- sumes that all agents in game agree on a joint blueprint policy Top and that one or more agents may choose deviate from Thp at test time. We assume a two player setting the following discussion. There are two versions of SPARTA.
In single-agent SPARTA, only one agent may deviate from the blueprint. We denote the search agent as agent i and the other agent as -i. Agent maintains their private belief over the trajectories they are in given their AOH, (T²) P( (z))) At action selection time, agent compute the Monte Carlo estimation for the value of each action assuming both players will follow blueprint policy Thp thereafter,
Q(Ti, az) = (erp-BBinn.rr---Prrrraaa.. Re (T(T)),
(I)
where R₂(TT) = IT Fr. is the forward looking return from f to termination step T. Agent i will pick arg max,, a, Q(T ar) instead of the blueprint action abp if max., Q(Ti 0.) Q(= abp) > c.
In multi-agent SPARTA, both agents are allowed to search at test time. However, since agent i's belief over states depends on agent -''s policy, it is essential that agent replicate the search policy of agent -i. Since agent i does not know the private state of agent -i, agent must therefore compute agent -i's policy for every possible AOH 7; agent may have seen, process referred to as range-search. Once a correct belief is computed. agent -i runs their own search. The range- search can be prohibitively expensive to compute in many cases because players might have millions of potential AOH. For example, Hanabi might have up to about 10 million AOHs. To mitigate this problem, the authors introduce way to execute multi-agent search only when the range of AOHs is small enough that it is computationally feasible to do so, and the algorithm uses single-agent search otherwise. Even so, SPARTA multi-agent search limited to one-ply search - each agent calculates the expected value of only the next action. The computational complexity grows exponentially with each additional ply. In contrast, our non-tabular multi-agent search technique makes possible to conduct 2-ply, and deeper, search.

---

2.3 Reinforcement Learning
Our goal is to learn stationary policy {(as)) such that the expected non-discounted return ((T) E. Bs)= Bn Et=o r(st, 04) is maximized, where p(s) is the initial state distribution. For any policy π we define its value (*+) = E, (i== r(s+41, (()))s))))))) its Q-value (* at) = En (2 7(5+41, (x+1)) ( st. (t) and its advantage (st,,, ar) (st, at) - (se).. In reinforce- unknown, and the policy is
ment learning (RL), the transition probability p and the reward learned by sampling trajectories 7 = ((so.au), ($r, ar)) from the environment.
the sampled trajectories to learn the Q-function of the optimal policy, given by the Bellman equation: Q(s, a) = (r(s, a) + y max,' Q(s', a')). The learned policy is greedy with respect to Q: =[als) arg max, Q(s, a). In DQN, the Q-value is approximated with neural network Qo that is trained by taking gradients of the mean squared Bellman error:
In Q-learning,
ValE(Qu (r(s,a) max Q (s',
where 0' is a fixed copy of 9 and É denotes the empirical average over a finite number of batches.
In policy gradient (PG), we directly learn parametrized policy by performing gradient descent using an estimator of J:
(3)
where is an estimate of the advantage of xy-
2.4 Decision-Time Planning
To make better decisions at test time, we can specialize the learned policy, which we refer to as the blueprint policy (also known as the prior policy), to the current state by using as prior an online planning algorithm.
One of the most popular planning algorithms is Monte Carlo tree search (MCTS), which has famously enabled superhuman performance in go, chess, and many other settings [3G B7, BX, [2]]. MCTS builds tree of potential future states starting from the current state. Each node keeps track of its action-state visitation counts N(s, a) and an estimate of its Q-values (ss, a). refined each time the node is visited. The prior policy is used to select the next action take during the tree traversal.
MCTS don't scale well in environments with large branching factors caused by a high amount of stochasticity or partial observability. For example, MCTS becomes very expensive the size of the action space increases because it needs to build an explicit tree to keep statistics about state-action pairs, and nodes need to be visited several times for the method be effective. Furthermore, MCTS can in the worst case expand very deeply a branch that it misidentifies to be optimal due to lack of exploration exacerbated in high dimensions- which leads worst-case sample complexity worse than uniform sampling [28].
3 Reinforcement Learning Fine-Tuie Tin-TTning
To alleviate the inefficiency of tabular search methods like MCTS in large state and action spaces, we propose to formulate online planning as a RL. problem that should be quickly solvable given a sufficiently optimal blueprint policy. We do by using RL as a multiple-step policy improvement operator [12]]. More specifically, we bias the initial state distribution towards the current state s* and we reduce the horizon of the problem, leading the following truncated objective:
max (6) Eno. 1. 1.81:8 7 *
Σ (r(st, az)) + Eg oo 88 1:00 1=0
Σ (r(s,, a:))) t=
(4)
where Pa- (s) 1 (s s*) and π is the blueprint policy. If is optimal, then n' is solution to the problem. We call this procedure RL Fine-Tuning or RL Search and we use the two terms interchangeably in the remaining.
The blueprint w is either directly parameterized by 0 or greedy with respect to a Q-value param- eterized by 8. In both cases, improve for the next H steps by following the gradient of the
3

---

truncated objective with respect to A. We present the two approaches in more detail in the following subsections.
3.1 Policy Fine-Tuniee Fine-Tuning
In policy fine-tuning, we any actor-critic algorithm train blueprint policy and value network. For our experiments in the Atari environment, we use PPO [39]] At action selection time, to solve objective (l) we use the blueprint policy to initialize the online policy and the blueprint value to truncate our objective:
max 8
B (s) Buo.
Σ (r(st, + Vo((H))
(5)
If Vi is perfect estimator of V. then exactly optimize the truncated objective
The fine-tuned policy za is obtained by performing N gradient steps with the PPO objective. The policy improvement step via policy gradient is presented in Algorithm M
Algorithm 1: Policy Gradient Improvement. We use a standard PPO update to fine-tune the blueprint policy Tyy and value Va from some state 8t.
Input :current state 8t, number of updates N, global policy parameter 0, global value parameter Φ, horizon H, number of rollouts M
parameter θ
Init:
80 0, 00 σ
for i +- to N do
Collect M trajectories of H time steps starting from 8y using T8, Compute the generalized advantage estimate using:
5. = ri + 7V (81+1) - Vo, (8,)7l E [0, H -
(6)
- Vo.,(
(0r, 0.) + (1)
return ON
3.2
Q-value Fine- Tine-Tuning
In Q-value fine-tuning, we train a Q-network using any offline RL algorithm. At action selection time, to solve objective ( use the blueprint Q-value to truncate objective:
H-1 (r(s), az)) + max Qo(sH,
max Boo. 1. Σ
The online policy स is greedy with respect to the fine-tuned Q-network Q; obtained by performing N gradient steps with the mean squared Bellman error. To alleviate instabilities, the transitions used to fine-tune the Q-network are sampled with probability p from the global buffer replay and with probability p from the trajectories sampled action selection time. The Q-value improvement step is presented in Algorithm 2.
3.3 Scaling Multi-Agent Search in Dec-POMDPs with RL Search
3.3.1 Scaling Single-Agent RL Search
In a test game, agent maintains a belief over possible trajectories given their own AOH, B'(T)) = P(ToTTE). At action selection time, agent performs Q-value fine-tuning (Algorithm D of the

---

V-Vuuee Improvement. We use a standard Bellman residual update to fine-tune the blueprint Q function Qe from some state s,.
Algorithm 2:
Input :currett state St, number of updates N, global Q-network parameter 0, horizon H, number of rollouts M, batch size B uudput: parameter 0
Init:
L 0o 0
Collect M trajectories of H time steps starting from using an e-greedy policy wrt Qa. For each trajectory, if the environment is not terminated, replace r++H-1 with
T + max. Q0($++H, a)
for i f to N do
Sample B transitions with probability from the global buffer and probability 1 - p from the M collected trajectories.
0. Vo.,, ((,,,,QQ a) (r(s,a) +- max, Q0 Qs ,())))
return NN
blueprint policy Thp on the current belief B(T²). This produces new policy x². Next, we eval- uate both and Thp on E trajectories sampled from the belief. If the expected value of nt least higher than the expected value of Thp, the agent plays according to nt for the next H moves and search again at the (H + 1)th move. Otherwise, the agent sticks to playing according to Top for the current move and searches again on their next turn.
3.3.2 Scaling Multi-Agent RL Search
critical limitation of single-agent SPARTA, multi-agent SPARTA, and single-agent RL search that the searching agent assumes the other agent will follow the blueprint policy on all future turns. It is thus impossible for those methods to find joint deviations (o azz"") where it is beneficial for agent i deviate to action at* if and only if the other agent -i deviates to a,+1 an action that will not be selected under blueprint. Searching for joint deviation with general tabular methods is computationally infeasible in large Dec-POMDPs such Hanabi because the effective branching factor is about 20 in Hanabi.
RL search enables searching for the next H moves for all agents jointly conditioning on the common knowledge. The observations in a Dec-POMDP can be factorized into private observation and public observations pub For example, in Hanabi the private observations are the teammates hands and the public observation are the played cards, discarded cards, hints given, and previous actions. The public observation is common knowledge among all players. We can define the common knowledge public belief as (p Quud is the public observation function. We then draw policy is obtained via Q-value fine-tuning, in the same way as in single-agent RL search. Since the search procedure uses only public information, it can in principle be replicated by every player independently. However, to simplify the engineering challenges of our research, we conduct search just once and share the solution with all the players. If the newly trained policy better than the blueprint by at least every player will use it for their next H moves.
{Tr (2 (Tt) = rrud) where T.Pu {oo 01 ri,...,,of ooab) } and 1Boub(?? repeatedly. The new
4 Experiments
4.1 Hanabi Experiments
Hanabi is a 2-5 player partially observable fully cooperative card game and a popular Dec-POMDP benchmark. A detailed description of the rules of the game and an explanation of its challenges can be found in [S]]. The deck in Hanabi consists of 5 color suites and each suite contains 10 cards with three 1s, two 2s, two 3s, two 4s and one 5. At the beginning
the game each player draws or cards. The main twist of the game is that players cannot see their own hand but instead observe all other players' hands. All players work together to play cards from to in order for each suite. At each turn, the active player may choose to give a hint to another player at the cost of hint token.
5

---

However, the players start with only 8 available hint token, and must discard a card in order to gain a new hint token.
In recent years, there has been tremendous progress in developing increasingly strong agents for Hanabi [5], I.3, (22, 18. 19, 20] to the point where the latest agents are approachingn near-perfect scores, As a result, it is difficult to distinguish performance differences between different techniques. For this reason, we additionally evaluate our technique on harder version of Hanabi in which the maximum number of hint tokens is 2 rather than 8.
We test both single-agen ageet RL search and multi-agent joint RL search in 2-player Hanabi. Blueprint policies are trained with Q-learning and the same learning method is used for test-time fine-tunn tuning. We compare methods against SPARTA [22]]. the state-of-the-at search technique that has been applied in this domain.
Blueprint Training. We train blueprint policy using independent Q-learning with distributed learning setting. The entire process consists of two modules that run simultaneously. The first is simulation module that runs M copies of the Hanabi game distributed across N parallel running workers. The simulation module collects AOH TT at the end of each game and writes those data into a centralized prioritized replay buffer [ET, I7]. Meanwhile, training module samples from the replay buffer to train the policy and send a new copy of the policy to the simulation module every K gradient steps. We set M 6400, N 80 and K = 10, the first two of which chosen to make the replay buffer write speed of the simulation module roughly the same as the replay buffer read speed of the training module. We use the canonical techniques such as double Q-learning [H6] and dueling DQN [48] for better performance. We use the same Public-LSTM network architecture as in [20] to simplify the belief update procedure. For simplicity,
independent Q-learning [[]]] We train the blueprint policy for 2 million gradient steps/batches and each batch contains 128 AOHS Tr Our blueprint gets 24.23 points on average, which comparable to the strongest blueprint (24.29) in [20] which is trained with a value decomposition network.
Results. We present results in Table M1 For single-agen RL search we set the search horizon H = 3, the number of gradient steps G = 5000, the number of evaluations for comparing tine-tuned policy against blueprint E 10, 000 and the deviation threshold 0.05. For multi-agent RL search we set H = 1, G = 10000, E = 10, 000, and e 0.035. For comparison we also run both the single-agent and multi-agent version of SPARTA can be extremely expensive to run depending on the maximum size allowed for range-search, so pick the hyper-parameter to make it consume roughly the same amount of time per game as multi-agent RL search. Each cell in Table is evaluated on 2000 games. The numbers on top shows the average score and standard error of the mean, and the number at the bottom of each cell the percentage of winning games (perfect score games). In normal Hanabi, single-agent RL search beats agent SPARTA and multi-agent RL search beats multi-agent SPARTA. However, the scores are close enough that the differences are difficult to distinguish. This is likely because the blucprint in normal Hanabi is already extremely strong. so the benefit of additional search is diminished. On the harder 2-hint variant, RL search, and especially multi-agent RL search, significantly outperforms SPARTA. This indicates that searching for joint deviations is helpful, and RL search scalable way to find them.
blueprint. Multi-agent SPARTA
Whether or not RL search is preferable SPARTA depends in part on the computational budget. Single-agent SPARTA need only search over about 20 possible actions, so it takes 4 seconds to make a move using CPU cores and 1 GPU. For comparison, single-agent RL search would take 69 seconds per move when searching one move ahead with 20 CPU cores and 2 GPUs. However, RL search scales much more effectively. The branching factor in Hanabi is around 500 and the game involves a high amount of stochasticity due to new cards being drawn, so searching even 3 moves ahead is essentially intractable for single-agent SPARTA. In contrast, single-agent RL, search that looks 3 moves ahead takes only 88 seconds per move. Similarly, searching two moves ahead (one for each player) in multi-agent SPARTA would mean searching over about 20 joint policies, which would be intractable. In contrast, multi-agent RL search is able to do this while taking only 180 seconds per move.

---

SPARTA (Single)
SPARTA (Multi)
RL Search (Single)
RL Search
Variant
Blueprint
(Multi)
24.23 + 0.04
24.57 = 0.03 73.90%
24.61 + 0.02 75.46%
24.59 + 0.02 75.05%
24.62 + 0.03
Normal
63.20%
75.93%
22.99 + 0.04
23.60 + 0.03
23.67 + 0.03
23.61 + 0.03
23.76 + 0.04
2 Hints
17.50%
25.85%
26.87%
27.85%
31.01%
Table 1: Performance on Hanabi. Each cell is averaged over 2000 games. The number in the upper half of
the cell is the average score + standard error of mean (s.e.m.) and the number in the lower half is the percentage of winning games where agents score 25 points.
4.2 Atari
We demonstrate the generality of our approach by comparing policy gradient fine-tuning to MCTS two Atari environments, Ms. Paeman and Space Invaders [B]] Specifically, we aim answer the following questions:
1. Does RL. Fine-Tuning outperforms MCTS in terms of search time and sample complexity? Yes, RL Fine- Fine-Tuning obtains higher scores in Macman than MCTS for smaller search time budget and smaller number of samples per step.
2. Does RL Fine- Tuning performs well even with weak blueprint? Yes, RL Tunnnn obtains strong results with a weak blueprint in Ms. Paeman and improve the policy much more sample-efiie eeficceefi way than carrying the PPO training of the weak policy.
3. Are the optimal hyperameters robust across different environments? Yes, an ablation study the search horizon hyperparameter reveals that the optimal search horizon is the same for Ms. Pacman and Space Invaders.
4.2.1 Implementation
Blueprint Training. We train PPO [G_]] agent until convergence in Ms. Pacman and Space In- vaders. In both cases 10" samples are necessary to converge to the optimal PPO blueprint. We also save weak blueprint after 2. samples in Psmman to answer question E1 The weak blueprint obtains a score that is times smaller than the optimal PPO policy.
MCTS. At every testing time step, we build tree starting from the current state. We use the blueprint policy to guide the action selection during the tree traversal and we use the value network every time we reach a node never seen before or we reach the depth limit of the tree. In our ex- periments we use depth limit of 100. We can significantly improve the performance MCTS by allowing for an additional hyperparameter to balance policy prior and visitation counts in the second term:
VL. N(s,a')
arg max Q(s, a) + To(a))))
(8)
1+ ((,
We obtain the best performance with c = 5 and 3 = 0.1.
Policy Gradient Fine- TaninTnnngg Our method achieves a small average time budget by amortizing the search time across multiple steps. In Ms. Paeman, we solve finite-horizon problem of horizon 30 and we only need to replan every 30 steps (Algorithm D) We have also tried to amortize MCTS across multiple steps, where we update the tree only after 30 steps. this setting however, the episode return is worse than what is achieved by the blueprint, emphasizing the need to replan at every timestep, which not necessary when performing RL search. To optimize an infinite-horizon problem rather than finit-hhoizonn problem, for the last step of each trajectory. The problem still simplified due to the biased initial state distribution. In Ms. Pacman, we have observed that this setting leads to similar improvements.
the refined value instead of the blueprint
4.2.2 Results
RL Fine-Tuning outperforms MCTS for fixed search time budget. Both MCTS and policy gradient fine-tuning are multi-step improvement operators that optimize objective (8) using value
7

---

(Deterministo Enstronment) Riw-Tartng
MOTS
MCTS
8000
8000
8000
4000
8000
Avarage search
Samples per stop
10
200
300
400
500
(a) Time Budget
(b) Samples
Figure I: MCTS vs RL Fine-Tuning (a)When the average time budget is on the order of 1-10 seconds, RL Fine- Tuning consistently outperforms MCTS. (b)RL Fine- Tuning also outperforms MCTS in terms of sample efficiency. The shaded area represent the min/max range across 5 seeds. The curves are smoothed with an exponential moving average.
Additional Samples
3.10$
4.10$
8.10$
RL Fine-TTuning
1880
3940
4580
5510
PPO Training
1880
1900
1900
1920
Table Performance on Ms. Pacman with weak blueprint. It is more sample efficient to use RL Fine- Tuning to improve a weak blucprint rather than carrying on the PPO training.
estimate to truncate the objective. Therefore we expect both methods to achieve the same asymptotic performance. We compare both methods with a finite time budget of the order of 10s. Figure WI shows the return achieved by the agent when performing either MCTS RL search at action selection time, versus the average search time budget. We see that RL search consistently outperforms MCTS, contrasting with recent work showing that policy gradient
than MCTS for planning in the game Hex [G]] The difference of performance with this previous work might be due to the fact that using PPO.
they using vanilla PG while
RL Fine-Tumine Tani-TTnig is more sample efficient than MCTS. With RL search, we need an average of 621 samples and 1.2 seconds per step to achieve a return of 8080 which is more than 2 times the return achieved by our asymptotic PPO policy. The total number of additional samples needed 502,000, which is less than 5% of the samples needed for the blueprint PPO policy to converge. In contrast, MCTS requires an average of 4489 samples per step to reach a cumulative reward of 5820.
RL Vine-Tuning obtains strong results even with a weak blueprint. We run experiments in the Ms. Pacman environment using poorer blueprints and compare the average cumulative reward obtained by continuing the PPO training versus performing RL Fine-Tuning for the same number of additional samples. For a blueprint trained during 2000 epochs 1024 samples (around 1/5 of convergence and obtaining an average cumulative reward of 1880), RL fine-tuning can reach an average cumulative reward of 5510 with an online search using on average 1145 samples per step. In contrast, continuing the PPO training of the blueprint using the same number of additional samples used by RL fine-tuning yields a policy that reach an average cumulative reward of 1920 only (see table 2). We also test randomly initialized blueprint: RL fine-tuning can reach an average cumulative reward of 2730 with an online search using on average 1360 samples per step while the PPO training of this blueprint with the same number of additional samples yields an average cumulative reward 1280 only (see table B)
The hyperparamters of RL Fine- Hoie-TTning are robust across different environments. After per- forming the same ablation study on the horizon both Ms. Pacman and Space Invaders, we have found that the optimal horizon is 32 for both environments. Thus there is reason to think that this value is nearly optimal several other Atari games and readers willing to apply our method to other Atari games should start experimenting with this value.

---

Additional Samples
2.10$
4.10$
8.105
RL Fine-Tuning
60
1180
1800
2730
PPO Training
60
689
732
1280
Table 3: Performance on Ms. Paeman with random blueprint. RI. Fine-Tuning also outperforms PPO in term of sample efficiency when the blueprint is randomly initialized.
5 Related Work
Fine-tuning in supervised learning. Instead of searching for the optimal action sequence in the action space, which is what MCTS does, we are effectively searching in the parameter space of the neural network. We do that by following the gradient of the truncated objective. Since the truncated objective and the biased data distribution are close to the objective and the data distribution the blueprint neural network was trained on, the performance our method should reflect the success of neural network fine-tuning in supervised learning [43] GO].
Fine-tuning in reinforcement learning. Replacing action search with neural network parameter search is reminiscent of recent work in continuous control performing cross-entropy method in the parameter space of the neural network [_7]. They argue that the improvement over searching in the action space is due to the smoothness of the neural network loss landscape [[a]]. We take this idea further by making full use of the parameter space. While they are using scalar information the return of each rollout we use richer information - the gradient of the return with respect to 6. By doing so, we argue that we lose less information about the blueprint than other online planning methods, which enables us to stay efficient in high dimensions.
Policy gradient fine-tuning is related recent work iterative amortized policy optimization [AA]]] Considering policy optimization in the lens of variational inference [23], they argue that amortizing the policy into a neural network leads to the same suboptimal gap than amortizing the posterior in VAES and propose to inject gradient information into the policy network's 's input at action selection time to close the gap. While their policy network meta-learns how to use the gradient, we use fixed update rule. Meta-lcarning the update rule is promising future direction that could be stacked on top of our method. However, we suspect learning the update rule at blueprint training time to be intractable in complex environments such as Hanabi.
A large range of previous work has proposed combine RL with tabular search [АЛ] In Expert Iteration [2, 52], a policy network is used to amortize an expert tree search algorithm using imitation learning. Rather than amortizing the traversal policy, Hamrick et al. propose to train a Q-network via model-free RL to amortize the Q-values returned by MCTS [[G]]. Our method differs from this line of work because we don't 't use any tree search for maximum scalability.
Finally, there has been previous work investigating some variants of RL search. Temporal-differene search was introduced in [BS] and applied Computer Go. Policy gradient search [B] performs vanilla policy gradient to fine-tune a rollout policy and selects the action by performing MC rollouts with this policy. They perform worse than MCTS in the game Hex. Springenberg al. [200]] compare a variant of policy gradient fine-tuning to tree search in continuous control environments and show that PG fine-tuning performs almost as well tree search, but not equally well. By contrast to these rather negative results, our positive results show that RL search outperforms MCTS in Ms. Pacman and achieves state-of-the-arr performance in Hanabi. Additionally, prior work has looked to generalize between states during MCTS, though without fine-tuning of a pre-trained network [[9]]
Discussion
Despite the incredible success of neural network function approximation in reinforcement learning environments [2G,, IT5. [l], search algorithms have largely remained tabular, as can seen for exam- ple in AI algorithms for go [36 37 38), poker [27, 7 8, B), and Atari [32) 33, 2TI]. Tremendous research has gone into adding heuristics that enables tabular search to perform better in these do- mains, especially in
deterministic environments [[]] [T]] However, argue

---

that in order to scale to larger, more realistic environments, especially those involving stochasticity and partial observability, is necessary to move past purely tabular search algorithms.
The jump from tabular search to RL search comes at a steep upfront cost. However, we show in this paper that in sufficiently complex environments, including the established Al benchmark Hanabi, RL search enables state-of-the-art performance and scales much more effectively. We show this also holds in the Atari Ms. Paeman environment, and we expect it to be true wide variety of other domains as well. As computational costs continue to decrease, it is likely that the relative perfor- mance of RL search will improve even further. To quote Richard Sutton's The Bitter Lesson [[]]], "One thing that should be learned from the bitter lesson is the great power of general purpose meth- ods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning."
While a lot of methods combining RL with tabular search have been developed, we have shown that we can achieve better results by completely amortizing online planning into a neural network via RL fine-tuning, Most importantly, we have shown that RL search can effectively search complex domains where tabular search intractable. This enables us achieve state-of-the-arr results in the Al benchmark Hanabi, which features both stochasticity and partial observability, both of which are prevalent in the real world.
While we achieve good results in discrete environments, our algorithms are currently focused on discrete environments. promising direction for future work would be investiga- tion of similar techniques in continuous control domains and a comparison to existing techniques (mentioned in the main text) in those domains. A potential limitation of our method compared to gradient-free free methods is that we might be constrained by the local minimum of a too weak blueprint. In other words, we may be unable to discover drastically new solutions to the control problem that an optimal global method solving the problem from scratch would find.
Limitations:
Broader impact: RL search is decision-time planning method aimed at improving search in very high-dimensiona. spaces. There has been a rich history of previous work in this area, where the goal has been to develop more efficient search methods. This work is in line with previous work and should not directly cause broader harm to society. We are not using any dataset or tool that will perpetuate enforce bias. Nor does method make judgements on our behalf that could be misaligned with our values.
References
[1] B. Amos and D. Yarats. The differentiable cross-entropy method. In International Conference on Machine Learning, pages 291-302. PMLR, 2020.
[2] T. Anthony, Z. Tian, and D. Barber. Thinking fast and slow with deep learning and tree search. arXiv preprini arXiv: 1ao:::005.033 2017.
[3] T. Anthony, Nishihara, Moritz, T. Salimans, and J. Schulman. Policy gradient search: Online planning and expert iteration without search trees. arXiv preprint arXiv:1904.032. 036466. 2019.
[4] A. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[5] N. Bard, J. N. Foerster, S. Chandar, N. Burch, M. Lanetot, H. F. Song, E. Parisotto, Du- moulin, S. Moitra, E. Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020.
[6] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[7] N. Brown and T. Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, page ca30 2017.
[8] N. Brown and T.. Sandholm. Superhuman AI for multiplayer poker. Science, page eaay2400, 2019.
10

---

[9] N. Brown, A. Bakhtin, A. Lerer, and Q. Gong. Combining deep reinforcement learning and search for imperfect-infoomation games. arXiv preprint arXiv:2007.33544, 2020.
[10] M. Campbell, A. J. Hoane Jr, and F.-h. Hsu. Deep Blue. Artificial intelligence, 134(1-2):57-83, 2002.
[11] Cowling, E. J. Powley, and D. Whitehouse. Information set monte carlo tree search. IEEE Transactions on Computational Intelligence and Al in Games, 4(2):120-1433 2012.
[12] Y. Efroni, G. Dalal, B. Scherrer, and S. Mannor. Multiple-ste greedy policies in approximate and online reinforcement learning. In NeurlPS, 2018.
[13] J. Foerster, F. Song. E. Hughes, N. Burch, I. Dunning, S. Whiteson, M. Botvinick, and M. Bowl- ing. Bayesian action decoder for deep multi multi-agent reinforcement learning. In International Conference Machine Leurning, pages 1942-1951, 2019.
[14] J.-B. Grill, F. Altché, Y. Tang. T. Hubert, M. Valko, I. Antonoglou, and R. Munos. Monte- carlo tree search as regularized policy optimization. In International Conference on Machine Learning, pages 3769-3778. PMLR, 2020.
[15] Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
[16] B. Hamrick, V. Bapst, A. Sanchez-Gonzale,, T. Pfaff, T. Weber, Buesing, and P. W. Battaglia. Combining q-learning and search with amortized value estimates. arXiv preprint 1rXiv:1912.028077 2019.
[17] D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver. Distributed prioritized experience replay. In 6th International Conference on Learning Repre- sentations, ICLR 2018, Vancouver; BC, Canada, April 30 May 3, 2018, Conference Track Pro- ceedings. OpenReview.net, 2018. URL https://openreviiee
net/ httt://orrmmdddHiyy-----
[18] H. Hu and N. Foerster. Simplified action decoder for deep aulti--gent reinforcement learn- ing. In International Conference on Learning Representations, 2019.
[19] H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster. other-play for zero-shot coordination. In International Conference Machine Learning, pages 4399-44111 4410. PMLR, 2020.
[20] H. Hu, A. Lerer, N. Brown, and J. N. Foerster. Learned belief search: Efficiently improving
policies in partially observable settings, 2021. URL http::://ooeeeeree...ttttt////// wttp://wwwwiettttooo//o..eoc.add
XP37gRVKa_D U.
[21] T. Hubert, J. Schrittwieser, I. Antonoglou, M. Barekatain, S. Schmitt, and D. Silver. Learning and planning in complex action spaces. arXiv preprint arXiv:2104.063002 2021.
[22] A. Lerer, H. Hu, J. N. Foerster, and N. Brown. Improving policies via search in cooperative partially observable games. In The Thirty-Fourts AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7187-7194. AAAI Press, 2020. URL Attps://aaa/: http:/aaaa.rr/aaaaa//////// stt////s/ind/eee/// phtpddee..pp//eeee /ttp///AAAA/rrrccc///////////
[23] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint
/80. 2018.
[24] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets. arXiv preprint arXiv: 1712.09913 2017.
[25] J. Marino, A. Piché, A. D. Ialongo, and Y. Yue. Iterative amortized policy optimization. arXiv preprint arXiv:2010. 10670, 2020.
[26] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 51 518(7540):529-533 2015.
[27] M. Moravčík, M. Schmid, N. Burch, V. Lisý, D. Morrill, N. Bard, T. Davis, K. Waugh, M. Jo- hanson, and M. Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit
poker. Science, 356(6337):508-513 2017.
[28] R. Munos. From bandits to monte-carlo tree search: The optimistic principle applied to opti- mization and planning. 2014.
11

---

[29] F. A. Olichoek, M. T. Spaan, and N. Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
[30] M. Oquab, L. Bottou, Laptev, and Sivic. Learning and transferring mid-level image rep- resentations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1717-1724, 2014.
[31] Schaul, Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:
//arxiv. org/ abs///b///// 1511. 15912
[32] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. arXiv preprint arXiv:1911.0822.
2019.
[33] J. Schrittwieser, T. Hubert, A. Mandhane, M. Barekatain, I. Antonoglou, and D. Silver. On- line and offline reinforcement learning by planning with a learned model. arXiv preprint arXiv:2104.06294 2021.
[34] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1oXiv:1707.0634/ 2017.
[35] D. Silver. Reinforcement learning and iimulation-basel search in computer go. 2009.
[36] D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifrre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484, 2016.
[37] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
[38] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419) 140-1144, 2018.
[39] E. J. Sondik. The optimal control of partially observable markov decision processes. PhD) the sis, Stanford University, 1971.
[40] T. Springenberg, N. Heess, D. Mankowitz, Merel, A. Byravan, A. Abdolmaleki, J. Kay,
J. Degrave, J. Schrittwieser, Y. Tassa, et al. Local search for policy iteration in continuous control. arXiv preprint 100.:20000005555, 2020.
[41] R. Sutton. The bitter lesson. http://wwy wttp://ccoopleess.aaaeeeeeeeeeea
BitterLesson html. Accessed: 2021-05-27.
[42] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[43] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, A survey on deep transfer learning. In International conference on artificial neural networks, pages 270-279. Springer, 2018.
[44] M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[45] G. Tesauro. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215-219, 1994.
[46] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In D. Schuurmans and M. P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2094-2100. AAAI Press, 2016. URL http://www aaai wwwpoo//oo///oooooooooeeoooooooo indexxndexxx/// http/AA/AAAAAA///////////
viex/12389.
[47] T. Wang and J. Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv: 1906.08049, 2019.
[48] Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas. Dueling net- work architectures for deep reinforcement learning. In M. Balcan and K. Q. Weinberger, ed- itors, Proceedings of the 33nd International Conference on Machine Learning, ICMI. 2016, New York City, NY, USA. June 19-24, 2016, volume 48 of JMLR Workshop and Conference
12

---

Proceedings. pages 1.8/sangff6.htm.
JMLR.org, 2016. URL htsp://proceedding. mlr press/
[49] T. Yee, V. Lisý, and M. H. Bowling. Monte carlo tree search in continuous action spaces with execution uncertainty. In I/CAI, pages 690-697, 2016.
13


Reviews:
[{"review_id": "_j09v401zGw", "reviewer": "Reviewer_Ypiw", "summary": "This paper proposes to replace tabular search methods like MCTS with fine tuning of neural policies for online planning. \n\nThe motivation is that fine tuning the neural policies online may be more efficient than doing MCTS which has no generalization between states during the search. \n\nThe key algorithmic contribution is the biased RL objective used for more efficient online fine tuning of the neural policy. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: I find the idea of replacing MCTS with online fine tuning of neural policies interesting. However, I don't think overall the paper has enough algorithmic contributions. \n\nQuality: The method seems correct. Limitations see below. \n\nClarity: The paper is easy to follow. \n\nSignificance: As mentioned, I think this is an interesting direction. Also, I think the general problem setup of this paper, which is how to make good online decisions with a blueprint policy and a model important to study. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My main concern is that this paper does not have enough algorithmic contributions.\n\nMy second concern is that this method may only work well when the given blueprint policy is well trained. This is because the key potential advantage of this method over tabular search methods is the effective generalization between states and actions. This is not possible with a poorly trained neural network. In the Atari experiments, a PPO agent is trained to convergence. I would be very interested to see how would this method perform when given blueprint policies that are not so well trained. I am also wondering why there is still a performance gain from online fine tuning given that the PPO agent has already been trained to convergence? Does this imply that the training of the PPO agent is not optimal or perhaps a larger network can be used? Nevertheless, I think requiring a good blueprint policy to be fine tuned over should be a major limitation of this method. \n\nMy third concern is that the method has only been tested and compared on Hanabi and one Atari game. More experimental results from other domains, for example other Atari games, will make the results more convincing.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "V-so1enAnLl", "reviewer": "Reviewer_2XCZ", "summary": "This paper proposes to replace a Monte-Carlo tree search style search method by instead fine-tuning a neural network by running reinforcement learning on trajectories generated by a perfect model of the environment starting from the current state at inference time. They apply this idea in a cooperative multi-agent planning in the game Hanabi, as well as in single-agent planning in the game Ms. Pacman where the agent has is given access to the simulator. Experimental results suggest that the fine-tuning procedure is able to perform similarly to or better than a strong tabular search method (SPARTA) in Hanabi. In Atari, they demonstrate that the procedure seems to outperform MCTS, particularly as the total search budget is made larger for each algorithm.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I feel the approach considered in this work is quite interesting and important. It makes sense that ultimately we will want to replace MCTS like procedures with something that generalizes more naturally during the search procedure itself. The experiments presented here seem like a good proof of concept for such methods and the fact that they include results for both a single agent game and a multi agent benchmark is a plus.\n\nMy main uncertainty is that the experimental details included are a bit lacking (though at least the code is made available). In particular, see 'Questions about Experimental Details' heading below.\n\nAlso, the paper could be made more self-contained in terms of ensuring that all the variables that are used are clearly defined (see Minor Comments and suggestions).\n\nI also am not very knowledgeable of the relevant multi-agent search literature to evaluate the multi-agent portion of the work. But from what I can tell it seems generally well done.\n\nQuestions about Experimental Details\n====================================\n\nSome details of the experimental methodology such as various hyper-parameters and how they were selected seem to be missing:\n\nLine 131: 'the transitions used to fine-tune the Q-network are sampled with probability p from the global buffer replay' what is this p and how is it chosen? Also, why is the global replay buffer and how is it generated?\n\nLine 236: 'We obtain the best performance with c = 5 and $\\beta$ = 0.1' what other values were searched over here?\n\nWhat hyper-parameters were used for the blueprint Q-learning and PPO agents in Hanabi and Ms. Pacman respectively and how were they chosen?\n\nMinor Comments and Suggestions\n==============================\n\nLine 33: 'non-tabular planning algorithms improve performance'->'non-tabular planning algorithms to improve performance'\n\nBetween Line 59-60 (line numbers missing here for some reason): is $\\Omega^i(\\tau_0)$ defined somewhere?\n\nLine 137: Is $B(\\tau_t^i)$ explicitly defined somewhere? From looking at the cited SPARTA paper it seems like this should maybe be $B^i(\\tau_t)$ if the notation is supposed to be consistent with that work.\n\nAlgorithm 1: I think t is kind of overloaded here in that $s_t$ is the state at the current time but $\\delta_t$ is $\\forall t\\in[0,H-2]$ using different variable names would make this clearer.\n\nLine 260: 'like MCTS do'->'like MCTS does'\n\nFor the Ms. Pacman results, it would be interesting to see how MCTS and RL-fine-Tuning compare in terms of total number of states expanded in addition to average search time.\n\nUpdate\n======\nI thank the reviewers for their clarifications. My empirical concerns are addressed and I will maintain my score. The additional experiments are also welcome.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are reasonably well covered. I don't see much potential for negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "NPUD-qCEhF", "reviewer": "Reviewer_VMv6", "summary": "This paper proposes a new planning method, aiming at improving performance in a large and realistic environment. The proposed method performs as a planning module and can be integrated with different types of online learning agents. Different from tabular search algorithms, the method formulates the planning step as a reinforcement learning problem and updates the policy or action-value function by bootstrapping. Thus, the method reduces the horizon of search and gains learning efficiency in large search spaces and continuous control problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method improves search efficiency and is applicable in large search spaces. However, the discussion on how key parameters affect the performance is missing. It is also not clear if the hyperparameter setting in the submitted codebase is the same as the setting used for generating the result.\n\nMore details are provided below.\n\nThe paper discusses different cases separately in detail, including policy gradient methods and action value methods, as well as single-agent and multi-agent settings. This suggests the method can be integrated with different types of reinforcement learning agents. \n\nThe reported performance in Table 1 is averaged over 2000 games, which gives a reliable averaged value. The experiment reported in Figure 1 tests 5 seeds. Testing more seeds would be better, though 5 is acceptable.\n\nThe idea of truncating the target is similar to n-step bootstrapping (Chapter7 in [1]). I understand this paper focuses on planning instead of value function learning, though adding references about n-step bootstrapping might help the reader to understand the main idea.\n\nThe trajectory length H seems like a key parameter in the algorithm. If H equals the episode length, the update would be the same as MCTS, which means the search goes until the end of the episode. I assume this parameter needs to be tuned carefully in the experiment. It would be nice if there is an experiment discussing how performance changes regarding different search horizons. If the length of the horizon can be fixed and no need to tune, it will also be good to explain how this number should be chosen. This will make it easier for readers to apply this method to other problems.\n\nIt may be worth listing hyperparameter settings in the paper, such as network structure and learning rate for both the proposed method and baselines, for reproducibility. I understand that the code base is provided and there are default settings in the code, but it is not clear to me if the default setting is the same setting used for generating the results reported in the paper.\n\nSmall Things:\nIn Algorithm 1, the symbol used in output ($\\tilde{\\theta}$) is inconsistent with the return ($\\theta_N$)\n\n[1] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to replace tabular MCTS-like search methods with finetuning a pretrained policy for the states encountered online. The the high-level method is a good contribution despite being straightforward. The experiments are fairly limited though, for what is largely an empirical paper. Even with the addition of a few more Atari games as a result of the post-rebuttal discussion, the evaluation is weak by the standards of empirical RL papers. I'd expect at least 20-30 more Atari games, or the Atari games on which the authors already have the results and the Procgen suite on top of that. I'm not discounting Hanabi -- it's a hard benchmark, but it's still just one benchmark. Running experiments on more games/environments is especially important since, according to the additional results mentioned in the discussion, in low-data and weak-blueprint regimes RL-Fine-Tune's advantage over MCTS is far from apparent.

On balance, I cautiously recommend acceptance since RL-Fine-Tune seems novel and a strong baseline, and due to its simplicity it's likely to be experimented with and built on by others.
Decision: Accept (Poster)