TPO: TREE SEARCH POLICY OPTIMIZATION FOR CONTINUOUS ACTION SPACES
Authors: Amir Yazdanbakhsh
Ebrahim Songhori
Robert Ormandi
Anna Goldie
Azalia Mirhoseini
Venue:  (2020)
Keywords: monte-carlo tree search
reinforcement learning
tree search
policy optimization

Abstract: Monte Carlo Tree Search (MCTS) has achieved impressive results on a range of discrete environments, such as Go, Mario and Arcade games, but it has not yet fulfilled its true potential in continuous domains.In this work, we introduceTPO, a tree search based policy optimization method for continuous environments. TPO takes a hybrid approach to policy optimization.  Building the MCTS tree in a continuous action space and updating the policy gradient using off-policy MCTS trajectories are non-trivial. To overcome these challenges, we propose limiting tree search branching factor by drawing only few action samples from the policy distribution and define a new loss function based on the trajectories’ mean and standard deviations.  Our approach led to some non-intuitive findings.  MCTS training generally requires a large number of samples and simulations. However, we observed that bootstrappingtree search with a pre-trained policy allows us to achieve high quality results with a low MCTS branching factor and few number of simulations. Without the proposed policy bootstrapping, continuous MCTS would require a much larger branching factor and simulation count, rendering it computationally and prohibitively expensive. In our experiments, we use PPO as our baseline policy optimization algorithm. TPO significantly improves the policy on nearly all of our benchmarks.  For example, in complex environments such as Humanoid, we achieve a 2.5×improvement over the baseline algorithm.

Under review as conference paper at ICLR 2020
TPO: TREE SEARCH POLICY OPTIMIZATION FOR CONTINUOUS ACTION SPACES
Anonymous authors
Paper under double-blind review
ABSTRACT
Monte Carlo Tree Search (MCTS) has achieved impressive results range discrete environments, such as Go, Mario and Arcade games, but it has not yet fulfilled its true potential in continuous domains. In this work, we introduce TPO, a tree search based policy optimization method for continuous environments. TPO takes hybrid approach policy optimization. Building the MCTS tree in a continuous action space and updating the policy gradient using off-policy MCTS trajectories are non-trivial. To overcome these challenges, we propose limiting tree search branching factor by drawing only few action samples from the policy distribution and defining a new loss function based on the trajectories' mean and standard deviations. Our approach led to some non-intuitive findings. MCTS training generally requires a large number of samples and simulations. However, observed that bootstrapping tree search with a pre-trained policy allows us achieve high quality results with a low MCTS branching factor and few simulations. Without the proposed policy bootstrapping, continuous MCTS would require much larger branching factor and simulation count, rendering it prohibitively expensive. In our experiments, we use PPO as our baseline policy optimization algorithm. TPO significantly improves the policy on nearly all the environments. For example, in complex environments such as Humanoid with a 17 dimensional action space, we achieve a 2.5x improvement over the baseline algorithm.
INTRODUCTION
Fueled by advances in neural representation learning, the field of model-free reinforcement learning has rapidly evolved over the past few years. These advances are due in part to the advent of algorithms capable of navigating larger action spaces and longer time horizons [2, 32, 42], as well as the distribution of data collection and training across massive-scale computing resources [42, 38, 16, 32).
While learning algorithms have been continuously improving, it is undeniable that tree search methods have played a large role in some the most successful applications RL (e.g., AlphaZero [42], Mario [10] and Arcade games [48]). Tree search methods enable powerful explorations of the action space in a way which is guided by the topology of the search space, focusing on branches (actions) that are more promising. Although tree search methods have achieved impressive results range of discrete domains, they have not yet fulfilled their true potential continuous domains. Given that the number of actions is inherently unbounded in continuous domains, traditional approaches to building the search tree become intractable from a computational perspective.
In this paper, introduce TPO, Tree Search Policy Optimization for environments with continuous action spaces. We address the challenges of building the tree and running simulations by adopting a hybrid method, in which we first train a policy using existing model-free RL methods, and then use the pre-trained policy distribution to draw actions with which to build the tree. Once the tree has been constructed, we run simulations to generate experiences using an Upper Confidence Bounds for Trees (UCT) approach [33]. Populating the tree with the action samples drawn from pre-trained policy enables us to perform a computationally feasible search. TPO is a variation of the policy iteration method [35, 44, 42, 1]. Broadly, in these methods, the behavior of policy is iteratively updated using the trajectories generated by an expert policy. Then, the newly updated policy in return guides the expert generate higher quality samples. In TPO, we use tree search quality trajectories. Later, we employ the updated policy to re-populate the tree search. For tree search, use the Monte Carlo Tree Search (MCTS) [5] expansion and selection methods. However, it is
expert to generate high

---

Under review as conference paper at ICLR 2020
challenging directly infer the probability of selected actions for rollout; unlike in discrete domains where all actions can be exhaustively explored, in continuous domains, we cannot sample more than subset of the effectively innumerable continuous action space. Furthermore, to the trajectories generated by MCTS, we must perform off-policy optimization. To address this challenge, we define new loss function that uses the weighted mean and standard deviation of the tree search statistics to update the pre-trained policy. For ease of implementation and scalability, we use Proximal Policy Optimization (PPO) [38] and choose as our policy optimization baseline.
Our approach led to some non-intuitive findings. MCTS training generally requires a large num- ber of branches and simulations. For example, AlphaGoo uses 1600 simulations per tree search and branching factor of up to 362 [40]. How- ever, we observed that if we pre-train the policy, we require far fewer simulations to generate high quality trajectories. While we do benefit from ex- ploring a greater number of branches, especially for higher dimensional action spaces (c.g. Hu-
Distributed
Dittibuueed Optmmzztion Workers
Tioe Search Workers
Experience Heplay Butters
Polly & Value Networks
manoid), we observed diminishing returns after Figure 1: phase we perform policy gradient based optimization training build a target policy. In phase 2, we iteratively build an MCTS tree using the pre trained larget policy and update the target policy using roll-out trajectories from MCTS. Both training and data collection are done distributed manner.
Tree Search Policy Optimization (TPO) Framework.
only small number of branches (e.g., 32) across all of the evaluated environments. Furthermore, performance quickly plateaued as we increased the number of simulations past 32. This property did not hold when we initialized tree search with
an untrained policy. This is critical advantage of our method as it would otherwise be computationally infeasible to generate high quality trajectories using tree search. The main contributions of TPO are summarized as follows:
1. Tree search policy optimization for continuous action spaces. TPO is one of the very first techniques that integrates tree search into policy optimization for continuous action spaces. This unique integration of tree search into policy optimization yields a superior performance compared to baseline policy optimization techniques for continuous action spaces.
Policy bootstrapping. We propose policy bootstrapping technique that significantly improves the sample efficiency of the tree search and enables us to discretize continuous action spaces into only a few number of highly probable actions. More specifically, TPO only performs 32 tree searches compared to substantially larger number of tree searches (1600, 50x more) in AlphaGo [42]. In addition, TPO narrows down the number of tree expansion (actions) compared to discretization techniques such Tang et al. [45] which requires 7-11 1 bins per action dimension. This number of bins translates to a prohibitively large number of actions even in discrete domain for complex environments such Humanoid which has 17 dimensional action space. In contrast, TPO only samples 32 actions at each simulation step across all the environments.
3. Infrastructure and results. On the infrastructure side, we developed a distributed system (shown in Figure 1), in which both policy optimization and data collection are performed on separate distributed platforms. The policy optimization is done a TPU-v2 using multiple cores, and MCTS search is performed on a rack of CPU nodes. A synchronous policy update and data collection approach is used to train the policy and generate trajectories. TPO readily extends to challenging and high-dimensionall tasks, such the Humamoid benchmark [9]. Our empirical results indicate that TPO significantly improves the performance of the baseline policy optimization algorithm, achieving up to 2.5x improvement.
2 BACKGROUND
Policy iterations methods have shown promising results in reinforcement learning. AlphaZero [42], major breakthrough in reinforcement learning, uses MCTS as expert to train a discrete policy model for the game of Go. EXiT [1] established theoretical grounds for policy iterations using MCTS in discrete domains. A recent work [28] proposes some theoretical extensions to AlphaZero's 's MCTS method [42] for environments with continuous action spaces. However, the authors only evaluate their method on the Pendulum OpenAl Gym environment which has only a one dimensional action
2

---

Under review as conference paper at ICLR 2020
space. is not obvious how the approach can scale to more complex environments with much higher dimensions. While TPO also belongs to the category of policy iteration methods, it studies a rather less explored territory of using MCTS as expert for complex environments with continuous action spaces.
Policy optimization falls into two major categories: on-policy and off-policy optimization. In on-policy methods, the policy is optimized using samples generated from the current policy, whereas with off-policy algorithms, we can also train the policy using experiences from other sources, such as an expert. Although on-policy reinforcement learning methods, such as TRPO PPO and A3C [27] have been extremely successful, they are known to be sample inefficient [16, 12], as they must generate fresh trajectories between each gradient update and cannot reuse these trajectories. This effect becomes more pronounced as the complexity of the task grows, requiring millions of steps to learn a policy. On the other hand, off-policy methods (c.g.. DQN [26, 27]. DDPG [24]) are more sample efficient, as each example is used multiple times for training. However, these methods require more extensive syper-parameter tuning, suffering from instability and slow convergence [30, 18, 16, 34, 47].
PPO [S8],PGQL, [31],
A few recent methods have been developed embed differentiable planning procedures in computation graphs [15, 43,23 25, 14, 19J. MCTSnnets [15] is deep learning model that incorporates simulation-bart search with vector embedding for expansion, evaluation, and backup. The three policies are jointly optimized with an end-to-end gradient based approach. The authors only report results on the classic planning benchmark, Sokoban, which is a simple discrete optimization environment. Extension of this approach to continuous domains is non-obvious. SoRB [11] is another approach that combines planning with policy optimization. However, the target problem space of SoRB is only goal- reaching tasks. Although there has been previous work on continuous MCTS, none has been conducted in the context of deep reinforcement learning [8, 13, 7]. More sophisticated approaches to node expansion improve the quality of search [7], but TPO only uses simple scheme in which a fixed number of actions are sampled from the policy. We found our approach both simple to implement and effective, without the need for extensive fine-tuning or hyper-parameteer optimization. For example, we found that across the evaluated OpenAl Gym environments, a sample size of 32 yields strong results. In our experiments, we chose PPO [38], which has been shown to yield superior performance in various large-scale problems such as Hide and Seek [2] and Dota [32], to pre-train the policy used to populate the MCTS tree. However, our framework is independent of the specific choice of baseline policy and can be readily applied to other model-free RL algorithms, such as SAC [16] or TD3 [34].
MONTE CARLO TREE SEARCH FOR CONTINUOUS ACTION SPACES
Preliminaries and notations. Policy optimization in continuous environments can be well modeled by a Markov Decision Process (MDP) [3), defined by the tuple (S.A.p,r,7), where S(AA) is a set of states (actions) in continuous space. The state transition probability p(Se+1 (,,,aa) represents the probability desnsity of observing the next state in time, S SS, given the current S. E S and the current action al. E AA. Taking action ag at state Sy, the environment moves to a new state S and returns a bounded reward r(s;.a)) S x A + [maz]] Finally, y E (0.1) represents the discount factor, used to calculate the discounted cumulative expected reward. We use To (ag to represent the neural network policy with trainable parameters 0. We use (5:) to represent the value from a neural network with parameters Φ.
3.1 MONTE CARLO TREE SEARCH ALGORITHM FOR CONTINUOUS ACTION SPACE
Previous work on Monte Carlo Tree Search (MCTS) was mostly focused on zero-sum games with discrete action spaces [42, 41, 29, 40. 5, 8, 22, 39]. In this section, we will highlight our adaptation of MCTS for environments with continuous action spaces and non-zero immediate rewards. For the complete definition of MCTS, we invite the readers to see [42]. To make MCTS possible, we assume the environment can be set to an arbitrary state. This assumption can be relaxed using a learned model of the environment (See $6).
In this work, we use variant of Polynomial Upper Confident Tree (PUCT) algorithm in TPO for MCTS [33]. Nodes of the Monte Carlo tree are states of the environment s, E S and edges are actions, ag A. We store set of statisties for each edge of Monte Carlo tree: the number of visits N. the cumulative sum of action value W. and the average of the action value Q. We update these statistics after each MCTS simulation for the traversed edges. Each MCTS simulation starts from a root state
3

---

Under review as conference paper at ICLR 2020
and consists of four steps: select, expand, evaluate, and backup. Here we only overview the select and expand steps, as they are the contributions of TPO to enable tree search for continuous action spaces.
Select. We follow variant of Polynomial Upper Confident Tree (PUCT) algorithm [33, 42] to select actions. For each node, St, we select an action: a argmax Q(sr, as) + US2,, ag), where (s is:
(s:)
U ($).ag) = Cpuct To (a (se)
(1)
++N((s,a))
where Cpuct is a constant exploration coefficient and me (ag 52) represents the prior probability assigned to each action by the policy network. To further promote exploration, we add Dirichlet noise, Dir(a)), to the prior probabilities of the actions at the root node [42].
To support environments with immediate rewards, we propose to use the following equation for estimating the value of an action:
W (s;,a
=()(s,))
(2)
N (s,,aa)
where W(s, as) ) is the cumulative sum over the backed up values from its children.
Expand. In discrete environments such as the game of Go [42] or Chess/Shogi [41], all possible actions are considered in MCTS expansion. However, in environments with continuous action spaces, the set of possible actions is innumerable. Hence, enumerating all the possible actions for each node is impossible. To circumvent this challenge, we propose to draw samples from the policy distribution. These sampled actions will become the edges of the leaf node.
After fixed number of simulations, select action with the largest visit counts from the current root node and advances the environment. We use Q(s:, ag) (S; as tie-breaker between the actions with the same number of visit counts. We also define ( an unbiased estimate [46, 27] for the mean value of the tree policy at given node computed across each action space dimension:
jurs
NÍS. a6) N(s))
(3)
Later in Section 4, use this unbiased mean estimate optimize the policy distribution. All MCTS trajectories are added to a replay buffer, which is then used to optimize the policy and value networks.
4 TREE SEARCH POLICY OPTIMIZATION (TPO)
Neural network architecture. Both value, (.).), and policy networks, (a((), are represented by fully-connected multi-layer perceptron (MLP) with two hidden layers of 64 units. The output of each layer passed through a tanh(-) non- linearitty function. The weights of value and policy networks are not shared. The value network outputs single scalar value v(s,), whereas the policy network outputs vector of means (s:),, and variances (s.) for number of independent Normal distributions for each action dimension.
Loss functions. We first describe the loss functions for the on-policy phase and then delve into the loss functions of the proposed off-policy tree search optimization. For ease of implementation, we used PPO (38) for on-policy optimization. The policy loss function of PPO is:
I. CLP (0)= (max - (0) A., cl)pp(; - -c,1)+))A.)) where ro( To (ar
(4)
Tow. (a. Sz)
where t, (0) is the probability ratio between the new policy, Te, (ax (s:), and the old policy, maai a (ax (s:) PPO clips the probability ratio to be within the range [1-e, 1 +].. which penalizes large updates to the policy. Å, is the truncated generalized advantage estimate [38, 37], computed for a trajectory of length T'. To update the value network, we use a variant of PPO? that also applies clipping to the value function:
VLLIP (s:) - Voli (st Cllipp
Voiia (5:),- c,
(5)
Dir(a) returns a noise vector {77 (711.72....,77 that controls the shape of the generated noise vector.
.7,, where Σ,
o.. ER is Dirichlet noise scaling parameter
4

---

Under review as conference paper at ICLR 2020
Algorithm 1: TPO: Tree Search Policy Optimization
iteration <- i... 1.2 number of iterations
2
if ilitetationS x number of iterations
b IE (0,1)
then
No (((ata)
5
Le + L (0:0)
On-Policy Training: See Eq. 4, 5, and
else
We 5 T(
Tie + T (0;d)
Oft-Policy Training; See Eq. 6, and
Roll out trajectories (71. 72.... for timesteps each using n..
10
for - 1,2, number of epochs do
Optimize Le wrt to f and é with minibatch size M
Bold
13
Ootd
Table 1: The benchmark suite (six OpenAl Gym [4] environments) with their corresponding observationaaction spaces.
Environments
State Space
Action Space
Ant-v2
R II
-10,,1.0
HallCheetah-v2
R17
-100,1.0
Hopper-v2
R 11
-100,1.0]
Humanoid-v2
R2766
-0.4,0.4
Swimmer-v2
R
1.0,1.0
Walker2d-v2
R
-10,1.0)
(0) =
[ma -s.) VV (V)(a))V,
(6)
Finally, the total loss combines the policy surrogate and value function terms with a regularization coefficients c
In the off-policy phase, the same value function PPO for the value updates, as shown in Eq. and Eq. 6. For the policy updates, we define a new loss function, derived from the collected MCTS statistics:
1. (0)= = ((ATs (s:)) (ors (5:))
(() is the sum of the squared-erro loss between the unbiased mean estimate and the standard deviation estimate, calculated from the MCTS trajectories, and the mean and standard deviation of the policy distribution at given state s; The intuition for choosing this loss term for policy updates is to incentivize moving the mean of the policy distribution towards unbiased estimate computed from the tree search. Finally, we combine the defined TPO policy loss and value loss to obtain the following objective, which is maximized at each training step:
Lod (0:0) - (00))==(000)
the policy and value networks do not share parameters, the regularization coefficients ej and c3 become irrelevant. In neither Lon-r or Lor-x losses, we do not use any entropy bonus term.
Algorithm. Algorithm I fleshes the proposed TPO algorithm, which uses combination of policy gradient and MCTS. In the first phase of training, we use the policy network to roll out trajectories and optimize the PPO loss function (See Eq. 7). the second phase, we switch to an off-policy mode, in which we only use trajectories from MCTS to optimize the policy. That is, we construct the loss (See Eq. 9.) on the roll out trajectories generated by the tree search. We treat p. which defines the switching ratio between policy gradient and tree search,
hyperparameter of the TPO algorithm. The intuition behind bootstrapping tree search with a pre-trained policy is that continuous action spaces require relatively stable policy to encourage the exploration of trajectories with higher returns, and an untrained policy cannot adequately guide the tree search. Our experiments support this hypothesis; start from lightly trained policy (i.e. small switching ratio, p), the final performance is severely diminished. The same reasoning applies to choosing to start with policy gradient optimization the first phase of training followed by tree search policy optimization.
5

---

Under review as conference paper at ICLR 2020
5 EXPERIMENTS
We evaluate TPO on variety of continuous environments from OpenAI Gym [4] (Table 1). Table show the hyperparameters used for training the value and policy networks and those used for tree search. Our policy training done on a TPU-v2 platform with eight cores, and MCTS is performed on CPU. We chose PPO [38] for the on-policy optimization update. For each environment, we run TPO for four million time steps to estimate the gradients.
To efficiently perform MCTS sampling for trajectory rollouts, we use BatchEnv [17] that extends the OpenAl Gym [4] interface to multiple parallel environments. Using BatchEnv, we evaluate up to 32 actions in parallel and obtain their associated rewards. We use 32 Intel CPUs, each assigned to one environment instance, in order to parallelize simulation. We found that 32 is reasonable choice for the maximum number of MCTS simulations per each action selection, as it strikes good balance between search latency and policy performance. Using our parallel infrastructure, the latency of action selection is somewhere between 0.2 sec when we use 32 MCTS simulations per action.
Unless stated otherwise, in all of the experi- ments, we report the average return of the last 100 episodes across five runs (each with differ- ent random seed). To make a fair comparison with the baseline algorithm (PPO), we repro- duce the reported results in [38] using our in- frastructure. In all the environments, we use the best obtained results for PPO using our infrastructure, which in some environments (HalfCheetah-v2 and Walker2d-v2) outperform the reported results in [38]. In all the experi- ments, use Adam [21] as our optimizer and run training for 10 epochs per policy update.
Table TPO hyperparameters.
MCTS Hyperparameters
Value
Number of MCTS simulations
32
Maximum number of actions node
32
Guus
Temperature (r)
Dirichlet noise scaling parameter (a)
1.75
Dirichlet noise weight (w)
0.5
Training Hyperparameters
Value
Policy network architecture
Value network architecture
FC:64 -C:64
Adam learning
3 10 and 9x 10
Horizon (T)
2048
Summary of results. Table 3 depicts the per- formance comparison between TPO and PPO across six environments with continuous action spaces. In all the environments, TPO yields better total return compared PPO, gener- ally by a significant margin. Using TPO, we observe a maximum improvement of roughly 2,900 in Ant-v2 and minimum improvement of only in Swimmer-v2. Swimmer-v2 is the sim-
Switching ratio (p)
Clipping coefficient (c)
0.2
Number of epochs
10
Minibatch size (M)
128
Gradient norm clipping coefficient
0.5
Discount (x)
0.99
GAE parameter (X)
0.95
plest environment in the set of evaluated benchmarks with an action space of only two dimensions. the other hand, Ant-v2 is much more complex benchmark with an action-space of eight dimen- sions. The delta in performance improvement suggests that TPO is better able to leverage tree search exploration to improve policies in complex environments with large action spaces. Figure shows the training progress of TPO and PPO across six environments. To conduct this experiment, pre-training with PPO. At ~ 2M time steps, TPO restores checkpoints from PPO training and continues training using tree search. As such, the total return curve for both TPO and PPO overlap before reaching 2M time steps. In all the environments, we observe a sharp spike in the total return curve after switching tree search, which can be attributed to the effectiveness of the proposed tree search method for policy optimization. For complex environments, such as Humanoid-v2, we observe continuing improvement after switching to MCTS based policy optimization. Ablation Studies To better understand the impact of key components of TPOO we perform a set of ablation studies. Concretely, we show the results for
start by
Table Performance comparison between TPO
atter 4M time-steps. values difference between PPO for each category. A positive A value indicales that TPO oulperforms PPO.
Min.
Avg. TPO
Max.
Benchmarks
PPO
TPO
PPO
A
PPO
TPO
A
Ant-v2
1722.75
4455.57
+2732.81
1873.85
4777.06
+2903.21 21
2022.08
5011 0008
+2989.0
HalfCheetah-vv
1658.70
1788.77
+ +30.00
2530.81
3949.05
+1418.24
5049.07
6734.45
+ 1685.38
Hopper-v2
1744.24
2205.47
+461.22
1937.82
3196.14
+1258.32
2051.89
3613.36
+ +1561.47
Humanold-v2
4269.44 44
5391 43
+1121.99
4439.58
5460.93
+1021.33
4609.72
5530.43
+920.71
Swimmer-v2
109.91
119.48
+9.57
113.74
120.81
+7.07
116.74
122.39
+5.65
Walker2d-v2
3605.27
3788.83
+ +18..56
3942.57
4092.21
+149.63
4155.37
4667 64
+512.27

---

Under review as conference paper at ICLR 2020
x107
x100
4.0 =2002
5
3.5
3.0
Reau
3
2.
0-
TPD
TPO PPO
TPO PPO
PPO
0.0
3.5 4.0
0.0
0.5
1.0
2.0
3.5 Number of Timesteps
1.5 2.0
3.0
Number Timesteps
x10
Number of Timesteps
(a) Ant-v2
HalfCheetahh-22
(e) Hopper-v2
1.4
-10
1.2
1.0
Return
Return
0.8
0.6
2-
0.4
0.2
1-
PPO
0.0
PPO
TPO FPO
-0.2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0 0.5
1.0
1.5 2,0 2.5 3.0
3.5 4.0 x10
Number Timesteps
x10
Timesteps
x10
Timesteps
(d) Humanoid-v2
(e) Swimmer-v2
(f) Walker2d-v2
Figure Comparison between PPO and TPO training curves. Each data point represents average return of the last 100 training episodes. MCTS rollout starts alter 50% of the total number of timesteps, lee. ~2M
<10'
Sericht 101
switchirg rello
SWithing 50% SAiriine
switch racs 5O% Sw F900 NOYE
switching rito S0n Swiccing rills 101
MW
3.0
5
2.5
Retum
Retur
4.
2.0
1.5
2-
1-
0.0
0.5
1.0
0.0 0.5
1.0
2.0 2.5 3.0
0.0
0.5 1.0
1.5
2.0
3.0 3.5 4.0
Number Timesteps
Number Timestops
of Timestaps =10'
(a) Hopper-v2
ag E R
(b) Ant-v2 dg E R
Humanoid-v2 a4 E
Figure 3: TPO training curves when switching ratio Algorithm is 50%50% (default TPO),
three environments with different levels of action space complexity, namely Hopper-v2, Ant-v2, and Humanoid-v2 with action spaces of three, eight, and 17, respectively.
Switching ratio. First, we study the impact of varying the switching ratio (p in Algorithm 1) between 10%. 50% (default value in TPO), and 90% across three tasks. As shown in Figure 3, prematurely switching (p= 10%, blue line in Figure 3) to tree search optimization severely harms policy performance, showing no further improvement after switching to MCTS optimization. This behavior can be attributed to the fact that in tree search policy optimization for environments with continuous action spaces, it is of utmost importance to have a reasonably well-traines policy to guide the tree search toward trajectories with potentially higher return values. Switching to search at the very end of training (p 10%. green line in Figure 3) can be damaging as well. While switching to tree search at the 90% of the total training steps in Hopper-v2 does not improve the average return compared to switching after 50% of the total training steps, it slightly improves the average return in Ant-v2. That is, the point at which to switch tree search is hyperparameter that should be adjusted carefully for better policy performance.
Number of actions per node. Figure 4 shows the impact of varying the number of actions per node in tree search between 4, 8, and a2 (default TPO is 32). This study reveals the following findings. (I) variance reduction in return values; increasing the number of actions per node significantly reduces the variance of the total return value (i.e. shrinkage in the shaded area). which further improves the robustness of the training process. (2) better policy performance (higher return value); while the impact of increasing the number of actions per node marginal in simple environments, such as Hopper-v2 and Ant-v2, exploring more actions per node in environments with complex action
7

---

Under review as conference paper at ICLR 2020
4.0
<10
X100
-10F
3.5
4
Return
Return
3
1.5
2-
1.0
adlora per node
32
ww
1.0 1.5 25 3.0
3.0
15 2.0
3.0 3/5
Number of Timestops
X30
Number Timestops
Number of Timestops
x10*
(a) Hopper-v2 O E R
(b) Ant-v2 a E
(c) Humanoid-v2 a2 E E
Figure 4: TPO training curves with varylng the number of actions per node.
4.0
-10*
2.5
Return
Retum
2.0
3
1.5
MCT andee.ro NCT SrAMEM
2 METS ATARE 2 MANT
0.5 1.0
25 Number Timestops
0.0 0.5 1.
2.0
0.0
1.
2.0 2.5
3.0 3.5 4.0
Number Timestops
Number Timestaps
(a) Hopper-v2 04 R
(b) Ant-v2 ag R
(c) Humanoid-v2 0g. T
Figure 5: TPO training curves with varying number MCTS simulations.
space, Humanoid-v2 22, has significant benefit for policy performance. As shown in Figure 4c, changing the number of actions per node from eight 32 increases the average total return by ~1100
Number of MCTS simulations. Figure 5 shows the impact of the number of MCTS simulations (See Section 3) on the performance of the policy. While increasing the number of MCTS simulations generally improves performance, we found that increasing the branching factor (number of actions per node) had more impact. For example, for Humanoid-v2, the performance gap between 8 and 32 simulations (shown in Figure 5c), is less pronounced than the improvement from increasing the number of actions per node from and 32 (shown in Figure 4c). These studies suggest that is more beneficial to devote the tree search budget to exploring more actions per node, especially for more complex environments, rather than conducting larger number of tree search simulations.
DISCUSSION
In this paper, we have studied Monte Carlo Tree Search in continuous space for improving the performance of baseline on-policy algorithm [38]. Our results show that MCTS policy optimization can indeed improve the quality of policy in choosing better actions during policy evaluation at the cost of more samples during MCTS rollout. We show that bootstrapping tree search with a pre- trained policy enables to achieve high performance with a low MCTS branching factor and few simulations. On the other hand, without pre-training, we require a much larger branching factor and simulation count, rendering MCTS computationally infeasible. One of the future research direction is to explore techniques for improving the sample efficiency and removing the need for having a reset-able environment. To achieve these goals, we can use a trained model the environment similar to model-based reinforcement learning approaches [20, 23, 6]. instead of interacting directly with the environment in MCTS. Recently, MBPO [20] showed that they can train model of Mujoco environments that is accurate enough for nearly 200-step rollouts in terms of accumulated rewards. This level of accuracy horizon more than enough for the shallow MCTS simulations (32 simulations) that is employed in TPO. As mentioned earlier, TPO assumes access to an environment that can be restarted from an arbitrary state for MCTS simulations. While this assumption can be readily satisfied in some RL problems such playing games, it may be harder to achieve for physical RL problems like Robotics. This assumption can also be relaxed using a modeled environment to replace the interactions with the real environment during MCTS simulations.
8

---

Under review as conference paper at ICLR 2020
REFERENCES
[1] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In NIPS. pp. 5360-5370, 2017.
[2] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Hmergent Tool Use From Multi-Agent Autocurricula. arXiv preprint arXiv: /0..//99900722 2019.
[3] Richard Bellman. A Markovian Decision Process, Journal of Mathematics and Mechunics, pp. 679-684, 1957.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai Gym. arXiv preprini oo 2016.
[5] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. Survey of Monte Carlo Tree Search Methods. IEEE Transactions Computational Intelligence and Al in games, 4(1), 2012.
[6] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep Reinforcement Learning in Handful of Trials using Probabilistic Dynamics Models. NIPS, 2018.
[7] Adrien Coučtoux, Jean- Baptiste Hoock, Nataliya Sokolovska, Olivier Teytaud, and Nicolas Bonnard. Continuous upper confidence trees. In Proceedings of the Sth International Conference on Learning and Intelligent Optimization, 2011.
[8] Rémi Coulom. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. In International conference on computers and games. Springer, 2006.
[9] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep Reinforcement Learning for Continuous Control. In ICML, 2016.
[10] R. Greve E. Jacobsen and J. Togelius. Monte. Mario: Platforming with mets. 2014.
[11] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the Replay Buffer: Bridging Planning and Reinforcement Learning. arXiv preprint
2019.
[12] Rasool Fakoor, Pratik Chaudhari, and Alexander J. Smola. P3O: Policy-on Policy-off Policy Optimization. arXiv preprint 1905:::::90000// 2019.
[13] J. Uiterwijk H. van den Herik G. Chaslot, M. Winands. Progressive Strategies For Monte-Carlo Tree Search. New Mathematics and Natural Computation (NMNC), 2008
[14] M. Igl Farquhar, T. Rocktäschel and S. Whiteson. TreeQN and ATreeC: Differentiable tree planning for deep reinforcement learning, arXiv preprint I 1417, 2017.
[15] A. Guez, Т. Weber, I. Antonoglou, K. Simonyan, O. Vinyals, D. Wierstra, R. Munos, and D. Silver. Learning to Search with MCTSnets. arXiv preprint
2018.
[16] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. arXiv preprint 1801 2018.
[17] Danijar Hafner, James Davidson, and Vincent Vanhoucke. Tensorflow Agents: Efficient Batched Reinforcement Learning Tensorflow. arXiv preprint 17Xivv1709.02880 02878, 2017.
[18] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[19] S. Singh. J. Oh and H. Lee. Value Prediction Network. NIPS, 2017.
[20] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS, 2019.

---

Under review as conference paper at ICLR 2020
[21] Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980, 2014.
[22] Levente Koesis and Csaba Szepesvári. Bandit based Monte-Carlo Planning. In European conference on machine learning, pp. 282-293. Springer, 2006.
[23] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-Enseembe Trust-Region Policy Optimization. ICLR, 2018.
[24] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arxiv:/50.0029771 2015.
[25] G An E Theodorou M Pereira, D Fan. Mpe-inspired Neural Network Policies for Sequential Decision Making. arXiv preprint arXiv: /tti:::8880.000000 2018.
[26] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
[27] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In International conference on machine learning, 2016.
[28] Thomas M Moerland, Joost Broekens, Aske Plaat, and Catholijn M Jonker. A0c: Alpha zero in continuous action space. arXiv preprint 100.vvv800.00611111 2018.
[29] Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisy, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level Artificial Intelligence in Heads-up No-limit Poker. Science, 356(6337), 2017.
[30] K. Xu D. Schuurmans O. Nachum, M. Norouzi. Bridging the gap between value and policy based reinforcement learning. 2017.
[31] Brendan O' D'onoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning arXiv preprint 161 2016
[32] OpenAl. Openai five, 2018.
[33] Christopher D Rosin. Multi-armed Bandits with Episode Context. Annals of Mathematics and Artificial Intelligence, 6133):203-230, 2011.
[34] D. Meger Fujimoto, H. van Hoof. Addressing Function Approxxmation Error in Actor-Critic Methods, 2018.
[35] Bruno Scherrer. Approximate policy iteration schemes: a comparison. In International Confer- ence Machine Learming, pp. 1314-1322, 2014.
[36] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In 2015.
[37] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional Continuous Control using Generalized Advantage Estimation. arXiv preprint 150iv 02438, 2015.
[38] John Schulman, Filip Wolski. Prafulla Dhariwal. Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint 17Xvv:1707.06347, 2017.
[39] Brian Sheppard. Wordd-championstip-ccllee Scrabble. Artificial Intelligence, 134(1-2), 2002.
[40] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre. George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pannecrshelvam, Mare Lanctot, et al. Mas- tering the Game of Go with Deep Neural Networks and Tree Search. nature, 529(7587), 2016.
10

---

Under review as conference paper at ICLR 2020
[41] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering Chess and Shogi by Self-play with General Reinforcement Learning Algorithm. arXiv preprint 2017.
[42]
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et Mastering the Game of Go without Human Knowledge. Nature, 550(7676), 2017.
[43] Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal Planning Networks, arXiv preprint arXiv: 1804.80040000 00645, 2018.
[44] Wen Sun, Geoffrey Gordon, Byron Boots, and Bagnell. Dual policy iteration. In NIPS, pp. 7059-7069, 2018.
[45] Yunhao Tang and Shipra Agrawal. Discretizing Continuous Action Space for On-policy Opti- mization. arXiv preprint arXiv:1901 10500, 2019.
[46] Philip Thomas and Emma Brunskill. Data-efficient Off-policy Policy Evaluation for Reinforce- ment Learning. In International Conference on Machine Learning, pp. 2139-2148, 2016.
[47] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv: /0t::66/1..00 1.10124, 2016.
[48] H. Lee R. L. Lewis X. Guo, S. Singh and X. Wang. Deep learning for real-time atari game play using offline Monte-Carlo tree search planning. 2014.


Reviews:
[{"review_id": "SkgKfHeRYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary\nThe paper proposes to use MCTS for fine-tuning a policy in continuous control tasks. Action selection is done by PUCT [42] and node expansion is done by sampling from the policy instead of trying all actions. Furthermore, the importance of good parallel implementation is highlighted. Results on MuJoCo tasks show a bit of a gain in performance but with quite high variance.\n\n# Decision\nThere are some concerns regarding the novelty of the proposed method. Furthermore, evaluations seem rather noisy to make a reliable judgement. Therefore, I currently refrain from recommending this paper for publication.\n\n# Discussion\n1) Being not an expert on MCTS, it nevertheless appears to me that it is a natural idea to extend it to continuous problems. A quick search reveals a number of papers, e.g.,\n\nhttps://ieeexplore.ieee.org/document/8401544\nhttps://tel.archives-ouvertes.fr/tel-00927252/document\nhttps://openreview.net/forum?id=SyiF5-23Z\nhttp://proceedings.mlr.press/v80/lee18b/lee18b.pdf\n\n=> Can you comment on what exactly is your contribution?\n\n2) Experiments\n    - I am not sure that showing the burn-in period of pure PPO learning in Fig. 2 is informative. Clearly, if one starts with a good policy, it can only get better from there. So, one can directly start with a PPO-pretrained policy.\n    - For the same reason, the switching ratio experiments in Fig. 3 seem superfluous.\n    - In MuJoCo experiments in Fig. 2, in the top-right two plots, there is quite high variance; in the bottom plots, on the other hand, the variance grows and shrinks. How can you explain such oscillating behavior of the variance among runs? How many trials were performed? The curves do not seem to be reliable.\n    - Evaluations in Figs. 4–5 are quite indecisive, except for Humanoid maybe. Perhaps, more representative environments could be chosen.\n    - The algorithm is not compared to any other approach. Are there no similar methods?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeT7Brx9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a hybrid approach that combines MCTS with policy optimization. The main idea is to use PO for policy improvement, then boostrapping for action selection. The proposed hybrid framework enables MCTS planning on continuous action problems. The method is demonstrated on continuous control tasks such as Humanoid with high-dimensional and continuous actions. Integrating planning into PO like this way is shown a large improvement over baseline methods.\n\nOverall, this paper pursues an interesting research problem. Integrating a planning ability to policy optimization would definitely be desired to achieve more data-efficiency. The idea proposed in this paper is quite straightforward. It simply proposed to use policy gradient algorithms to optimize the rolout policy and the action selection policy. In addition, some of the descriptions in the paper are unclear that makes it hard to understand. My main concerns are as follows.\n\n- The policy network is trained using a gradient-based approach. However, the original MCTS framework is expected to optimize a globally optimal policy. It would be more valuable if the authors pay some discussions for this limitation and demonstrate it in experiments. Probably if it is evaluated on a discrete domain first. \n\n- As said this hybrid approach not only aims to extend MCTS to continuous domains but could also be considered as a way of integrating a planning ability into policy optimization that helps data-efficiency. Therefore more related work and discussions, and drawing connections with planning embedding and model-based policy optimization are very helpful.\n\n- There are many technical details missing, hence making it difficult to understand: i) how a search tree can be built for continuous domains?, I do not see how the branching factor and the policy boostrapping are used to simplify a search tree construction for continuous problems; ii) how the description in section 3 is related to the proposed framework? iii) what is the meaning of the Number of actions per node, it is unclear which nodes? How the tree is constructed based on this number of nodes?\n\n- Experimental results are quite promising as expected. I wonder how TPO and PPO are compared in terms of the total computation time? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxaxoUqqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Tree search Policy Optimization (TPO) algorithm for tasks with continuous action spaces. TPO works after a well trained policy can be obtained (PPO is used in the paper). After a well trained policy is obtained, TPO firstly uses the policy to do Monte Carlo Tree Search (MCTS), with at most 32 actions at each node (state) in the tree (to make MCTS work for continuous action spaces). Then after searching, policy is updated by matching the statistics of MCTS policy (mean and variance), since usually MCTS induces a better policy than the current sampling policy. Experiments on MuJoCo tasks show that TPO has performance improvement over PPO.\n\n1. This paper is basically a special case of dual policy iteration methods (as mentioned), with some heuristics to make MCTS work in continuous action spaces. Unlike dual policy iteration, there is no theoretical justification why doing these heuristics are good/convincible.\n\n2. As a paper focusing on experiments, the results are not enough, in the following senses,\n\na) There are other existing work of MCTS in continuous action spaces, but they are not mentioned, like\n\"Monte Carlo Tree Search in Continuous Action Spaces with Execution Uncertainty\", Yee et al., 2016.\n\"Monte Carlo Tree Search for Continuous and Stochastic Sequential Decision Making Problems\", Couetoux, 2013.\nWithout comparing with other baselines, the current experiments are not convincing to claim the proposed TPO method is a better choice over other existing methods.\n\nb) The experiments only show application of TPO on PPO, which does not indicate that TPO can improve performances of other methods, like Soft Actor-Critic (SAC), and \"Relative Entropy Regularized Policy Iteration\", Abdolmaleki et al, 2018 (RERPI), which are known as better choices for Mujoco tasks. Actually, I was wondering how can TPO work with other true off-policy algorithms like the mentioned SAC and RERPI.\n\nc) The learning results/curves seem to be not efficient. The maximum timestep is 4M. What is timestep? Is it the number of iterations? Does that mean the environment steps are totally 4M *32 = 128 M (since every step there are 32 envs in parallel). Please clarify this, and also show how many environment steps (totally how many number of actions have been taken) are there for PPO here. If their environment steps are not the same, then this comparison is not fair (TPO actually uses more actions). And if 128 M environments are used, then the learning is quite inefficient, for example, the final score for Ant of TPO is about 5000, while SAC achieves 6000 after 3 M environment steps.\n\n3. The MCTS policy update stage (second stage) uses mean and variance matching to update policy Eq. (8). I did not see why this objective is good. There is no intuition or comparison to support it. For the Gaussian policy here, KL divergence between MCTS policy and the current policy induces another different matching of mean and variance. How is this objective compared with KL? Please use experiments to justify.\n\n4. TPO works under restricted requirements (require well performed policies), which makes the claimed contribution of \"making MCTS work for continuous action spaces\" weak (nearly not hold). Actually, the experiments show that if rho = 0.1, there is no good enough policy, then the proposed method does not work at all. This means the contribution is \"MCTS works for continuous action spaces in special cases, that a good policy can be used for sampling\". The 32 branch factor is enough also because of this reason. The MCTS in discrete action space is guaranteed to converge (UCT algorithm). However, here there is no evidence that this proposed method will achieve similar results if it starts from initialization rather than well trained policies. From this perspective, I consider this method not really a method that \"makes MCTS work for continuous action spaces.\"\n\n5. What is the replay buffer size for MCTS? Are the trajectories in MCTS buffer going to be used again (or thrown away after calculate the mean and variance of MCTS policy)? I suppose the simulations will be just used for once and the MCTS tree will be thrown away (next iteration a new tree will be constructed). If this is the case, then the proposed method is not really a true \"off-policy\" method as claimed. It does not have the same sample efficiency as other off-policy methods (like SAC, DQN, DDPG), and it does not need to face the same difficulty like importance ratio corrections. Therefore, it is more a policy update step within dual policy iteration framework, rather than true off-policy learning (with replay buffer storing trajectories, and those trajectories will be reused for multiple times.)\n\n6. TPO requires knowledge of maximum timestep, meaning it is not an any-time algorithm (unlike most existing algorithms).\n\nOverall, this paper has no contributions on theory. And I found the experiments cannot show: 1) the proposed TPO is better than other baselines; 2) TPO can work with other methods, especially off-policy algorithms; 3) the learning efficiency, proposed objective/algorithm are questionable; 4) the proposed method is not really a method that makes MCTS work for continuous action spaces, except with restricted requirements.\n\n\n=====Update=====\nI have read the rebuttal and I keep my rating, since there is no revision to see any improvements.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a tree search based policy optimization methods for continuous action state spaces. The paper does not have a theoretical guarantee, but has empirical results.

Reviewers brought up issues such as lack of using other policy optimizations methods (SAC, RERPI, etc.), sample inefficiency, and unclear difference with some other similar papers. Even though the authors have provided a rebuttal to address these issues, all the reviewers remain negative. So I can only recommend rejection at this stage.
Decision: Reject