Title: TPO: TREE SEARCH POLICY OPTIMIZATION FOR CONTINUOUS ACTION SPACES

Abstract: Monte Carlo Tree Search (MCTS) has achieved impressive results on a range of discrete environments, such as Go, Mario and Arcade games, but it has not yet fulfilled its true potential in continuous domains. In this work, we introduce TPO, a tree search based policy optimization method for continuous environments. TPO takes a hybrid approach to policy optimization. Building the MCTS tree in a continuous action space and updating the policy gradient using off-policy MCTS trajectories are non-trivial. To overcome these challenges, we propose limiting tree search branching factor by drawing only a few action samples from the policy distribution and defining a new loss function based on the trajectories' mean and standard deviations. Our approach led to some non-intuitive findings. MCTS training generally requires a large number of samples and simulations. However, we observed that bootstrapping tree search with a pre-trained policy allows us to achieve high quality results with a low MCTS branching factor and few simulations. Without the proposed policy bootstrapping, continuous MCTS would require a much larger branching factor and simulation count, rendering it prohibitively expensive.
In our experiments, we use PPO as our baseline policy optimization algorithm. TPO significantly improves the policy on nearly all the environments. For example, in complex environments such as Humanoid with a 17 dimensional action space, we achieve a 2.5× improvement over the baseline algorithm.

1. INTRODUCTION:

Fueled by advances in neural representation learning, the field of model-free reinforcement learning has rapidly evolved over the past few years. These advances are due in part to the advent of algorithms capable of navigating larger action spaces and longer time horizons [2, 32, 42] , as well as the distribution of data collection and training across massive-scale computing resources [42, 38, 16, 32] .

While learning algorithms have been continuously improving, it is undeniable that tree search methods have played a large role in some of the most successful applications of RL (e.g., AlphaZero [42] , Mario [10] and Arcade games [48] ). Tree search methods enable powerful explorations of the action space in a way which is guided by the topology of the search space, focusing on branches (actions) that are more promising. Although tree search methods have achieved impressive results on a range of discrete domains, they have not yet fulfilled their true potential in continuous domains. Given that the number of actions is inherently unbounded in continuous domains, traditional approaches to building the search tree become intractable from a computational perspective.

In this paper, we introduce TPO, a Tree Search Policy Optimization for environments with continuous action spaces. We address the challenges of building the tree and running simulations by adopting a hybrid method, in which we first train a policy using existing model-free RL methods, and then use the pre-trained policy distribution to draw actions with which to build the tree. Once the tree has been constructed, we run simulations to generate experiences using an Upper Confidence Bounds for Trees (UCT) approach [33] . Populating the tree with the action samples drawn from a pre-trained policy enables us to perform a computationally feasible search. TPO is a variation of the policy iteration method [35, 44, 42, 1] . Broadly, in these methods, the behavior of policy is iteratively updated using the trajectories generated by an expert policy. Then, the newly updated policy in return guides the expert to generate higher quality samples. In TPO, we use tree search as an expert to generate high quality trajectories. Later, we employ the updated policy to re-populate the tree search. For tree search, we use the Monte Carlo Tree Search (MCTS) [5] expansion and selection methods. However, it is challenging to directly infer the probability of selected actions for rollout; unlike in discrete domains where all actions can be exhaustively explored, in continuous domains, we cannot sample more than a subset of the effectively innumerable continuous action space. Furthermore, to use the trajectories generated by MCTS, we must perform off-policy optimization. To address this challenge, we define a new loss function that uses the weighted mean and standard deviation of the tree search statistics to update the pre-trained policy. For ease of implementation and scalability, we use Proximal Policy Optimization (PPO) [38] and choose as our policy optimization baseline. Our approach led to some non-intuitive findings. MCTS training generally requires a large number of branches and simulations. For example, AlphaGo uses 1600 simulations per tree search and a branching factor of up to 362 [40] . However, we observed that if we pre-train the policy, we require far fewer simulations to generate high quality trajectories. While we do benefit from exploring a greater number of branches, especially for higher dimensional action spaces (e.g. Humanoid), we observed diminishing returns after only a small number of branches (e.g., 32) across all of the evaluated environments. Furthermore, performance quickly plateaued as we increased the number of simulations past 32. This property did not hold when we initialized tree search with an untrained policy. This is a critical advantage of our method as it would otherwise be computationally infeasible to generate high quality trajectories using tree search. The main contributions of TPO are summarized as follows:

1. Tree search policy optimization for continuous action spaces. TPO is one of the very first techniques that integrates tree search into policy optimization for continuous action spaces. This unique integration of tree search into policy optimization yields a superior performance compared to baseline policy optimization techniques for continuous action spaces. 2. Policy bootstrapping. We propose a policy bootstrapping technique that significantly improves the sample efficiency of the tree search and enables us to discretize continuous action spaces into only a few number of highly probable actions. More specifically, TPO only performs 32 tree searches compared to substantially larger number of tree searches (1600, 50× more) in AlphaGo [42] . In addition, TPO narrows down the number of tree expansion (actions) compared to discretization techniques such as Tang et al. [45] which requires 7-11 bins per action dimension. This number of bins translates to a prohibitively large number of actions even in discrete domain for complex environments such as Humanoid which has a 17 dimensional action space. In contrast, TPO only samples 32 actions at each simulation step across all the environments. 3. Infrastructure and results. On the infrastructure side, we developed a distributed system (shown in Figure 1 ), in which both policy optimization and data collection are performed on separate distributed platforms. The policy optimization is done on a TPU-v2 using multiple cores, and MCTS search is performed on a rack of CPU nodes. A synchronous policy update and data collection approach is used to train the policy and generate trajectories. TPO readily extends to challenging and high-dimensional tasks, such the Humanoid benchmark [9] . Our empirical results indicate that TPO significantly improves the performance of the baseline policy optimization algorithm, achieving up to 2.5× improvement.

2. BACKGROUND:

Policy iterations methods have shown promising results in reinforcement learning. AlphaZero [42] , a major breakthrough in reinforcement learning, uses MCTS as an expert to train a discrete policy model for the game of Go. EXiT [1] established theoretical grounds for policy iterations using MCTS in discrete domains. A recent work [28] proposes some theoretical extensions to AlphaZero's MCTS method [42] for environments with continuous action spaces. However, the authors only evaluate their method on the Pendulum OpenAI Gym environment which has only a one dimensional action space. It is not obvious how the approach can scale to more complex environments with much higher dimensions. While TPO also belongs to the category of policy iteration methods, it studies a rather less explored territory of using MCTS as expert for complex environments with continuous action spaces.

Policy optimization falls into two major categories: on-policy and off-policy optimization. In on-policy methods, the policy is optimized using samples generated from the current policy, whereas with off-policy algorithms, we can also train the policy using experiences from other sources, such as an expert. Although on-policy reinforcement learning methods, such as TRPO [36] , PPO [38] , PGQL [31] , and A3C [27] have been extremely successful, they are known to be sample inefficient [16, 12] , as they must generate fresh trajectories between each gradient update and cannot reuse these trajectories. This effect becomes more pronounced as the complexity of the task grows, requiring millions of steps to learn a policy. On the other hand, off-policy methods (e.g., DQN [26, 27] , DDPG [24] ) are more sample efficient, as each example is used multiple times for training. However, these methods require more extensive hyper-parameter tuning, suffering from instability and slow convergence [30, 18, 16, 34, 47] .

A few recent methods have been developed to embed differentiable planning procedures in computation graphs [15, 43, 25, 14, 19] . MCTSnets [15] is a deep learning model that incorporates simulation-based search with a vector embedding for expansion, evaluation, and backup. The three policies are jointly optimized with an end-to-end gradient based approach. The authors only report results on the classic planning benchmark, Sokoban, which is a simple discrete optimization environment. Extension of this approach to continuous domains is non-obvious. SoRB [11] is another approach that combines planning with policy optimization. However, the target problem space of SoRB is only goal-reaching tasks. Although there has been previous work on continuous MCTS, none has been conducted in the context of deep reinforcement learning [8, 13, 7] . More sophisticated approaches to node expansion improve the quality of search [7] , but TPO only uses a simple scheme in which a fixed number of actions are sampled from the policy. We found our approach both simple to implement and effective, without the need for extensive fine-tuning or hyper-parameter optimization. For example, we found that across the evaluated OpenAI Gym environments, a sample size of 32 yields strong results. In our experiments, we chose PPO [38] , which has been shown to yield superior performance in various large-scale problems such as Hide and Seek [2] and Dota [32] , to pre-train the policy used to populate the MCTS tree. However, our framework is independent of the specific choice of baseline policy and can be readily applied to other model-free RL algorithms, such as SAC [16] or TD3 [34] .

3. MONTE CARLO TREE SEARCH FOR CONTINUOUS ACTION SPACES:

Preliminaries and notations. Policy optimization in continuous environments can be well modeled by a Markov Decision Process (MDP) [3] , defined by the tuple (S,A,p,r,γ), where S(A) is a set of states (actions) in continuous space. The state transition probability p(s t+1 ,s t ,a k ) represents the probability desnsity of observing the next state in time, s t+1 ∈ S, given the current s t ∈ S and the current action a k ∈ A. Taking action a k at state s t , the environment moves to a new state s t+1 and returns a bounded reward r(s t ,a k ) : S ×A → [r min ,r max ]. Finally, γ ∈ (0,1) represents the discount factor, used to calculate the discounted cumulative expected reward. We use π θ (a k |s t ) to represent the neural network policy with trainable parameters θ. We use v φ (s t ) to represent the value from a neural network with parameters φ.

3.1. MONTE CARLO TREE SEARCH ALGORITHM FOR CONTINUOUS ACTION SPACE:

Previous work on Monte Carlo Tree Search (MCTS) was mostly focused on zero-sum games with discrete action spaces [42, 41, 29, 40, 5, 8, 22, 39] . In this section, we will highlight our adaptation of MCTS for environments with continuous action spaces and non-zero immediate rewards. For the complete definition of MCTS, we invite the readers to see [42] . To make MCTS possible, we assume the environment can be set to an arbitrary state. This assumption can be relaxed using a learned model of the environment (See §6).

In this work, we use a variant of Polynomial Upper Confident Tree (PUCT) algorithm in TPO for MCTS [33] . Nodes of the Monte Carlo tree are states of the environment s t ∈ S and edges are actions, a k ∈ A. We store a set of statistics for each edge of Monte Carlo tree: the number of visits N , the cumulative sum of action value W , and the average of the action value Q. We update these statistics after each MCTS simulation for the traversed edges. Each MCTS simulation starts from a root state and consists of four steps: select, expand, evaluate, and backup. Here we only overview the select and expand steps, as they are the contributions of TPO to enable tree search for continuous action spaces.

1 Select. We follow a variant of Polynomial Upper Confident Tree (PUCT) algorithm [33, 42] to select actions. For each node, s t , we select an action:

EQUATION

where c puct is a constant exploration coefficient and π θ (a k |s t ) represents the prior probability assigned to each action by the policy network. To further promote exploration, we add a Dirichlet noise, Dir(α) 1 , to the prior probabilities of the actions at the root node [42] .

To support environments with immediate rewards, we propose to use the following equation for estimating the value of an action:

EQUATION

where W (s t ,a k ) is the cumulative sum over the backed up values from its children.

2 Expand. In discrete environments such as the game of Go [42] or Chess/Shogi [41] , all possible actions are considered in MCTS expansion. However, in environments with continuous action spaces, the set of possible actions is innumerable. Hence, enumerating all the possible actions for each node is impossible. To circumvent this challenge, we propose to draw samples from the policy distribution. These sampled actions will become the edges of the leaf node.

After a fixed number of simulations, we select an action with the largest visit counts from the current root node and advances the environment. We use Q(s t ,a k )+U (s t ,a k ) as a tie-breaker between the actions with the same number of visit counts. We also define μT S (s t ) as an unbiased estimate [46, 27] for the mean value of the tree policy at a given node computed across each action space dimension:

EQUATION

Later in Section 4, we use this unbiased mean estimate to optimize the policy distribution. All MCTS trajectories are added to a replay buffer, which is then used to optimize the policy and value networks.

4. TREE SEARCH POLICY OPTIMIZATION (TPO):

Neural network architecture. Both value, v φ (.), and policy networks, π θ (.), are represented by a fully-connected multi-layer perceptron (MLP) with two hidden layers of 64 units. The output of each layer is passed through a tanh(•) non-linearity function. The weights of value and policy networks are not shared. The value network outputs a single scalar value v(s t ), whereas the policy network outputs a vector of means µ(s t ), and variances σ(s t ) for a number of independent Normal distributions for each action dimension.

Loss functions. We first describe the loss functions for the on-policy phase and then delve into the loss functions of the proposed off-policy tree search optimization. For ease of implementation, we used PPO [38] for on-policy optimization. The policy loss function of PPO is:

EQUATION

where r t (θ) is the probability ratio between the new policy, π θt (a k |s t ), and the old policy, π θ old (a k |s t ), PPO clips the probability ratio to be within the range [1-, 1+ ], which penalizes large updates to the policy. Ât is the truncated generalized advantage estimate [38, 37] , computed for a trajectory of length T . To update the value network, we use a variant of PPO2 that also applies clipping to the value function: 

EQUATION

Environments State Space Action Space:

Ant-v2

EQUATION

Finally, the total loss combines the policy surrogate and value function terms with a regularization coefficients c 1 :

EQUATION

In the off-policy phase, we use the same value function as PPO for the value updates, as shown in Eq. 5 and Eq. 6. For the policy updates, we define a new loss function, derived from the collected MCTS statistics:

EQUATION

L PF (θ) is the sum of the squared-error loss between the unbiased mean estimate and the standard deviation estimate, calculated from the MCTS trajectories, and the mean and standard deviation of the policy distribution at a given state s t . The intuition for choosing this loss term for policy updates is to incentivize moving the mean of the policy distribution towards an unbiased estimate computed from the tree search. Finally, we combine the defined TPO policy loss and value loss to obtain the following objective, which is maximized at each training step:

EQUATION

If the policy and value networks do not share parameters, the regularization coefficients c 1 and c 3 become irrelevant. In neither L On-π or L Off-π losses, we do not use any entropy bonus term.

Algorithm. Algorithm 1 fleshes out the proposed TPO algorithm, which uses a combination of policy gradient and MCTS. In the first phase of training, we use the policy network to roll out trajectories and optimize the PPO loss function (See Eq. 7). In the second phase, we switch to an off-policy mode, in which we only use trajectories from MCTS to optimize the policy. That is, we construct the loss (See Eq. 9.

) on the roll out trajectories generated by the tree search. We treat ρ, which defines the switching ratio between policy gradient and tree search, as a hyperparameter of the TPO algorithm. The intuition behind bootstrapping tree search with a pre-trained policy is that continuous action spaces require a relatively stable policy to encourage the exploration of trajectories with higher returns, and an untrained policy cannot adequately guide the tree search. Our experiments support this hypothesis; if we start from a lightly trained policy (i.e. small switching ratio, ρ), the final performance is severely diminished. The same reasoning applies to choosing to start with policy gradient optimization in the first phase of training followed by tree search policy optimization.

5. EXPERIMENTS:

We evaluate TPO on a variety of continuous environments from OpenAI Gym [4] (Table 1 ). Table 2 show the hyperparameters used for training the value and policy networks and those used for tree search.

Our policy training is done on a TPU-v2 platform with eight cores, and MCTS is performed on CPU. We chose PPO [38] for the on-policy optimization update. For each environment, we run TPO for four million time steps to estimate the gradients.

To efficiently perform MCTS sampling for trajectory rollouts, we use BatchEnv [17] that extends the OpenAI Gym [4] interface to multiple parallel environments. Using BatchEnv, we evaluate up to 32 actions in parallel and obtain their associated rewards. We use 32 Intel CPUs, each assigned to one environment instance, in order to parallelize simulation. We found that 32 is a reasonable choice for the maximum number of MCTS simulations per each action selection, as it strikes a good balance between search latency and policy performance. Using our parallel infrastructure, the latency of action selection is somewhere between 0.2-0.3 sec when we use 32 MCTS simulations per action. Unless stated otherwise, in all of the experiments, we report the average return of the last 100 episodes across five runs (each with a different random seed). To make a fair comparison with the baseline algorithm (PPO), we reproduce the reported results in [38] using our infrastructure. In all the environments, we use the best obtained results for PPO using our infrastructure, which in some environments (HalfCheetah-v2 and Walker2d-v2) outperform the reported results in [38] . In all the experiments, we use Adam [21] as our optimizer and run training for 10 epochs per policy update.

Summary of results. Table 3 depicts the performance comparison between TPO and PPO across six environments with continuous action spaces. In all the environments, TPO yields a better total return compared to PPO, generally by a significant margin. Using TPO, we observe a maximum improvement of roughly 2,900 in Ant-v2 and a minimum improvement of only 7 in Swimmer-v2. Swimmer-v2 is the simplest environment in the set of evaluated benchmarks with an action space of only two dimensions. On the other hand, Ant-v2 is a much more complex benchmark with an action-space of eight dimensions. The delta in performance improvement suggests that TPO is better able to leverage tree search exploration to improve policies in complex environments with large action spaces. Figure 2 shows the training progress of TPO and PPO across six environments. To conduct this experiment, we start by pre-training with PPO. At ∼ 2M time steps, TPO restores checkpoints from PPO training and continues training using tree search. As such, the total return curve for both TPO and PPO overlap before reaching 2M time steps. In all the environments, we observe a sharp spike in the total return curve after switching to tree search, which can be attributed to the effectiveness of the proposed tree search method for policy optimization. For complex environments, such as Humanoid-v2, we observe continuing improvement after switching to MCTS-based policy optimization. Ablation Studies To better understand the impact of key components of TPO, we perform a set of ablation studies. Concretely, we show the results for three environments with different levels of action space complexity, namely Hopper-v2, Ant-v2, and Humanoid-v2 with action spaces of three, eight, and 17, respectively.

Switching ratio. First, we study the impact of varying the switching ratio (ρ in Algorithm 1) between 10%, 50% (default value in TPO), and 90% across three tasks. As shown in Figure 3 , prematurely switching (ρ = 10%, blue line in Figure 3 ) to tree search optimization severely harms policy performance, showing no further improvement after switching to MCTS optimization. This behavior can be attributed to the fact that in tree search policy optimization for environments with continuous action spaces, it is of utmost importance to have a reasonably well-trained policy to guide the tree search toward trajectories with potentially higher return values. Switching to tree search at the very end of training (ρ = 10%, green line in Figure 3 ) can be damaging as well. While switching to tree search at the 90% of the total training steps in Hopper-v2 does not improve the average return compared to switching after 50% of the total training steps, it slightly improves the average return in Ant-v2. That is, the point at which to switch to tree search is a hyperparameter that should be adjusted carefully for better policy performance.

Number of actions per node. Figure 4 shows the impact of varying the number of actions per node in tree search between 4, 8, and 32 (default in TPO is 32). This study reveals the following findings. (1) variance reduction in return values; increasing the number of actions per node significantly reduces the variance of the total return value (i.e. shrinkage in the shaded area), which further improves the robustness of the training process. (2) better policy performance (higher return value); while the impact of increasing the number of actions per node is marginal in simple environments, such as Hopper-v2 and Ant-v2, exploring more actions per node in environments with a more complex action space, Humanoid-v2, has significant benefit for policy performance. As shown in Figure 4c , changing the number of actions per node from eight to 32 increases the average total return by ∼1100.

Number of MCTS simulations. Figure 5 shows the impact of the number of MCTS simulations (See Section 3) on the performance of the policy. While increasing the number of MCTS simulations generally improves performance, we found that increasing the branching factor (number of actions per node) had more impact. For example, for Humanoid-v2, the performance gap between 8 and 32 simulations (shown in Figure 5c ), is less pronounced than the improvement from increasing the number of actions per node from 8 and 32 (shown in Figure 4c ). These studies suggest that it is more beneficial to devote the tree search budget to exploring more actions per node, especially for more complex environments, rather than conducting a larger number of tree search simulations.

6. DISCUSSION:

In this paper, we have studied Monte Carlo Tree Search in continuous space for improving the performance of a baseline on-policy algorithm [38] . Our results show that MCTS policy optimization can indeed improve the quality of policy in choosing better actions during policy evaluation at the cost of more samples during MCTS rollout. We show that bootstrapping tree search with a pretrained policy enables us to achieve high performance with a low MCTS branching factor and few simulations. On the other hand, without pre-training, we require a much larger branching factor and simulation count, rendering MCTS computationally infeasible. One of the future research direction is to explore techniques for improving the sample efficiency and removing the need for having a reset-able environment. To achieve these goals, we can use a trained model of the environment similar to model-based reinforcement learning approaches [20, 23, 6] , instead of interacting directly with the environment in MCTS. Recently, MBPO [20] showed that they can train a model of Mujoco environments that is accurate enough for nearly 200-step rollouts in terms of accumulated rewards. This level of accuracy horizon is more than enough for the shallow MCTS simulations (32 simulations) that is employed in TPO. As mentioned earlier, TPO assumes access to an environment that can be restarted from an arbitrary state for MCTS simulations. While this assumption can be readily satisfied in some RL problems such as playing games, it may be harder to achieve for physical RL problems like Robotics. This assumption can also be relaxed using a modeled environment to replace the interactions with the real environment during MCTS simulations.

:

Dir(α) returns a noise vector of {η1,η2,...,ηn} where n i=1 ηi = 1. α ∈ R + is Dirichlet noise scaling parameter that controls the shape of the generated noise vector.

https://github.com/openai/baselines/tree/master/baselines/ppo2

Figures and Tables:

Figure 1: Tree Search Policy Optimization (TPO) Framework. In phase 1, we perform a policy gradient based optimization training to build a target policy. In phase 2, we iteratively build an MCTS tree using the pre-trained target policy and update the target policy using roll-out trajectories from MCTS. Both training and data collection are done in a distributed manner.

Figure 2: Comparison between PPO and TPO training curves. Each data point represents the average return of the last 100 training episodes. MCTS rollout starts after 50% of the total number of timesteps, i.e. ∼2M.

Figure 3: TPO training curves when the switching ratio (ρ in Algorithm 1) is 10%, 50% (default value in TPO), and 90%.

actions per node = 4 # actions per node = 8 # actions per node = 32 (a) Hopper-v2a k ∈ R actions per node = 4 # actions per node = 8 # actions per node = 32 (b) Ant-v2a k ∈ R actions per node = 4 # actions per node = 8 # actions per node = 32 (c) Humanoid-v2a k ∈ R 17

Figure 4: TPO training curves with varying the number of actions per node.

Figure 5: TPO training curves with varying number of MCTS simulations.

Algorithm 1: TPO: Tree Search Policy Optimization 1 for iteration ← 1, 2, ..., number of iterations do The benchmark suite (six OpenAI Gym[4] environments) with their corresponding observation/action spaces.

Table content: <table><tr><td>2</td><td>if iteration≤ρ × number of iterations</td></tr></table>

TPO hyperparameters.

Table content: <table><tr><td>MCTS Hyperparameters</td><td>Value</td></tr><tr><td>Number of MCTS simulations</td><td>32</td></tr><tr><td>Maximum number of actions per node</td><td>32</td></tr><tr><td>cpuct</td><td>3</td></tr><tr><td>Temperature (τ )</td><td>5</td></tr><tr><td>Dirichlet noise scaling parameter (α)</td><td>1.75</td></tr><tr><td>Dirichlet noise weight (ω)</td><td>0.5</td></tr><tr><td>Training Hyperparameters</td><td>Value</td></tr><tr><td>Policy network architecture</td><td>FC:64 -FC:64</td></tr><tr><td>Value network architecture</td><td>FC:64 -FC:64</td></tr><tr><td>Adam learning rate (η)</td><td>3×10 -4 and 9×10 -4</td></tr><tr><td>Horizon (T)</td><td>2048</td></tr><tr><td>Switching ratio (ρ)</td><td>0.5</td></tr><tr><td>Clipping coefficient ( )</td><td>0.2</td></tr><tr><td>Number of epochs</td><td>10</td></tr><tr><td>Minibatch size (M)</td><td>128</td></tr><tr><td>Gradient norm clipping coefficient</td><td>0.5</td></tr><tr><td>Discount (γ)</td><td>0.99</td></tr><tr><td>GAE parameter (λ)</td><td>0.95</td></tr></table>

Performance comparison between TPO and PPO after 4M time-steps. ∆ values show the difference between TPO and PPO for each category. A positive ∆ value indicates that TPO outperforms PPO.

Table content: <table><tr><td>Benchmarks</td><td>PPO</td><td>Min. TPO</td><td>∆</td><td>PPO</td><td>Avg. TPO</td><td>∆</td><td>PPO</td><td>Max. TPO</td><td>∆</td></tr><tr><td>Ant-v2</td><td>1722.75</td><td colspan="3">4455.57 +2732.81 1873.85</td><td colspan="3">4777.06 +2903.21 2022.08</td><td>5011.08</td><td>+2989.0</td></tr><tr><td colspan="2">HalfCheetah-v2 1658.70</td><td>1788.77</td><td>+130.06</td><td>2530.81</td><td colspan="3">3949.05 +1418.24 5049.07</td><td colspan="2">6734.45 +1685.38</td></tr><tr><td>Hopper-v2</td><td>1744.24</td><td>2205.47</td><td>+461.22</td><td>1937.82</td><td colspan="3">3196.14 +1258.32 2051.89</td><td colspan="2">3613.36 +1561.47</td></tr><tr><td>Humanoid-v2</td><td>4269.44</td><td colspan="3">5391.43 +1121.99 4439.58</td><td colspan="3">5460.93 +1021.35 4609.72</td><td>5530.43</td><td>+920.71</td></tr><tr><td>Swimmer-v2</td><td>109.91</td><td>119.48</td><td>+9.57</td><td>113.74</td><td>120.81</td><td>+7.07</td><td>116.74</td><td>122.39</td><td>+5.65</td></tr><tr><td>Walker2d-v2</td><td>3605.27</td><td>3788.83</td><td>+183.56</td><td>3942.57</td><td>4092.21</td><td>+149.63</td><td>4155.37</td><td>4667.64</td><td>+512.27</td></tr></table>

Peer review (iclr2020):
This paper proposes Tree search Policy Optimization (TPO) algorithm for tasks with continuous action spaces. TPO works after a well trained policy can be obtained (PPO is used in the paper). After a well trained policy is obtained, TPO firstly uses the policy to do Monte Carlo Tree Search (MCTS), with at most 32 actions at each node (state) in the tree (to make MCTS work for continuous action spaces). Then after searching, policy is updated by matching the statistics of MCTS policy (mean and variance), since usually MCTS induces a better policy than the current sampling policy. Experiments on MuJoCo tasks show that TPO has performance improvement over PPO. 1. This paper is basically a special case of dual policy iteration methods (as mentioned), with some heuristics to make MCTS work in continuous action spaces. Unlike dual policy iteration, there is no theoretical justification why doing these heuristics are good/convincible. 2. As a paper focusing on experiments, the results are not enough, in the following senses, a) There are other existing work of MCTS in continuous action spaces, but they are not mentioned, like *Monte Carlo Tree Search in Continuous Action Spaces with Execution Uncertainty*, Yee et al., 2016. *Monte Carlo Tree Search for Continuous and Stochastic Sequential Decision Making Problems*, Couetoux, 2013. Without comparing with other baselines, the current experiments are not convincing to claim the proposed TPO method is a better choice over other existing methods. b) The experiments only show application of TPO on PPO, which does not indicate that TPO can improve performances of other methods, like Soft Actor-Critic (SAC), and *Relative Entropy Regularized Policy Iteration*, Abdolmaleki et al, 2018 (RERPI), which are known as better choices for Mujoco tasks. Actually, I was wondering how can TPO work with other true off-policy algorithms like the mentioned SAC and RERPI. c) The learning results/curves seem to be not efficient. The maximum timestep is 4M. What is timestep? Is it the number of iterations? Does that mean the environment steps are totally 4M *32 = 128 M (since every step there are 32 envs in parallel). Please clarify this, and also show how many environment steps (totally how many number of actions have been taken) are there for PPO here. If their environment steps are not the same, then this comparison is not fair (TPO actually uses more actions). And if 128 M environments are used, then the learning is quite inefficient, for example, the final score for Ant of TPO is about 5000, while SAC achieves 6000 after 3 M environment steps. 3. The MCTS policy update stage (second stage) uses mean and variance matching to update policy Eq. (8). I did not see why this objective is good. There is no intuition or comparison to support it. For the Gaussian policy here, KL divergence between MCTS policy and the current policy induces another different matching of mean and variance. How is this objective compared with KL? Please use experiments to justify. 4. TPO works under restricted requirements (require well performed policies), which makes the claimed contribution of *making MCTS work for continuous action spaces* weak (nearly not hold). Actually, the experiments show that if rho = 0.1, there is no good enough policy, then the proposed method does not work at all. This means the contribution is *MCTS works for continuous action spaces in special cases, that a good policy can be used for sampling*. The 32 branch factor is enough also because of this reason. The MCTS in discrete action space is guaranteed to converge (UCT algorithm). However, here there is no evidence that this proposed method will achieve similar results if it starts from initialization rather than well trained policies. From this perspective, I consider this method not really a method that *makes MCTS work for continuous action spaces.* 5. What is the replay buffer size for MCTS? Are the trajectories in MCTS buffer going to be used again (or thrown away after calculate the mean and variance of MCTS policy)? I suppose the simulations will be just used for once and the MCTS tree will be thrown away (next iteration a new tree will be constructed). If this is the case, then the proposed method is not really a true *off-policy* method as claimed. It does not have the same sample efficiency as other off-policy methods (like SAC, DQN, DDPG), and it does not need to face the same difficulty like importance ratio corrections. Therefore, it is more a policy update step within dual policy iteration framework, rather than true off-policy learning (with replay buffer storing trajectories, and those trajectories will be reused for multiple times.) 6. TPO requires knowledge of maximum timestep, meaning it is not an any-time algorithm (unlike most existing algorithms). Overall, this paper has no contributions on theory. And I found the experiments cannot show: 1) the proposed TPO is better than other baselines; 2) TPO can work with other methods, especially off-policy algorithms; 3) the learning efficiency, proposed objective/algorithm are questionable; 4) the proposed method is not really a method that makes MCTS work for continuous action spaces, except with restricted requirements. =====Update===== I have read the rebuttal and I keep my rating, since there is no revision to see any improvements.