Title: EXPLORING BY EXPLOITING BAD MODELS IN MODEL-BASED REINFORCEMENT LEARNING Anonymous

Abstract: Exploration for reinforcement learning (RL) is well-studied for model-free methods but a relatively unexplored topic for model-based methods. In this work, we investigate several exploration techniques injected into the two stages of modelbased RL: (1) during optimization: adding transition-space and action-space noise when optimizing a policy using learned dynamics, and (2) after optimization: injecting action-space noise when executing an optimized policy on the real environment. When given a good deterministic dynamics model, like the groundtruth simulation, exploration can significantly improve performance. However, using randomly initialized neural networks to model environment dynamics can implicitly induce exploration in model-based RL, reducing the need for explicit exploratory techniques. Surprisingly, we show that in the case of a local optimizer, using a learned model with this implicit exploration can actually outperform using the ground-truth model without exploration, while adding exploration to the ground-truth model reduces the performance gap. However, the learned models are highly local, in that they perform well only for the task for which it is optimized, and fail to generalize to new targets.

1. INTRODUCTION:

Reinforcement learning (RL) with deep neural networks combines two extremely data inefficient techniques, increasing the need for efficient exploration. In model-free methods, exploration is usually done by adding noise to the action suggested by the optimized policy. Here, exploration helps primarily by encouraging the agent to visit new states that might have better reward. On the other hand, model-based RL (MBRL) has been more focused on reaching parity with model-free methods in asymptotic performance on environments with high-dimensional state-and action-spaces (Chua et al. (2018) ). As a result, exploration in MBRL is a relatively nascent field, though it is still thought to help by generating diverse environment data to train the dynamics model (Pathak et al. (2019) ). However, exploration methods designed for the model-free case may result in non-trivial, potentially problematic, effects when used in MBRL.

In this work, we take a step back and analyze how exploration behavior can be injected into MBRL, and extensively evaluate the effect of various exploration strategies on learning performance. Most MBRL approaches iterate between two phases for learning a task: 1) simulation/planning phase: using the current approximation of the dynamics, optimize a policy which accomplishes the task, and 2) environment interaction phase: the optimized policy is executed in the real world, which allows the agent to collect new observations; the approximate dynamics model is then updated with these new observations. Exploration can be injected during both phases. First, exploration can take place during the realworld execution phase by perturbing the optimized actions, similar to how many model-free methods implement exploration. We call this exploration type action-space exploration. Second, exploration can also be performed during the planning phase, either by performing action-space exploration during policy optimization itself or by considering the distribution of predicted trajectories, and using a risk-seeking approach to encourage the exploration of parts of the model with high uncertainty. We call this type of exploration transition-space exploration.

In this paper, we thoroughly investigate the effect of these different types of exploration strategies on the performance of MBRL. We especially focus our analysis on algorithms that use the itera-tive Linear Quadratic Regulator (iLQR) (Li & Todorov (2004) ) as the policy optimization method; a well-established approach both in RL and optimal control settings for real robotic applications (Levine et al. (2016) ; Bechtle et al. (2019) ; Viereck et al. (2018) ; Neunert et al. (2016) ). This choice is motivated by our desire to focus on methods that are demonstrably applicable to real robotic motor control problems. We focus our analysis to a reaching task for a 7-DoF arm. Our experimental results lead to the following surprising observations:

• Action-space exploration, especially the commonly-cited iLQR solution for maximumentropy RL, results in significant deterioration of performance.

• Some exploration methods can improve performance of iLQR with a ground-truth dynamics model, but do not improve performance for iLQR using learned dynamics models.

• In our setting, using learned models with iLQR results in implicit exploration that can lead to better performance than using a ground-truth dynamics model. In the process of learning a model, iLQR is able to escape local minima despite a poor initialization, while iLQR with a good model is more sensitive to initialization.

We conclude that certain choices for modeling dynamics, such as randomly initialized neural networks, implicitly provide an exploration mechanism in model-based RL and reduce the need and benefit of explicit exploration. However, policy optimization with using a strong, deterministic model, such as the ground-truth dynamics, can benefit from exploration because they lack this implicit exploration mechanism.

2. RELATED WORK:



2.1. EXPLORATION FOR MODEL-FREE RL:

Model-free methods have worked exceptionally well using only simple forms of exploration, such as injecting noise in different stages of a greedy (short-term optimal) algorithm. Noise can be injected in the action space for the discrete case through -greedy exploration (Sutton et al. (1998) ) and for the continuous case through additive Gaussian noise or by sampling directly from stochastic policies. Alternatively, noise can be injected into the parameter-space of the policy (Rückstieß et al. (2008); Fortunato et al. (2017) ; Plappert et al. (2017) ), or explore by modeling data from previous tasks (Bogdanovic & Righetti (2019) ).

A related line of work also maximizes an auxiliary objective along with the main objective to increase exploration, such as entropy or expected improvement to encourage diverse solutions and sample-efficiency. Pelikan et al. (1999) ; Snoek et al. (2012) use Bayesian optimization to model the objective landscape and explore by incorporating both the mean and uncertainty about the prediction in an auxiliary optimization. Williams & Peng (1991) ; Ziebart (2010) ; Haarnoja et al. (2017; 2018) augment the standard RL objective with an entropy maximization term, which is deeply linked with risk-sensitive optimal control theory (Rawlik et al. (2013) ), which we build upon in our work.

2.1.1. USING MODELS FOR EXPLORATION IN MODEL-FREE RL:

Curiosity-driven exploration can be summarized as "explore what surprises you"; often, this takes the form of incentivizing exploration of states with high prediction error with respect to a dynamics model (Schmidhuber (1991; 2006) ) even while using model-free algorithms to learn to act. This line of work can be scaled up to higher-dimensional environments including images Stadie et al. (2015) ; Pathak et al. (2017) . This may perform well in the absence of extrinsic reward, as Burda et al. (2018a; b) demonstrate, and even when disregarding the temporal structure of RL as in Conti et al. (2018) .

2.2. EXPLORATION FOR MODEL-BASED RL:

There are comparatively fewer exploration methods designed specifically for model-based RL. As mentioned previously, model-based RL methods can perform exploration during the planning phase as well as the system rollout phase; we categorize the following related work based on this distinction. 2019), which uses risk-sensitive stochastic optimal control and the uncertainty of probabilistic dynamics models to provide exploration signal. We follow their formulation but use ensembles of deep neural networks as the dynamics model choice for better scaling properties, simplify the variance estimate used to provide exploration signal, and perform much larger scale simulation experiments.

3. MODEL-BASED REINFORCEMENT LEARNING (MBRL):

Algorithm 1 MBRL(s 1 , F, Fθ , T , n iter) Algorithm 2 run policy(s 1 , π, f, T )

1: τnew = {}, st=1 = s1 2: for t = 1, . . . , T do 3: at ∼ π(st) 4: st+1 = f (s, at) 5:

τnew = τnew ∪ {(st, at, st+1)} 6: end for 7: return τnew Algorithm 3 derive policy iLQR( Fθ , τ nominal , T ) 1: A ← line search parameters [0, . . . , 1] 2: τ * , J * ← τnom, cost of (τnom) 3: for opt iter in 1, . . . , max opt iters do 4:

kt, Kt ← Riccati equation 5:

for α ∈ A do 6:

τsimulated ← simulate policy π using Fθ 7:

Jsimulated ← cost of τsimulated 8:

if Jsimulated < J * then 9:

τ * , J * ← τsimulated, Jsimulated 10:

π * ← π 11: end if 12:

if converged then 13: break 14:

end if 15:

end for 16: end for 17: return π * Following Bellman (1957) , consider a finite-horizon Markov decision process (MDP) with the time horizon T ∈ N. In this setting, the aim of reinforcement learning is to find a stochastic policy π which minimizes the expected total cost C(π) = E s1,a1,... T t=1 c(s t , a t ) where s 1 ∼ ρ 1 (s 1 ), a t ∼ π(a t |s t ), s t+1 ∼ F (s t+1 |s t , a t ). s t , a t is the state and action at time t respectively, and the dynamics function F induces a distribution over the next state s t+1 . MBRL (Algorithm 1) approaches this problem by approximating the dynamics function F with a model F , parameterized by some θ. Given Fθ , MBRL derives a policy π Fθ by optimizing the predicted long-term cost. This can be done through sampling-based methods like the crossentropy method (Chua et al. (2018) ) or gradient-based optimal control methods like iterative LQR (Li & Todorov (2004) ). Bechtle et al. (2019) show that MBRL with iLQR can be applied to highdimensional real-world tasks.

3.1. DYNAMICS MODEL LEARNING THROUGH EPNNS:

A recently popular choice for modeling dynamics are ensembles of probabilistic neural networks (EPNNs) (Chua et al. (2018) ). EPNNs provides a probabilistic prediction of the dynamics, and scale well with respect to dataset size. Following Lakshminarayanan et al. ( 2017): we define a probabilistic neural network (PNN) f θ (x) as a deep neural network which outputs the parameters of a probability distribution, in this case the mean µ(x) and diagonal covariance Σ(x) of a multivariate Gaussian distribution. The parameters θ are learned by minimizing the negative log likelihood of data collected on the true dynamics. We define a B-sized ensemble of probabilistic neural networks (EPNN) as the bootstrapped ensemble of B PNN models and treat the output of the ensemble as a uniformly-weighted Gaussian mixture model.

3.2. POLICY OPTIMIZATION VIA ITERATIVE LINEAR QUADRATIC REGULATOR (ILQR):

Given a model F , finite horizon T , a nominal trajectory τ nom = {(s nom 1 , a nom 1 ), . . . , (s nom T , a nom T }, and a known cost function c(s, a), iLQR returns a set of control parameters k t , K t for a time-varying locally-linear feedback policy:

a t = π t (s t ) = a nom t + αk t + K t (s nom t -s t ), with step size α chosen by line search.

In more detail: given the nominal initial actions a t , t = 1, . . . , T , iLQR in the forward pass rolls out the actions using the learned dynamics model s t+1 = F (s t , a t ). The policy is then optimized in the backward pass by computing a linear approximation of the dynamics and a quadratic approximation of the cost using Riccati equations. We make a linear approximation at each time step s t+1 = Fst s t + Fat a t to construct the local value function Q t and its derivatives. This is used to update the parameters of the time-varying locally-linear feedback policies k t , K t . See Algorithm 3 for details and Li & Todorov (2004) for a derivation.

4. EXPLORATION IN MODEL-BASED REINFORCEMENT LEARNING:

We consider two dimensions of exploration in MBRL. We first note that we can apply exploration in both the environment-interaction phase (when collecting data on the real environment), and during the simulation phase (when simulating a rollout with a learned dynamics model). Here, we consider two classes of exploration: action-space noise, by applying Gaussian noise to the actions of a deterministic policy, and transition-space noise, by eliciting risk-seeking behavior over modeling uncertainty in the simulated state transitions. Action-space noise can be applied during both the environment-interaction phase and the simulation phase, while the transition-space noise operates only in the simulation phase.

4.1. EXPLORATION IN ACTION SPACE FOR ILQR:

Even though iLQR returns a deterministic policy, we can explore by inducing a probabilistic policy

π t (a t |s t ) ∼ N (a nom t + αk t + K t (s nom t -s t ), Σ πt ) (1)

for some choice of time-dependent Gaussian noise covariance Σ πt Here, we consider two options for the covariance of the time-varying linear-Gaussian policies:

Action-fixed-covariance: We can set diag(Σ πt ) = [ 2 1 , • • • , 2 daction ]

for some fixed (i.e. not timevarying) 2 1 , . . . , 2 daction where d action is the dimensionality of the action space. This is a popular exploration method in model-free RL for continuous action spaces. In this work, we make the further assumption that the noise is isotropic, and set 2 i = 0.

2 for i = 1, • • • , d action .

Action-maximum-entropy: A more principled choice for Σ πt for each timestep t is the curvature of the value function Q t , i.e. Σ t = Q -1 at,at , which is shown in Levine & Koltun (2013) to be the solution of the maximum entropy control problem min T t=1 E (st,at)∼π(st,at) [c(s t , a t )-H(π(a t |s t ))] where H(•) measures the entropy of action distribution induced by the policy π t (a t |s t ). This solution has been used in subsequent work (e.g. Levine & Abbeel (2014) ; Levine et al. (2016) ).

4.2. EXPLORATION IN TRANSITION SPACE FOR ILQR:

We can also explore through the transitions of the dynamics model by inducing a Gaussian distribution over our dynamics: s t+1 ∼ N (F (s t , a t ), Σ trans,t )

(2) We consider two options for Σ trans,t :

Transition-model-uncertainty: If the dynamics are modeled through a probabilistic model, a natural choice for Σ trans,t is the predictive uncertainty of the learned probabilistic dynamics model, i.e. diag(Σ trans,t ) = var[s t+1 ∼ F (s t , a t )]. Intuitively, this encourages iLQR to explore states for which the learned model is uncertain.

Transition-fixed-covariance: Alternatively, we can induce exploration by fixing diag(Σ trans,t ) to be some vector [ 2 1 , . . . , 2 dstate ], which is equivalent to assuming fixed isotropic uncertainty throughout the state space. In this work, we set 2 i = 1.0 for i = 1, . . . , d state . As described in Bechtle et al. (2019) , in this setting, the variance of the trajectory cost C(π) = E π [ T t c(s t , a t )] can be incorporated in iLQR by taking an exponential transform of the total cost function C = exp[σC(π)]. A first order Taylor expansion of the cost gives

EQUATION

where var[C] is the variance of the trajectory cost. Bechtle et al. (2019) solves this optimal control problem using a variant of the Riccati equation, following results from stochastic optimal control (Farshidian & Buchli (2015) ).

The parameter σ weights the effect of Σ trans,t in the Riccati equation. σ = 0 is equivalent to unmodified iLQR, while σ < 0 leads to exploratory behavior by incentivizing states which lead to trajectories with high uncertainty var [C] . In this work we set σ = -0.3, but extensively test the effect of modifying this parameter in Appendix A.

5. EXPERIMENTS:

For our experiments, we consider a PyBullet (Coumans & Bai (2016) ) simulation of a 7 degree-offreedom robotic manipulator, the Kuka iiwa7 robot arm. This physics simulator has been used in prior work to learn policies which transfer onto a real robot (Tan et al. (2018) ). Our state representation includes the position and velocity of each joint, with dimensionality d state = 2 • 7 = 14, and the action is the applied joint torques after gravity compensation, with dimensionality d action = 7.

The cost function per timestep is the distance from the target joint state and a penalty on velocity and action norm:

c(s, a) = q pos s pos -s target pos 2 2 + q vel s vel 2 2 + r action a 2 2 (4)

Figure 1 : For each of 3 distances of 0.1, 0.45, 0.80 meters representing 3 difficulty tiers, we generate 5 targets at that distance away from the initial endeffector position. We visualize the initial position and example target joint configurations for each. with cost parameters q pos = 5, q vel = 0.1, and r action = 1e-7. iLQR optimizes the summed total cost over the trajectory T t=1 c(s t , a t ). The target joint state is generated using inverse kinematics for a random end-effector position. We generate targets three distances 0.1m, 0.45m, 0.8m away from the initial endeffector position as visualized by Figure 1 , and 5 targets per distance, with a total of 15. These target end-effector positions signify increasingly difficult scenarios for the algorithm. We check to make sure all targets are achievable through inverse kinematics.

For each target, we initialize 10 random torque trajectories and run iLQR with learned EPNN dynamics, measuring the quality of the solution through the iLQR cost over the real unrolled trajectory. To evaluate, we compute the mean and standard deviation of the trajectory cost across targets in each tier of difficulty. Figure 2 : MBRL learning curves with learned dynamics models. Here, we plot the best reward found so far averaged over 3 seeds. PETS takes much longer to reach convergence, so we additionally plot the best reward over any trajectory seen by PETS over the full 300 episode training.

We start out by analyzing the effect of adding exploration to the model-based RL loop (Algorithm 1). We note that action-space exploration can be applied both to the real rollout, which occurs in the outer loop as in line 6 of Algorithm 1, and within the simulated rollout, which occurs in the inner loop as in line 6 of Algorithm 3.

For completeness, we also provide a comparison to a state-of-the-art model-based RL algorithm, PETS, in Figure 2b tasked with optimizing the same cost function. PETS does receding-horizon MPC by replanning at every timestep, while iLQR optimizes over the full trajectory. As a result, iLQR trajectories can be unrolled on real systems, while PETS can take up to 1s for inferring actions at each time step. PETS quickly reaches a comparable solution to iLQR in terms of final distance to the desired joint position, but fails to reduce iLQR cost to comparable levels.

As a first step, we visualize the training curve for one of the randomly-generated targets in Figure 2a , to give some insights into convergence behavior for different exploration behaviors. Based on these plots, the first result is that, at least for this target, there is little to no benefit of adding exploration to the MBRL loop.

As a second step we perform all possible combinations of the two types of exploration methods and show results in Table 1 . The most significant findings across this large-scale experiment:

• Both action-phase exploration methods, produce better results when applied in the realsystem rollout phase (outer loop). Action-fixed-cov performs better than Action-max-ent.

• All transition-exploration based methods perform better than the action-based exploration methods. • However, most notably, none of the exploration methods improve the performance of our basic MBRL loop without exploration.

5.2. EXPLORATION FOR TRAJECTORY OPTIMIZATION USING GROUND-TRUTH DYNAMICS:

In an attempt to get a better understanding of the surprising result that exploration does not aid MBRL, we aim to isolate the effect of policy optimization from dynamics model learning. In our next set of experiments, we use a ground-truth dynamics model, wherein the simulator itself is used as the dynamics model. We use the finite-difference approximation of the model gradients when required by iLQR. We run the same set of experiments as above, but only with inner-loop exploration. Results are summarized in Table 2 . Here, we clearly see a benefit of adding transition-exploration for the first set of targets. Second, we note the high variance of iLQR without exploration results, clearly demonstrating the inability of iLQR to find good solutions consistently, across all target difficulties. iLQR with fixed-covariance and transition-space exploration leads to better mean results, as well as reduced variance of the solutions for nearby targets.

This exploration bonus likely only comes up because of the local nature of our optimizer and our choice of globalization strategy (simple line search). This effect might be mitigated when using more advanced (and more complex) globalization strategies (Nocedal & Wright (2006) ) or using a multiple shooting version of iLQR, less sensitive to initial conditions (Mastalli et al. (2019) ). Nevertheless, most optimizers in high-dimensional spaces are necessarily local when used on real robots for computational efficiency reasons and will remain prone to getting stuck in local minima even when using sampling methods.

While this set of experiments confirms our intuition that exploration can help, it is unclear why this effect is not visible during the MBRL experiments. However, when taking a step back, and looking at all results generated so far, Tables 1 and 2 , we observe a surprising result: the MBRL loop via regular iLQR, which learns a dynamics model via EPNN, actually outperforms iLQR on ground truth dynamics. We discuss this below.

5.3. EXPLORATION THROUGH EXPLOITATION FOR MODEL-BASED RL:

For clarity, we present the key results again, but from a different perspective. We compare iLQR on ground truth dynamics with and without exploration with iLQR on learned dynamics (Table 3 ). These results show that using iLQR within a MBRL loop to learn the dynamics model leads to a better performance consistently.

This effect can be explained as follows: starting from a randomly initialized dynamics model in MBRL has an unexpected benefit: it results in an implicit exploration. During policy optimization, the poor predictions from the dynamics model will initially lead to undirected policies, that perform poorly on the task but creates diverse data to update the dynamics model, and a better initialization for the optimizer. Because of this implicit exploration from the bad dynamics model, the advantage of explicit exploration is less visible in MBRL. Incorporating fixed transition-covariance in the trajectory optimization with ground-truth dynamics improves its performance, bringing it closer to learned dynamics. This reinforces our hypothesis that the poor performance on ground-truth is due to lack of exploration.

5.3.1. MODEL GENERALIZATION TO NEW TARGETS:

Finally, we evaluate the generalization capabilities of the dynamics models learned from the various explorations methods. For each dynamics model learned for targets of distance d meters, we use the standard iLQR algorithm (with no exploratory behavior) to optimize a policy for other targets at the same distance d. We roll out the policy onto the real environment once (i.e. no model learning). In this way, we aim to measure the ability of the dynamics models themselves to generalize without additional training. The results show that exploration does provide some benefit -it helps learn somewhat better models, although there is no consistent "winner". Also, note that the trained models do not actually generalize well to new targets, as compared to EPNN trained on a target (old cost for d = 0.10 was 16.77 ± 8.05), or using ground truth (cost for d = 0.10 was 36.71 ± 60.75).

Thus, we conclude that though iLQR with EPNN can perform better than iLQR with ground-truth for a particular target, the learned EPNN models fail to generalize to new targets without retraining. This demonstrates that the models learned in our model-based RL loop are effective local models: highly specific to the task for which it is trained but not well-suited for solving new tasks without further training, in contrast to global models like the ground-truth dynamics.

6. CONCLUSION:

In this work, we studied the effect of exploration in different stages of a model-based RL loop. We added action-space noise when executing a policy on the real system, as well as when rolling out on the simulated dynamics. Whether isotropic Gaussian noise or the maximum-entropy solution, action-space noise applied to the optimized policy degrades performance across all our experiments. We also added transition-space noise to the dynamics when doing trajectory optimization, using a risk-seeking iLQR formulation from literature. In this case, we observe that iLQR with learned dynamics and no exploration performs better than all the transition-space exploration techniques considered. In fact, for our experimental setup, using ground-truth dynamics without exploration performs worse than EPNN with iLQR due to the exploratory nature of imperfect dynamics models.

Although our conclusions are limited by the choice of policy optimization algorithms, model representations and experimental environments, our results shed lights on effects that are seldom discussed in MBRL. Future work will explore the effect of exploration with different classes of MBRL algorithms and other robotic tasks.

:



Figures and Tables:

D ← run policy(s1, πbabbling, F, T ) 2: θ ← train model( Fθ , D) 3: for iter=1, . . . , n iter do 4: τlast ← last trajectory from D 5: πiter ← derive policy( Fθ , τlast, T ) 6: D = D ∪ run policy(s1, πiter, F, T ) 7: θ ← train model( Fθ , D) 8: end for 9: return πn iter, Fθ

(a) Initial (b) 0.1m (c) 0.45m (d) 0.80m

(a) iLQR with different exploration methods (b) iLQR vs. PETS (iLQR cost) (c) iLQR vs. PETS (joint dist)

Moldovan et al. (2015) use the principle of "optimism in the face of uncertainty" by learning a Dirichlet process mixture of linear models for model-predictive control.Shyam et al. (2018) also use an ensemble of deep neural networks to model forward dynamics and use disagreement between models to provide exploratory signal.Boedecker et al. (2014) use probabilistic dynamics models with iLQR to encourage exploration through model uncertainty; however, they consider lower-dimensional problems using Gaussian Processes, which do not scale well to large number of data points. Our work is most closely related toBechtle et al. (

Table content: <table><tr><td>Exploration during environment interaction: Wiering &amp; Schmidhuber (1998) propose using</td></tr><tr><td>an exploration Q-function to generate exploratory actions for model-based RL. Lopes et al. (2012)</td></tr><tr><td>empirically measures learning progress to drive exploratory behavior. Hester &amp; Stone (2017) use</td></tr><tr><td>random forests for modeling dynamics and encourage exploration using model uncertainty and dis-</td></tr><tr><td>tance measure from previously explored regions. All of these methods only consider the tabular</td></tr><tr><td>case, while we examine exploration for model-based RL in high-dimensional continuous state-and</td></tr><tr><td>action-spaces.</td></tr><tr><td>Exploration during planning:</td></tr></table>

Exploration with EPNN model.

Table content: <table><tr><td colspan="7">Full trajectory cost, averaged over targets with equal distance from initial end-effector</td><td/></tr><tr><td>Inner</td><td>Outer</td><td>0.10m</td><td>std</td><td>0.45m</td><td>std</td><td>0.80m</td><td>std</td></tr><tr><td>No exploration</td><td/><td>16.77</td><td>8.05</td><td colspan="4">190.12 139.45 383.11 53.53</td></tr><tr><td>Trans-model-uncertainty</td><td/><td>18.18</td><td>8.03</td><td colspan="4">205.18 146.09 391.96 43.59</td></tr><tr><td>Trans-fixed-cov</td><td/><td>18.70</td><td>9.22</td><td colspan="4">196.26 138.44 397.74 53.47</td></tr><tr><td>Action-fixed-cov</td><td/><td>38.89</td><td>9.40</td><td colspan="4">230.08 133.09 456.40 64.71</td></tr><tr><td>Action-max-ent</td><td/><td colspan="6">236.98 25.99 471.01 132.17 755.74 63.21</td></tr><tr><td/><td>Action-fixed-cov</td><td>20.76</td><td>8.34</td><td colspan="4">205.85 152.07 395.90 49.62</td></tr><tr><td/><td>Action-max-ent</td><td colspan="6">112.96 11.34 284.14 138.00 481.44 42.25</td></tr><tr><td colspan="2">Trans-model-uncertainty Action-fixed-cov</td><td>20.63</td><td>8.05</td><td colspan="4">194.86 135.92 393.06 41.09</td></tr><tr><td>Trans-fixed-cov</td><td>Action-fixed-cov</td><td>21.65</td><td>8.06</td><td colspan="4">231.65 148.29 406.17 40.65</td></tr></table>

Exploration with ground-truth model.

Table content: <table><tr><td colspan="7">Full trajectory cost, averaged over targets with equal distance from initial end-effector</td></tr><tr><td>Algorithm</td><td>0.10m</td><td>std</td><td>0.45m</td><td>std</td><td>0.80m</td><td>std</td></tr><tr><td>None</td><td>36.71</td><td colspan="5">60.75 205.63 128.92 512.73 210.29</td></tr><tr><td>Action-fixed-cov</td><td>73.70</td><td colspan="5">60.76 222.40 105.10 563.15 218.87</td></tr><tr><td>Action-max-ent</td><td colspan="4">214.53 79.96 432.82 90.46</td><td colspan="2">801.94 186.32</td></tr><tr><td>Trans-fixed-cov</td><td>24.52</td><td colspan="5">25.67 196.81 109.81 535.25 190.15</td></tr></table>

iLQR with learned dynamics vs ground truth dynamics.

Table content: <table><tr><td colspan="7">Full trajectory cost, averaged over targets with equal distance from initial end-effector</td></tr><tr><td>Problem setting</td><td colspan="2">0.10m std</td><td>0.45m</td><td>std</td><td>0.80m</td><td>std</td></tr><tr><td>Ground-truth with no exploration</td><td>36.71</td><td colspan="5">60.75 205.63 128.92 512.73 210.29</td></tr><tr><td>Ground-truth with fixed-cov</td><td>24.52</td><td colspan="5">25.67 196.81 109.81 535.25 190.15</td></tr><tr><td>EPNN with no exploration</td><td>16.77</td><td>8.05</td><td colspan="4">190.12 139.45 383.11 53.53</td></tr></table>

EPNN model generalization. Models are taken after training and tested on new targets.

Table content: <table><tr><td colspan="7">Full trajectory cost, averaged over targets with equal distance from initial end-effector</td></tr><tr><td>Algorithm for dynamics data</td><td colspan="2">0.10m std</td><td>0.45m</td><td>std</td><td>0.80m</td><td>std</td></tr><tr><td>EPNN without exploration</td><td>56.21</td><td colspan="5">20.49 365.33 218.38 1402.28 1246.29</td></tr><tr><td>Action-fixed-cov (inner)</td><td>54.93</td><td colspan="5">16.62 307.65 145.74 1414.48 1556.82</td></tr><tr><td>Trans-model-uncertainty</td><td>58.78</td><td colspan="5">18.07 329.90 150.20 1539.96 1547.73</td></tr><tr><td>Trans-fixed-cov</td><td>59.18</td><td colspan="5">24.09 325.34 152.16 1267.18 1011.21</td></tr></table>

Peer review (iclr2020):
This paper investigates the effect of adding noise to action selection and transition dynamics estimation in model-based reinforcement learning (MBRL) for improved exploration. The authors use the iterative Linear Quadratic Regulator (iLQR) algorithm with their methods for inducing noise and evaluate the performance on a robotic arm simulator. I think the paper explores an interesting problem which does not receive a lot of attention (exploration in MBRL). However, given that it is an empirical paper, I don*t find the empirical evaluation thorough enough and some of the results found seem questionable. Further, the paper is completely focused on iLQR on the robotic arm task, so it*s not at all clear whether the results carry through to other algorithms or other environments. Some more detailed comments: - Intro: *combines two extremely data inefficient techniques*: are you referring to model-free versus model-based? they*re not really *combined* in RL, and *extremely data inefficient* seems rather strong. - Intro: *In model free methods, exploration is usually done by adding noise to the action suggested by the optimized policy.*: not really, most of the interesting exploratory algorithms use more sophisticated techniques like intrinsic rewards, pseudo-counts, and many even have theoretical guarantees (see all the PAC-MDP work). - Sec 2.1: There*s a whole line of related work missing here. For example: * *Near-Optimal Reinforcement Learning in Polynomial Time*, Kearns & Singh, 2002 * *R-max – A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning*, Brafman & Tennenholtz, 2002 * *PAC Optimal Exploration in Continuous Space Markov Decision Processes*, Pazis & Parr, 2013 * *Unifying Count-Based Exploration and Intrinsic Motivation*, Bellemare et al., 2016 * *Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment*, Taiga et al., 2019 - Algorithm 3 line 6: Where does hat{pi} come from? - Section 3: *aim of reinforcement learning is to find a stochastic policy* it doesn*t have to be stochastic - Section 4.1, Action-maximum-entropy: You use , what does this mean? Why does Q have two a_t subscripts? - Section 4.2, Transition-fixed-covariance: *we set *: this seems quite high. is there a reason why? did you try other values? - Equation (4): Is this a standard cost function or is it only specific to this work? If the former, add reference, if the latter, explain the choice. - Section 5: It*s not clear what you mean by *inverse kinematics* - Figure 1: It*s not clear where the targets are. It might help if you draw a trajectory from the initial to target position on each, otherwise it*s hard to tell them apart. - Figure 2 (a): What*s the difference between the three plots? - Figure 2 (a): The labels on the legend don*t match anything that was previously introduced, so it*s not clear what*s what. - Section 5: What*s MPC? - Table 1: There are a number of different hyperparameter choices for the different exploration methods you evaluate. How did you pick them? Did you do a sweep? Are the results averaged over multiple runs? How many? - Section 5.2: What is *finite-difference approximation*? - Section 5.2: *We run the same set of experiments as above, but only with inner-loop exploration*: why? - Table 2: The algorithm with no exploration does *much* worse when using the ground truth dynamics compared with using the model in table 1. This is totally counter-intuitive: shouldn*t the results be strictly better given that there*s zero model approximation error? I*m not sure I buy the justification you gave in section 5.3. There is a >2x degradation when using ground truth without exploration for the 0.10m target, which is significantly worse than the degradation for the harder 0.45m target. If exploration really is the cause, I would expect to see the decrease in performance correlate with the difficulty. - Section 5.3.1: This section seems somewhat unrelated to the main thesis of the paper. If you do want to investigate this problem, it calls for a much more thorough investigation. Right now the investigation provided is somewhat superficial, which detracts from the paper. - Section 5.3.1: *Thus, we conclude that... for a particular target*: Is this really that surprising? Ground truth dynamics are agnostic to the target, whereas the learned models build their estimates from collected trajectories, which are gathered using the current best policy, which is aiming to minimize the cost to the specific target. One minor comment: - Section 5: *We generate targets three distances 0.1m, 0.45, 0.8m away from the initial...* would read better as *We use targets at three different distances from the initial end-effector: 0.1m, 0.45m, 0.8m*.