Inherently Robust Control through Maximum-Entropy Learning-Based Rollout
Authors: Felix Bok
Atanas Mirchev
Baris Kayalibay
Ole Jonas Wenzel
Patrick van der Smagt
Justin Bayer
Venue: Accepted by TMLR ()

Abstract: Reinforcement Learning has recently proven extremely successful in the context of robot control. One of the major reasons is massively parallel simulation in conjunction with controlling for the so-called ``sim to real'' gap: training on a distribution of environments, which is assumed to contain the real one, is sufficient for finding neural policies that successfully transfer from computer simulations to real robots. Often, this is accompanied by a layer of system identification during deployment to close the gap further. Still, the efficacy of these approaches hinges on reasonable simulation capabilities with an adequately rich task distribution containing the real environment. This work aims to provide a complementary solution in cases where the aforementioned criteria may prove challenging to satisfy. We combine two approaches, $\textit{maximum-entropy reinforcement learning}$ (MaxEntRL) and $\textit{rollout}$, into an inherently robust control method called $\textbf{Maximum-Entropy Learning-Based Rollout (MELRO)}$. Both promise increased robustness and adaptability on their own. While MaxEntRL has been shown to be an adversarially-robust approach in disguise, rollout greatly improves over parametric models through an implicit Newton step on a model of the environment. We find that our approach works excellently in the vast majority of cases on both the Real World Reinforcement Learning (RWRL) benchmark and on our own environment perturbations of the popular DeepMind Control (DMC) suite, which move beyond simple parametric noise. We also show its success in ``sim to real'' transfer with the Franka Panda robot arm.

Published in Transactions on Machine Learning Research (12/2025)
Inherently Robust Control through Maximum-Entropy Marning-Baseed Rollout
Felix Bok
felin. bok-tum. de
Volkswogen Group & Technical University of Munich
Munich
Atanas Mirchev
Volkswagen Group Munich
Barış Kayalıbay Foundation Robotics Labs Munich
Ole Jonas Wenzel Foundation Robotics Labs Munich
Patrick van der Smagt Poundation Robotics Labs ELTE Universiby Munich & Budopest
Justin Bayer
Reviewed on OpenReview: http:://pparrrriw..aa/////ottoop
Abstract
Reinforcement Learning has recently proven extremely successful the context of robot control. One of the major reasons is massively parallel simulation in conjunction with con- trolling for the so-called "sim to real" gap: training distribution of environments, which is assumed to contain the real one, is sufficient for finding neural policies that successfully transfer from computer simulations to real robots. Often, this is accompanied by layer of system identification during deployment to close the gap further. Still, the efficacy of these approaches hinges on reasonable simulation capabilities with an adequately rich task distribution containing the real environment. This work aims to provide complemen- tary solution in cases where the aforementioned criteria may prove challenging to satisfy, We combine two approaches, mazimum- reinforcement learning (MaxEntRI.) and rollout, into an inherently robust control method called Maximum-Entropy Learning- Based Rollout (MELRO). Both promise increased robustness and adaptability on their own While MaxEntRI. has been shown to be an adversarially-robuss approach in disguise, rollout greatly improves over paramotric models through an implicit Newton step on a model of the environment. We find that our approach works excellently in the vast. majority of cases on both the Real World Reinforcement Learning (RWRL) benchmark and on our own environment perturbations of the popular DeepMind Control (DMC) suite, which move be- yond simple parametric noise. We also show its success in "sim to real" transfer with the Franka Panda robot arm.

---

Published in Transactions on Machine Learning Research (12/2025)
1 Introduction
Reinforcement Learning (RL) identifies optimal behavior strategies through an automated process of trial and error and constitules one of today's standard approaches for addressing complex sequential decision-making problems. In particular, the application of RL to humanoid robots has achieved extremely impressive results. For example, they can now successfully navigate through complex tcrrain (Sun al., 2025), dextcrously manipulate real-world objects purely from vision (Lin al., 2025), and even perform complex dynamic movements (Zhuang et al., 2024). In most cases, learning policies for such tasks rely on massively parallel simulations, given that training directly on hardware is impractical in nearly all instances: often, millions of interaction steps are required. However, training control policies exclusively in simulation frequently uncovers a phenomenon referred to as the "sim to real" gap: a policy that has been extensively optimized in simulation demonstrates degraded performance, or even complete failure, during real-world deployment.
One successful and commonly used approach domain randomization (Tobin et al., 2017), where the policy is exposed to wide range of simulated environments. Key simulation parameters are randomized during policy training, creating more diverse experiences and forcing policies to become robust to environmental variations. The real world is then assumed to be just another varialion within that distribution. Fulfilling this assumption rests on two key desiderata: (i) all critical enviromment parameters and their ranges need to be carefully identified such that the real world is close or contained within; (ii) the space needs to be sufficiently tight to avoid overly conservative policies that need to comprouise performance over too many different instances. Whereas the last point can be addressed parts by domain identification, this process requires precise domain knowledge, extra engineering effort, and additional computing power to work well (Josifovski et al., 2022; 2024; Dulac-Arnold et al., 2021).
In this paper, we present an orthogonal approach to domain randomization tailored to instances where it is challenging to specify all key simulation parameters or too costly to exhaust all. To that end, we in- tegrate mazimum-entropp reinforcement learning (MaxEntRL) (Ziebart et al., 2008) and rollout (Tesauro, 1994; Bertsekas, 2024) into an inherently robust method called Maximum- Eancimmm-Entropy Learning- Based Rollout (MELRO). While MaxEntRL enhances policy robustness by promoting exploratory behaviors, rollout com- pensates for residual uncertainties through real-time re-planning. We demonstrate in extensive evaluations that MELRO features and extends the strengths of both frameworks. Firstly, conduct multiple experi- ments on an illustrative toy example that aim to highlight the differences between the strategies generated by traditional MaxEntRL, domain randomization and MELRO. These experiments also provide in-depth insights into the effects of entropy-regularizzttnn planning. Secondly, we benchmark its performance against a variety of tasks in the Real World Reinforcement Learning (RWRL) suite. Furthermore, we assess its capabilitics on a custom set of challenging environment perturbations applicd to the popular DeepMind Control (DMC) suile, which are designed to probe robustness beyond simple parametric noise. Finally, we present a successful "sim to real" transfer of MELRO on the Franka. Panda robot arm.
2 Preliminaries
2.1 Problem
We assume that the problem consists of an unknown number of Markov Decision Processes (MDPs) (Bell- mann, 1957) sharing the same action and state space. This can be formalized as a partially observable Markov Decision Process (POMDP) (Kaelbling et al., 1998), which is characterized by the tuple A, o, Pa. Po:7), where S, A and Ω are the state, action and observation spaces, p S x A S is the transition function, r S x A + R is the reward function, o S + Ω is the observation function, and Po is the initial state distribution (Humplik el al., 2019). The state space S then consists of the state space of the MDPs, denoted as S.M, and a set of task distributions T. Initially, the task specification T is sampled from T and maintained through time, assuming it the only unobscrvable part of the state. This results in S (SM) x T), Ω SM and o(ss, + s, where s E S.M The transition and reward functions and the initial state distribution are also conditioned on 7. The goal is then to find policy S.M + A
[0, 1] is discount factor,
2

---

Published in Transactions on Machine Learning Research (12/2025)
that maximizes the expected discounted return under the sampled but unknown task specification
T Σ Et'((sr 7.
(ag (s)) arg max max Er
where 1 = Sk.7 is state trajectory where the states are sampled from the transition function of the MDP
7, and 5 T
2.2
Eeximum-EEntrop Reinforcement Learning
Maximum-entropy reinforcement learning (MaxEntRL) exlends traditional RL by augmenting Equation 1 with a conditional regularization term, the policy entropy Hzi
T Σ Ext(snn t=k
(ax arg max max Ep-
+ (ax
(2)
ok
Ha (ox (8k) = Bawx(au (ex) |- log (a)(s)))]
where 17 > 0 is a temperature parameter controlling the entropy trade-off and the entropy Hi (ax|sk) ) is ap- proximated with Monte Carlo integration. Solving for Equation 2 results in stochastic policies characterized by non-zero probability for every action at each state. The entropy is higher when more actions lead similar rewards and lower when one action is substantially better.
2.3 Learning-Baa Based Rollout
R 1 is a control method employing dynamic system model to predict future behavior and optimize controls online. An elegant theoretical interpretation of rollout positions it as implementing Newton step in value space, perspective described by Bertsekas (2024). At each time step, rollout optimizes the controls for given lookahead horizon I. E N to minimize the predicted future costs and applies the first optimized control in a receding-horizon horizon fashion:
k+4-1
Σ - ( + (((sHt) 1=
Ty (ax = arg max
max Er T(sk, + dg 1-
(3)
where R: S. + R+ is the terminal reward function estimating all future rewards and 1 SkckLt The rollout components, the dynamics model, reward, and terminal cost. function are typically hand-engineered. Instead, we use multi-layer perceptrons (MLPs) whose parameters are learned from environmental interac- tions.
3 Maximum-Entropy Learning- Based Rollout
MELRO uses the control framework introduced by Bertsekas (2022). It consists of two stages: off-line training and on-line play. The approach is visualized in Figure 1.
3.1 Off-line Training
In offe training, we use a model-based reinforcement learning (MBRT) framework to train model, base policy, and a critic.
Model Learning It is important to note that MELRO model-agnostic; consequently, can be utilised in conjunction with any prediction model. In this study, the model architecture and training are based on the variational state-space model utilised in Bayer al. (2021), with the following modifications. Given that
Other related terms model predictive control (MPC), receding horizon control, limited lookahead control. We will term "rollout", as the work of Bertsekas (2024).

---

Published in Transactions on Machine Learning Research (12/2025)
Off-line Training
On-line Play
09
o),
Rewards
Generation
Predicted reward +
horizon
entropy
s,
Simulated Data
Policy & Critic Learning
Predicted reward
Entropy
Model Learning
a
replay buffer
a
Planning
(1, O)
Figure 1: Overview of MELRO. We propose framework that uses both maximum-entroppy reinforcement learning and rollout to achieve inherently robust control complementary to domain randomization. Our method incorporates maximun-entropp regularization during off-line training and on-line planning. To achieve this, the world model is trained on observations and action distribution parameters. Planning then occurs directly within the action distribution parameter space.
each individual task considered here is fully observable, the emission model directly propagates the states forward as the mean of Gaussian with fixed variance:
p(rx(s))= = N (æell = st, 0.01).
(4)
Morcover, cost-function head is introduced, which maps states to rewards:
ra, (rels,,,, 0x) oT. (81,
(5)
where (n, is the mean and the standard deviation of Gaussian distribution. The transition is modeled as Gaussian residual component where the output is convex combination of the previous state St-1 and update S1-1. The mixing coefficient, p and the update are outputs of MLP with parameters 02:
Po. (s, =N (1= RCa, (8t-1, a. 1), 0.001)
(6)
RCo, (8t-1, -c-1) = p 81-1 + (1 - p) 8
(7)
(p. (p)8 MLP 0--1; p p= (softsign(() 1).
This enables states propagate more easily through transitions. All model components are trained replay buffer containing all previous policy-envirommet interactions.
Policy Learning The policy is modeled normal distribution, parametrized by o, where two-headed MLP estimates the mean and standard deviation:
(a)ls)) = N (arl(1) MLP Ms))
The critic 0x is modeled as a deterministic MLP with parameters X- Both the policy and the critic use layer normalization (Ba et al., 2016). The critic loss La, is given by a supervised learning objective on (Ar)tttasgss which are computed on imagined model and policy rollouts:
H - t=
La, = Earwsy
(5t, ax) - - R(st)11
(10)
H
4

---

Published in Transactions on Machine Learning Research (12/2025)
with targetts
R(s,) = (s) + riH ( (ailsr) + ((1 - A)ag (st, (st, + AB(
(11)
R(s++H.) = sg (0x ($1+ ((+H.)),
(12)
where ag is the graddentt operator. The policy loss Cap is given by the expected values of the (D) targets:
H Σ R(sr))
L. = E Bay
(13)
The optimal parameters for the policy and critic are obtained by maximizing the weighted sum of the policy and critic loss on model rollouts. Additionally, a regularization term consisting of the norm the gradients of the policy and critic loss is introduced to improve the convergence of the optimization problem by increasing the conservativity (Mescheder al., 2017). The policy-critic loss is then given by:
V V V
VL 12 VL
Lr (LiT Lox +
+
(14)
where the loss is scaled by weighting hyperparameter and the conservativity regularization term by n. During training, the gradients of all components are clipped by the Euclidean norm, with bound specified for each component.
3.2 On-line Play
In on-line play, we combine the trained components into the rollout algorithm. We thereby improve standard rollout in the following ways. First, online planning happens in the space action means and standard deviations of a Gaussian policy. Secondly, we generate imagined rollouts for the planning horizon based on the learned world model and the base policy. The resulting base policy action distribution parameters, i.e. means and standard deviations over time, utilized as initial values for gradient descent. Thirdly, in an analogous manner to standard MaxEntRL, the rollout cost function, as defined in Equation 3, is regularized by the entropy of the action distribution. The idca behind this regularization is that it will lead to more robust actions being selected during planning. Furthermore, immediately following the model rollouts and before applying the critic, a fixed number of base policy steps are performed, as suggested by Bertsekas (2024). This truncated rollout with the base policy enhances the stability property of the rollout policy at low additional computational cost.
The resulting rollout problem is then given by the following equations:
(MELRO(@/355) arg max An
max
LMELRO,
k+L-1
k+L-1
MELRO E
ro, (ok,
Σ
xi kr. (s., + REtL +1 Σ jok
(15)
i=1+1
k+L+m-1
R&+L
ro, (5,, + k+ k+ ( ax+L+m),
j=
where à; = (1), Oj N (Hs,o;) Vi E (K, +1. k L, a + L m+),, OL+L-+++--1 is sampled from the base policy тe, Letm is sampled from the transition function pa, and RK+ r. is the approximation of the future rewards starting from the time step ++L. The final controls are then computed in a receding- horizon fashion, optimizing Equation 15 using Adam (Kingma Ba, 2015), where the search dircction the space of action means and standard deviations is generated using Augmented Random Search (ARS) (Mania al., 2018).
5

---

Published in Transactions on Machine Learning Research (12/2025)
DR
DR
Base Policy
MaxEnt Base Policy
MELRO
(First Domain)
(Second Domain)
709
493
668
647
691
Original
50
217
443
647
186
Perturbed
Figure 2: Results for the toy maze. 100 evaluation trajectories in the unperturbed training maze and the perturbed evaluation maze of the entropy base policy, MaxEnt base policy, MELRO, and two policies trained using domain randomization with different domains. The mean reward for each method and environment is shown in the top left corner of every maze. The goal is visualized by the red star. The red blocks in the original environment of the two policies trained using domain randomization indicate where an individual block can appear during training in a rollout with a probability 0.5. During the evaluation of the original environment, no block was in the maze.
4 Experiments
To evaluate the robustness and transfer capabilities, experiments were conducted for an illustrative toy example, in simulation, and on a real Franka Panda robot arm. Training and validation for all experiments have been conducted on a MIG 1g. 10gb partition of an NVIDIA A100 GPU (compare to NVIDIA (2025) for more details). An evaluation of the computational costs is provided in Appendix C.
4.1 Illustrative Toy Example
To illustrate the challenges posed by traditional MaxEntRL methods and domain randomization, and to highlight the advantages of MELRO, we use simple 2-DoF point maze environment modeled with the Gymnasium--oo Robotics suite (de Lazcano et al., 2024). The task in this environment is to move force-actuated ball from the bottom middle the top middle of closed maze, with the exponential negative Euclidean distance between the current position and the goal position as reward. In order simulate perturbations and the "sim to real" gap, the training of all components is performed without the incorporation of any blocks in the middle of the maze. The evaluation is then conducted directly positioned between the starting position and the goal. To approximate the optimal level of entropy regularization we use the perturbed environment lo validate multiple MaxEnt base policies with different entropy weights.
maze that contains three blocks
Challenges of traditional MaxEntRL The three left columns of Figure 2 present 100 trajectories and their mean rewards for the base policy without entropy regularization, the MaxEnt base policy, and MELRO. The results highlight that policies with no or insufficient entropy regularization tend to overfit to the un- perturbed environment and fail completely once perturbations are introduced, as all trajectories become stuck at the blocks. When the entropy weight is tuned appropriately, MaxEntRL generates more entropic and diverse trajectories, improving robustness and enabling more trajectories reach the goal. However, performance the unperturbed environment deteriorates significantly, and identifying the appropriate reg- ularization level is computationally costly, change of the entropy weight necessitates complete MBRI.

---

Published in Transactions on Machine Learning Research (12/2025)
700
650
600
Reward
400
200
Base Policy (No Entropyl
MaxEnt Base Policy (Cptimal Entropy)
Demain!
Environment Steps
1e5
Entropy
Optimal Entropy
High Entropy
Figure 3: Effects of the entropy weight the rollout plans. Visualization of rollout plans of MELRO with different entropy reg- ularization during planning on the toy maze example.
Figure 4: Training curves toy example. Reward during train- ing for all the methods evaluated for the toy maze example.
run. Moreover, high entropy regularization results in highly stochastic trajectories that can degrade policy learning, shown in Figure 4.
Advantages of MELRO In contrast, MELRO consistently achieves the optimal balance: it relinquishes only a minimal amount of reward in the unperturbed environment and attains the maximum mean reward under the perturbation. One reason for this is its capability for adapting the entropy regularization during the on-line play phase for the specific perturbation. This makes tuning the regularization far more effi- cient. Furthermore, adjusting the entropy weight within MELRO has an intuitive and interpretable effect of smoothly broadening the diversity of the paths taken. This the direct effect of entropy regularization increasing the standard deviation of the action distribution. Consequently, this results in a greater degree of diversification ion in the plans during the planning process, illustrated in Figure 3. Lastly, given that the components of the best-performing MELRO configuration are trained in a low-entropy regularized MBRL run, the trajectories are more directed towards the goal. The reason for this is that MELRO can maintain dynamic entropy adaptation a manner analogous to the low-entropy base policy. Further evaluations of MELRO can be found Appendix G.1.
Comparison of MELRO to Domain Randomization To compare MELRO with domain randomiza- tion, two policies have been trained under two randomized domains. The domains differ in regions where blocks can appear, during training and for each single rollout, with a probability of 0.5. These and the resulting trajectories are depicted in the most right columns of Figure 2. The policy trained on the first domain converged to conservative strategy that nearly always avoids the area where blocks can appear, resulting identical trajectorics across mazes. While this approach yiclds the highest mean reward in the perturbed environment, it slightly underperforus MELRO in the unperturbed set tini and risks failure if the sole traversed path becomes blocked. Conversely, the policy trained on the second domain, where every path to the goal may be obstructed, learns to follow approximately the shortest route, effectively disregarding perturbations. This strategy achieves slightly higher performance the unperturbed maze, but exhibits significantly lower level of robustness when compared with trajectories of MELRO the perturbed maze.
These results highlight the sensitivity of domain randomization to the chosen simulation parameter distri- bution: the randomized domain too narrow, resulting policies may become overly conservative or insuf- ficiently robust. Moreover, the comparison underscores that domain randomization and maximum-entrooy regularization induce robustness through complementary and orthogonal mechanisms.
4.2 Simulation
We evaluated MELRO across four simulated environments from the DMC suite (Tassa al., 2018), paired with five distinct perturbations. Four perturbation types torso and thigh length for "Walker Walk", and shin length and joint damping for "Quadruped Walk" have been adapted from the RWRL suite

---

Published in Transactions on Machine Learning Research (12/2025)
Walker
Cheetah
Disabled Joints
RWRL
900
CCC hip
right hip
back
front hip
Right
BACK Knee
Irgnt knee
frunt amble
right ankle
800
MaxEnt Base Policy
MaxEnt Base Policy
Reward
Quadruped
DR Base Policy
DR Base Policy
700
MELRO (ours)
MELRO tours)
Base Policy
Base Policy
back le't
beck right
600
Rollout
Rollout
Tront loft
front right Ince
Tront riggigttnnn anie
Tront right hip
500
Figure Disabled joints tasks. Vi- sualization of the Walker, Cheetah, and Quadruped joints, which have been indi- vidually disabled.
Figure 6: Results of all simluation experiments. Aver- age reward of all evaluated methods over all simulated envi- ronments and perturbations.
Disabled Joint Cheetah Run
Disabled joint Walker Walk
Disabled joint Quadruped Walk
Thigh Length Walkar Walk
1000
Back
Left
800
Reward
400
Right Back
ankle
hip
knee
ankle
hip
knee
ankle
hip
knee
0.125
0.15
0.35
0.4
Torso Length Walker Walk
Shin Length Quadruped Walk
Joint Damping Quadruped Walk
1000
Reward
400
0
0.1
0.45
0.5
0.55
0.75
1.0
2.0
5.0
3.0
5.0
7.5
10.0
Perturbation value
Base Policy
Rollout
MaxEnt Base Policy
MELRO tours)
Figure Individual results of all simulation experiments. Average reward and 95%-CI of the non- MaxEnt base policy, rollout, MaxEnt base policy, and MELRO for a subset of perturbation tasks from the RWRL and the DMC suite. The horizontal brackets indicate the positioning of the joints.
(Dulac-Arnold al., 2020). The fifth perturbation lype disabled joints for "Wakker Walk", "Quadruped Walk", and "Cheetah Run" has been extended Nagabandi et al. (2019). Even though the joints repeat for each extremity, we still observed differences in the performance of all evaluated methods, so we disabled the joints individually. This perturbation type introduces strong non-parametric disturbances, which move beyond simple noise. The joints for each environment. are visualized in Figure 5.
We ablate MELRO against MaxFnt base policy, the base policy trained without MaxEntRI, and standard rollout over the latter. Despite the evident advantage of policies trained under domain randomization in terms of the opportunity to encounter the specific validation perturbations during the MBRL phase, additional
8

---

Published in Transactions on Machine Learning Research (12/2025)
50
MaxEnt Model
Rollout
MELRO (ours)
Acc. absolute error
40
Non-MaxEnt Model
Proportion
30
0
200 0
200 400 600
5
9
15
19
Improvement over base policy (reward)
Prediction Step
Figure 9: MELRO vs. plain rollout im- provements. ECDF's of reward improve- ments of MELRO and standard rollout rel- alive to the respective base policy. More mass on the right translates to larger im- provements.
Figure 8: MaxEnt vs. Non-MaxEnt accumulated model prediction errors. Comparison of the absolute, accumulated error between the world model trained in MaxEnt framework and the one without for all perturba- lions.
comparison is made between MELRO and policies trained using domain randomization. The MaxEnt base policy, the base policy trained without entropy regularization, and their corresponding dynamics models were initially trained on standard DMC environments. We then picked the best-performing model, policy, and critic combination and optimized the rollout parameters for each perturbation through hyperparameter search. An evaluation of the hyperparameters and a sensitivity analysis can be found in Appendix B and D. The base policies using domain randomization have been trained directly on the respective perturbed simulations. During training, the specific disabled joint or perturbation value was randomly sampled from the entire domain. Final performance metrics were obtained by assessing each methodology across all perturbation tasks through ten independent random seed evaluations, each comprising ten independent rollouts.
Main Simulation Results The overall aggregated metrics visualized in Figure 6. The results for subset of perturbations of the entropy-regularized methods are shown in Figure 7, and of the domain randomization policies are shown in Figure 11 (Figures presenting all the perturbations are in Appendix G.2). In general, the MaxÉnt policies produce superior results to the non-MaxE't policies, and rollout enhances the performance of the respective base policy. This results in MELRO achieving the best overall performance compared with the non-regularized and entropy-regullrizeed methods. MELRO even outperforms the base policy trained using domain randomization in the disabled joint task.
Increased Inherent Robustness of MELRO The MaxEnt base policy has already demonstrated robust performance across a broad spectrum of perturbations, achieving consistently high returns relative to the perturbation, in particular for all "Quadruped Walk" environments and the "Walker Walk" thigh length perturbations. Nevertheless, there were still some perturbations where the MaxEnt base policy exhibited significant degeneration in performance, often underperforming the non-regularized base policy in those cases. This phenomenon is especially apparent in the disabled back hip joint perturbation of "Cheetah Run", the shin length perturbation of 1.0 for "Quadruped Walk", and the torso length perturbation of 0.5 for "Walker Walk", as shown in Figure 7. In the disabled back hip joint perturbation of " "Cheetah Run", the MaxEnt base policy even gets completely stuck at the beginning of all evaluated episodes. By contrast, MELRO consistently oulperformed the MaxEnt base policy across all tested perturbation scenarios, particularly in cases where the MaxEnt base policy's performance deteriorated significantly, thereby demonstrating an enhanced dynamic entropy adaptation capability.
Effects of Maximum-Entrow Regularization on World Models and Rollout world model using MaxEntRL. has already been demonstratod enhance exploration, consoquently leading more accurate world model (Ma al., 2022). As demonstrated in Figure 8, this approach also results in robust model and lower prediction errors. Additionally, the superior model yields superior control
The training of a

---

Published in Transactions on Machine Learning Research (12/2025)
Disabled Cheetah
Disabled Walker Walk
Disabled Walk
Thigh Length Walk
1000
800
Back
Front
Left
Right
Reward
600
400
200
ankle
hip
ankle
hip
ankle
hip
knee
knee
ankle hip
kneeankle hip knee
0.125
0.2
0.4
Torso Length Walker Walk
Shin Length Quadruped Walk
Joint Damping Quadruped Walk
800
Reward
600
400
200
0.1
0.3
0.5
0.25
1.0
3,0
3.0
7.5
20.0
Perturbation value
Base Policy DR
MELRO lours)
Figure 11: MELRO domain randomization perturbed simulation results. Average reward and 95%-CI of domain randomization and MELRO for subset of perturbation tasks from the RWRL and the DMC suite. The horizontal brackets indicate the positioning of the joints.
results. This is apparent from the more pronounced improvement of MELRO over its base policy, compared to the improvements shown by standard rollout (refer to Figure 9). A further explanation for this observa- tion could be provided by Ahmed et al. (2019), in which the authors demonstrate that maximum-entropp regulation smooths the optimization landscape. This would also benefit the online optimization of rollout in MELRO.
Limitations of Maximum-Entree Regularization Although maximum-entrope regularization has been demonstrated to improve robustness and exploration by introducing adaptive stochasticity in the ac- tion space, its effectiveness hinges on the environment. (Ahmed et al., 2019; Eysenbach Levine, 2022). Tn environments that require precise control, or where the perturbations differ just slightly from the training environment, maximun--nntropy regularization can negatively impacl performance. In our simulation experi- ments, this is evident in the "Cheetah Run" disabled back knee and front ankle, well some "Walker Walk" tasks, where both the MaxEnt base policy and MELRO perform slightly worse than the standard basc policy.
Disabled Joints
RWRL
Comparison to Domain Randomization
1000
As shown in the aggregated mean rewards in Figure 6, domain randomization achieves the best performance under the RWRL perturba- tion tasks, whereas MELRO outperforms it in the disabled joint settings. Overall, the in- dividual results in Figure 11 suggest gen- eral trend: Domain randomization is more effective in handling extreme perturbations, while MELRO provides superior robustness in scenarios where the perturbations have a smaller impact on the dynamics. notable exception is the disabled joint "Quadruped Walk" task, where MELRO demonstrates sig- nificantly higher robustness, particularly when
750
Reward
500
250
0
2
1e6
1e6
Environment Steps
Domain Randomization
MaxEnt Base Policy
Figure 10: MaxEntRL vs. domain randomization training curves. Mean and 95%-CI reward training curves for the base policies trained using MaxEntRL and domain randomization aggregated across all evaluated tasks.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Figure 12: Real-world Franka trajectory. Example of a real-world trajectory of the Franka Panda. robot arm, controlled by MELRO. The movement took approximatcly five seconds.
12
11.4
Abs. Goal Distance [cm]
11
15
MaxEnt Base Policy MELRO (ours) Zero difference
10.5
10.1
10-
MaxEnt Base Policy
MELRO w/o entropy
Difference [cm]
10
9
MELRO (ours)
5
8
7
-5
6
0
3
6
12
15
Time [s]
Figure 13: Overview of the "sim to real" results for the Franka Panda reach task. Mean of the absolute goal dis- tances in cm of the MaxEnt base policy, MELRO without entropy regularization during planning, and MELRO on the real-world Franka Panda.
Figure 14: MaxEnt base policy vs. MELRO individual differences after the "sim to real" transfer to the Franka Panda. Difference in em between the absolute distances between MELRO and the target, and between the MaxEnt base policy and the larget, for all test runs on the real Franka Panda arm. Values above zero mean that MELRO was closer to the target than the MaxEnt base policy. Vice versa for values below zero.
joints from the two front legs are disabled. This phenomenon can potentially be attributed the extensive variability induced by domain randomization in this task, forcing the policy to compromise across wide range of possible dynamics. The reward training curves in Figure 10 indicate that training using domain randomization requires more environment interactions until the policy converges, and is, in general, noisier. However, this must be taken with a caveat, as the reward for the domain randomized policy is computed in perturbed environments.
4.3 Franka Panda Robot Arm
We evaluated MELRO's "sim to real" capabilities by testing it on a simple reach task using the Franka Emika Panda robot arm. The objective was to maneuver the end-effector the end of the robot arm randomly selected position in front of the robot. To train the components in simulation, task in MuJoCo (Todorov et al., 2012) using the Franka Panda MuJoCo Menagerie model (Zakka et al., 2022). The reward function has been selected as the negative sum of squares of the difference between the current end-effector position and the goal, minus a scaled penalty for rolating the end-effector out of a neutral orientation. is important to note that no further reward shaping has been applied. Here, all policies the base policy, rollout, MaxEnt base policy, and MELRO have been trained to generate joint velocity actions given the current joint angles, joint velocities, and the current goal. Detailed information about the simulation is given in the Appendix E. The same training process explained in Chapter 4.2
modeled the
11

---

Published in Transactions on Machine Learning Research (12/2025)
has been used. During deployment, all policies generated new joint velocity actions at frequency of 10 Hz. These were then converted into joint torque commands using PD-controller running at 1 kHz.
Although we evaluated multiple sets of hyperparameters for both the base policy and rollout, none of them produced stable and safe actions on the real robot. Since we did not apply extensive reward shap- ing, we believe that exploration for the non-MaxEnt components during the MBRL stage was insufficient. Consequently, MELRO was only evaluated against the MaxEnt base policy and MELRO without entropy regularization during planning. The mean distance for each method over the 20 test runs on the real Franka Panda is shown in Figure 13. Additionally, Figure 14 visualizes the differences in the absolute distances be- tween MELRO and the target, and between the MaxEnt base policy and the target, for cach test trajectory individually.
MELRO tightens the "sim to real" gap On average, MELRO was 11.7% closer to the goal than the MaxEnt base policy. This was achieved through faster convergence and generally bringing the end-effector closer to the goal. As can be seen from MELRO's good performance without entropy regularization, rollout already significantly improves the MaxEnt base policy. Additionally, the introduction of maximum-entropy regularization during planning further decreased the overall distance to the goal.
5
Related Work
5.1 Maximum-Entropp RL
It is well-known that MaxEutRL improves exploration (Haarnoja et al., 2017; Huang et al., 2020). This approach is also widely adopted, underpinning prominent model-free algorithms like SAC (Haarnoja et al., 2018), influencing implementations of methods like PPO (Schulman et al., 2017), and featuring in model- based approaches such as Dreamer (Hafner et al., 2023) and TD-MPC2 (Hansen et al., 2024). The primary distinction between the usage of MaxEntRL in MELRO and the aforementioned algorithms is that MELRO incorporates maximum-entropy regularization in the online planning objective.
Recent theoretical work has established that MaxEntRL implicitly optimizes a lower bound on a robust RL objective, conferring inherent resilience against perturbations in dynamics and rewards without the need for specialized robust optimization techniques (Eysenbach & Levine, 2022; Brekelmans et al., 2022). Despite this advantage, the resulting policies can sometimes be overly conservative, potentially sacrificing peak performance (Josifovski et al., 2022). Furthermore, standard MaxEntRI, lacks explicit mechanisms for targeted adaptation, potentially hindering swift responses to rapidly evolving environments. Our work aims to address these limitations by harnessing the benefits of MaxEntRL while enabling more controlled robustness and faster adaptation by adding maximum-entropy regularized online planning.
5.2 Robustness in Model- Based Reinforcement Learning
In model-based reinforcement learning (MBRL), model inaccuracies are commonly addressed by incorporat- ing model uncertainty into the learning objective (Janner et al., 2019; Wang et al., 2024) or directly into the decision-making process ( (Chua et al., 2018; Wu et al., 2022). Epistemic uncertainty is typically estimated by ensemble disagreement, but can also be predicted directly by the model. Although this method encourages the selection of actions that are more predictable and therefore likely to be more robust, it also leads to less exploration and, consoquently, to suboptimal policics. Additionally, all methods rely on accurate estimates of uncertainty. Particularly for longer planning horizons, reduced performance and sampling elficiency have been observed as the inevitable model inaccuracies are interpreted as model uncertainty (Wang et al., 2024). Given that MELRO employs MaxEnt regularization, it is not subject to these limitations.
6 Conclusion
We introduced MELRO, a novel, inherently robust control method that combines maximum-cntropy rein- forcement learning (Haarnoja et al., 2018) with rollout, an online planning method (Tesauro, 1994). Instead of relying extensive domain randomization to bridge the "sim real" gap, explored methods that are
12

---

Published in Transactions on Machine Learning Research (12/2025)
inherently more robust to such deviations from the model at hand. It is hence able to cope with variations that are unknown a priori. Experimentally, this approach significantly enhances system robustness against diverse and unforeseen dynamical perturbations, as well as the success of "sim to real" transfers. Our results clearly confirm that rollout consistently improves performance and robustness across various scenarios, sub- stantiating and extending the findings presented by Bertsekas (2024). Another appealing aspect of MELRO is its capacity for adaptive online regularization; regularizing the lookahead reward during online planning allows for the on-the-fly modification of the entropy weight, thereby significantly facilitating and enhancing the computational efficiency of identifying the optimal regularization trade-off in comparison to conventional MaxEnt and domain randomization. We believe that our work constitutes an important contribution to the research and deployment of autonomous agents that are robust to the unexpected.
References
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization, 2019. URL htcps://artt..orr/bb////////
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.064B3 2016. URL http:///xxvvvvooraab//
06450.
Justin Bayer, Maximilian Soclch, Atanas Mirchev, Baris Kayalibay, and Patrick van der Smagt. Mind the gap when conditioning amortised inference in sequential latent-variable models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
URL wtw.pppwwweiiieeeeeeeterrrraaaaa
Richard Bellmann. A markovian decision process. Journal of Mathematics and Mechanics, 6(5):679 684, 1957. ISSN 00959057, 19435274.
Dimitri Bertsekas. Newton's method for reinforcement learning and model predictive control. Results in Control and Optimization, 7:100121, 2022. ISSN 2666-7207. doi: https///0bl.og//0.000///.ttttt 200200 2022.. 100121. URL https://www..sieneeeeeee http://ww.........c.....oco/a/aa
Dimitri P. Bertsekas. Model predictive control and reinforcement learning: A unified framework based on dynamic programming. CoRR, 20////200000222 2024. doi: 10.48550/ WAA...X2V0..000.... URL https:
org//orrg/0, 48550/arXiv. 2406. 00592.
Leo Breiman. Manual setting up, using, and understanding random forests v3. 1. Stalistics Department University of California Berkeley, CA, USA, 1(58):3-42) 42, 2002.
Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, and Pedro A. Ortega. Your policy regularizer is secretly an adversary. Trans. Mach. Learn. Res., 2022, 2022.
URL ptttps://openrevieet httppp///ppaaeeeeeeeeeee//o/ottp
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 31: Anruaa Conference on Newral Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 4759-4770, 2018. URL htttp:://proceeeinntt http://////////aaaaaooooooaaaaaa wtteeee--bbtttttttttaaaaaea
Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2024. URL http://github. com/Farram-Founddtioon wat.ii/oon/naaaiiummmaaaa Roooomaaaii--ooooiicc...........
Gabricl Duloc-Aroold, Nir Levine, Danicl J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. CoRR, abss/2003.1888) 2020. URL https://arxi:: https://arxxvvorg/abb 22000 11881.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Gabriel Dulac-Arnold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement. learning: definitions, benchmarks and analysis. Mach. Learn., 110(9):2419 2468, 2021. 100994-021-05900
doi: 10.1007/110991-02--66900-- URL http:://aoi.org/10.1000 org/10.1007/
Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nee net, 2022. URL https://openreviieee wttpr//iiiw...eefformmmaaaasSSS.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1352-1361. PMLR, 2017. URL http://prooceddings html.
mlr.
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jemnifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, 1CML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856- html.
1865. PMLR, 2018. URL http: /estp:/pprocee http://pcoceee...eeee/// mlr. press//80/haarno
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P. Lillicrap. Mastering diverse domains through world models. CoRR, abs/2301 htss/300..04104, 2023. doi: 10.48550/ARXI 230.X0..300.01004.. URL https://doi. org/ 10. 1448550/aaxiv 2301 04104.
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httth::://opeeeeeeeeee.o httpppaaeeeee...eecccoooaaaaaaaa
Shiyu Huang, Hang Su, Jun Zhu, and Ting Chen. SVQN: sequential variational soft q-learning networks. In 8th International Conference on Learning Representations, TCLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevvew.nnt, 2020. URL https:
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. CoRR, abs/1905.06424, 2019. URL http://arxiv/ 06424.
Michael Janner, Justin Fu andx Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information. Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pp. 12498-12509, 2019. URL hash/5faf461em (ttaa444..ee...09900776666666360 atsaaaacc....e444447ttttttt.....
wtwpeeeddinnn..uuipsss.e neurips. c2/paper/2019/
Josip Josifovski, Mohammadhossein Malmir, Noah Klarmann, Bare Luka Zagar, Nicolás Navarro-Guerrera Guerrero, and Alois C. Knoll. Analysis of randomization elfects on sim2real transfer in reinforcement learning for robotic manipulation tasks. In JEEE/RSJ International Conference IROS 2022, Kyoto, Japan, October 23-27, 2022, pp. 10193 10200. IEEE, 2022. doi: 10.1109/IROS47612. 2022.9981951. URL attps://doi: org/ 10. 1119 11099//00/76112..11 2022. 9981951.
Intelligent Robots and Systems,
Josip Josifovski, Sayantan Auddy, Mohammadhossscn Malmir, Justus H. Piater, Alois Knoll, and Nicolás Nuverro-Gueerrroo. Continual domain randomization. In JEEE/RS] International Conference on Inlelli- gent. Robots and Systems, IROS 2024, Alu Dhabi, United Arab Emirates, October 14-18, 2024, pp. 4965- 4972. IEEE, 2024. doi: 10.1109/1RO0558 200.05559222224.0880000 2024. 10802060.
URL https::/doii http::://dooooo/////// org/10..///0 1109/ 10t.11109/RRSS59... IROS58592.
Leslie Pack Kaelbling, Michael I. Littman, and Anthony R. Cassandra. Planning and acting partially observable stochastic domains. Artif. Intell., (01()-)):99-134, 1998. doi: 3000/0000000300200002223
URL https: /tttp:://bai.oor/1111111111 http://...111//000000 00023xX
14

---

Published in Transactions on Machine Learning Research (12/2025)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-real reinforcement learning for vision-based dexterous manipulation on humanoids. CoRR, abs /20//2252222999 2025. doi: 10.48550/A 418550/ARXIV. ARXIV. 2502.20396. URL https://doi. 10./10..// 000.1..885///// arXiv 2502. 20396.
Hongying Ma, Wuyang Xue, Rendong Ying, and Peilin Liu. Maxent dreamer: Maximum entropy reinforce- ment. learning with world model. International Joint Conference on Neural Networks, IJCNN 2022, Padua, Italy, July 18-23, 2022, pp. 1-9. IEEE, 2022. doi: 10.1109 http///JCNNG550022222222222222 https://doi.org: 100.111100//ICI5566.4.
URL
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 9-8, 2018, Montréal, Canada, pp. 1805 1814, 2018. URL https://procedding wewpeediinnnnn.uriippe wwwuriss.cc/papr///0////// Awteea-tttttttteeeeeeeeeeeeeeeee html.
Nicolò
Lars M. Mescheder, Sebastian Nowozin, and Andreas Geiger. The numeries of gans. In Tsabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neurol Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1825 1835, 2017. URL https:// Abstract chtmlalcct.tll......... html.
proceedings senceedings..neuips.... neurips. cc/pappr/2017/ httph//hhh//58//4455/5555 /BB88///AAAA11115556644444411110
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: httpps://openreviewt http://wwww....eeeee//orratttata
NVIDIA. NVIDIA Multi-Instance GPU User Guide, 9.2 A100 MIG Profiles. https://docs https://doos.nvidia.co nvidia. Batacenter/tesla/niii -uterr-guddd//ttl---iii---------
2025. [Accessed 16-10-2025].
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. CoRR, 063447,7 2017. URL http:// org. ab1/0777 107bb0067767777.
Wandong Sun, Baoshi Cao, Long Chen, Yongbo Su, Yang Liu, Zongwu Xic, and Hong Liu. Learning perceptive humanoid locomotion over challenging terrain. CoRR, 0aln//533.0099 2025. doi: 10.48550/ ARXIV 2503.00692. URL https://doi. org/10. 48550/8550500 48550/arXiv....////aaa www.v..ww................ 2550300693
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Frez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, Abs/1801.00690, 2018. URL http: http//axxvvv..r/ /wtp:/awwwww////////...../ abs///ab//.11.999
Gerald Tesauro. Td-gammon, self-teaching backgammon program, achieves master-level play. Neural Comput., 6(2):215 219, 1994. doi: 10.116//NECO.1994.66 1994. 2.2.2.... 215.
URL https://doi. wtt.://...rr/.....//eeeeee.. 1162//eco.........
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ International Conference on Intelligent Robols and Systems, IROS 2017, Vancouver, BC, Canada, September 24-28, 2017, pp. 23 30. IEEE, 2017. doi: https:///oi..o wt.//..///1//........ IRt///ROSS... 2017 021778820133...
URL
Emanuel Todorov, Tom Ercz, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10. 001100//ROS.2012.000
15

---

Published in Transactions on Machine Learning Research (12/2025)
Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang. Coplanner: Plan to roll out conservatively but to explore optimistically for model-based RI. Tn The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.ner net, 2024. URL https://openreet
jack
Zifan Wu, Chao Yu, Chen Chen, Jianye Hao, and Hankz Hankui Zhuo Plan to prediet: Learning uncertainty-foreseenngg model for model-based reinforcement learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advences in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orlewns, LA, USA, November 28- - December 9, 2022, 2022. URL http:: www.niip...ppppeeee files/ html.
Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors, MuJoCo Menagerie: A collection of high-quality simulation models for MuJoCo, 2022. URL httt:t////ggthbb.... wwwp:///oollleeedmdddmmmmmmm
menagerie.
Ziwen Zhuang, Shenzhe Yao, and Hang Zhao. Humanoid parkour learning. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard (eds.), Conference on Robot Learning, 6-9 November 2024, Munich, Germany, volume 270 of Proceedings of Machine Learning Research, pp- attps://prooeedinggs mlr. www.ppress/vvvie w2200/huang25a...mm html.
PMLR, 2024. URL
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Dicter Fox and Carla P. Gomes (ods.), Proceedings of the Twenty- Third AAAI Conference on Arlificial Intelligence, AAAI 2008, Chicago, Ilinois, USA, July 13-17, 2008, pp. 1433 1438. AAAI Press, 2008. URL http:// //tp://wwwww... aaai, atg/Librarr/AMAI/2066 Maai08-20 .hp..
16

---

Published in Transactions on Machine Learning Research (12/2025)
A MELRO in Detail
In Algorithm 1 and in Algorithm 2, we provide a detailed description of the off-linc training and on-line play of MELRO.
Algorithm 1: MELRO (off-line training)
input
Oer fr, Ogs th X; initial transition, reward, inference, policy and critic parameters
w. expected return weighting factor and discount factor
Minit Timsi: number of samples, and time steps for model pre-training
ne: Traiin: number of samples, and time steps for model training
D, D: optimization steps for model and for policy and critic
Holisyys policy: number of samples and time steps for policy and critic training B: replay buffer
entrrainn unperturbed training environment.
output: ro, and q0, learned transition, reward and inference functions To. Ux: learned neural base policy and critic
begin
// Pre-train model:
Sample Rinit trajectories (T (nit) Imnes each with length Timi from enctrain and random policy
B = BU {I ) // Add initial trajectories to buffer
while not converged do
1/ Train model:
(Thh:Hmaae
// Sample trajectories from buffer
for d d= D do
Update model parameters using the ELBO and Adam end
10
11
// Update policy and critic:
for d d= 1.....,, D do
13
Sample H policy trajectories (T)iii each with length T Poticy from pa and Po using ma Compute TD(A) returns R using Equation 11 Vt E {1 Tpoticy] } and vh E {1 Hpolicy) Update policy and critic parameters using Equation 14 and Adam
14
15
18
end
17
Sample n, trajectories (T)mm each with length Tirain from enttrain and T B = U (T) // Add trajectories to buffer
18
19
end
end
17

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm 2: MELRO (on-line play)
input To, and qa, learned transition, reward and inference functions Ty. Ux: learned neural base policy and critic
3, D: learning rate and optimization steps for MPC
eWVlepy perturbed deploy environment.
output: (01,...,r)): actions
begin
Sy -WWdeploy- reseet))))resst() for -= 1,..., T do
Sample initial action mean and standard deviation trajectory rę- = [(a = ... 624 and Ps,
from Te
5
Add small uniform noise to ro. for d 1..... do
Compute search direction Arg based on Equation 15 starting from rØ-1 using ARS, re, and
7
1/x Update action distribution parameter trajectory 1 in the direction Art using Adam end
10
a, ()(,,))) = 2c) // Sample action
11
81+1 = enUpeploy.slep((o))
12 end
13 end
18

---

Published in Transactions on Machine Learning Research (12/2025)
B Hyperparameter
In this scction, we list the hyperparamcter search space and the optimal paramcters found for all the methods used. Note that the entropy weight for the base policy for the policy parameters, as well as for rollout, was fixed to zero. Additionally, the exploration noise, which describes the standard deviation of a zero-centered Gaussian noise, has been added to the actions to enable exploration only in the non-entrroyy-regulriiee MBRL setting.
Table 1: MBRL Hyperparameter
Search Space
Maze
Simulation
Franka
Critic Parameter
max iter
1000
activation
softssggn elu
softsign
softsign 512
elu 1024
n hidden
[128, 256, 512, 1024
256
11 layers
2
Policy Paramcter
exploration noise
0.0, 0.05, 0.1]
0.1
0.0
0.01
step_size size
3e-4, le-4]
3e-4
3e-4
3e-4
max iter
200, 40, 60, 100, 120]
60
100
60
activation
sof softsign
n hidden
[64, 128, 256, 512]
256
64
64
n layers
2, 3]
2
3
discount
00.95, 0.975, 0.99, 0.995)
0.99
0.975
0.95
entropy weight
0.00, 0.0001, 0.005, 0.01, 0.05, 0.1]
3
0.01
0.0001
n samples
(64, 128, 256]
128
64
I time steps
[4 6, 8, 16, 24, 32]
6
lambda
0.9, 0.95, 0.99)
0.95
0.95
0.99
max grad norm
1, 10, 100]
100
10
100
Model Parameter
batch size
32, 64, 128, 256)
256
32
32
emission scale max grad norm
0.01, 0.05, 0.1, 0.2)
0.2
0.01
0.01
10, 100, 1000]
100
100
100
max iter
10, 20, 40, 100)
40
100
n time steps step size
4, 6, 8, 10)
4
10
3e-3, 1e-3, 3e-4, 1e-4]
3e-4
3e-4
3e-4
initial iter
100, 250, 500, 1000)
1000
100
250
rmn activation
softsign
rmn n hidden mlp activation
24, 64, 128
24
128
sof softsign
mip_1 n hidden
128, 256, 512)
256
256
128
mip n layers
2, 3)
3
MBRL Parameter
action _epeatt
2
conservativity critic Beightt_ehttttt
0.01, 0.05, 0.1
0.05
0.05
0.05
00.05, 0.1, 0.25, 0.5, 1.0)
0.05
1
0.25
n iter per _er__verllll eval
1e6
n steps per eval max env env_steps steps
10000
500_000
n collect steps initial time steps
[10, 20, 40, 100]
100
40
100
[100, 500, 1000, 2500)
100
1000
1000
We used the following search space for the entropy_weight for the maze [1, 2, 3, 4, 5,
19

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Rollout Hyperparameter Search Space
Parameter
Values
entropy weight
0.0, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1
base_ cacle__pllc____eeeeeeeeeeeeeee
0, 1, 2, 4, 6, 8]
ars n perturbations
8, 12, 16, 24]
ars std ars top_
0.05, 0.1, 0.15, 0.2, 0.25, 0.3
2, 4, 8]
discount
008, 0.9, 0.95, 0.999]
optimize steps
1, 2, 5, 10, 15, 20, 35, 45)
parallel rollouts
1
planning steps
[1, 2, 4, 6, 8, 12, 20)
optimizer
Adam
step size
00.001, 0.003, 0.01, 0.03, 0.1, 0.125, 0.15, 0.175, 0.2, 0.25
20

---

Published in Transactions on Machine Learning Research (12/2025)
C Analysis of the computational costs
Although the many benefits of rollout are evident, performing online optimization can quickly become computationally expensive. In order to run MELRO at 10 Hz, as stated in Section 4.3, we implemented all components of MELRO in JAX. We further compile all computationally intensive functions, such as all world model functions, online optimization, and base policy calls, using JAX's jit functionality. Additionally, we deployed MELRO for inference on a MIG 1g.10gb partition of an NVIDIA A100 GPU (compare NVIDIA (2025). This not only makes inference of the neural components faster, but also enables ARS to be completely parallelizable. As result, the computational time of MELRO scales only linearly with the number of planning steps, the nunber of optimization steps, and the number of base policy steps, visualized in Figure 15. By keeping those parameters as small as possible, MELRO can operate reliably at 10 Hz, as shown in Table 3. The same table also shows that entropy regularization during planning increases the inference time only marginally, and that the MaxEnt Base Policy can still run around ten times faster than MELRO.
Avg Inference Time (s)
0.20
0.15
0.05
10
blbbase_ppe b__bae_poolic_sste_ policy_steps
ars_n perturraatssss
n ors_n_top_
Avg Inference Time (S)
0.20
0.15
0.10
0
10
20
30
40
10
15
20
n_optimize_stte steesssssssssee
n_ n_planning_stepa steps
Figure 15: Computational Costs MELRO. Average inference times of MELRO for different hyperpa- rameters computed during the evaluation of the Franka experiments. The red dotted line shows the mean inference time for the optimal hyperparameters used for the Franka experiments.
Table 3: Inference time MELRO. Statistics of the inference time in seconds of the MaxEnt base policy, MELRO without entropy regularization, and MELRO using the hyperparameters used for the Franka cx- periments.
Method
average
min
max
standard deviation
MaxEnt Base Policy
0.00955
0.008731
0.013814
0.000678
MELRO w/o entropy
0. 0.105343
0.104021
0. 0.109615
0.000668
MELRO (ours)
0.105479
0.10413
0. 0.109755
0.000667
21

---

Published in Transactions on Machine Learning Research (12/2025)
D Sensitivity Analysis
20.0
17.5
Percentage of Experiments
0.15
12.5
0.10
10.0
7.5
5.0
0.00
n o.optinize steps entropy weight
n "_planning. steps
step size
n basee palpcyyllyyccc steps
perturb control scale
ars std discount
ars perturbations
ars top k
2.5
0.0
0.0
0.0001
0.0005
0.001
0.005
0.01
0.05
0.1
0.5
1.0
Entropy Weight
Figure 16: Importance analysis rollout pa- rameters. Mean Decrease Impurity (MDI) of all rollout features for all experiments. Higher value translates to higher importance.
Figure 17:
Scnsitivity analysis entropy
weight in rollout. Ratios of the entropy reg- ularization weight of the top-5% experiments of each evaluated environment.
Mean reward below 800
Mean reward above 800
20.0
Percentage of Experiinents
15.0
12.5
10.0
7.5
5.0
2.5
0.0
0.0001
0.0005
0.001
0.005
0.01
0.05
0.1
0.5
1.0
0.0
0.0001
0.0005
0.001
0.005
0.01
0.05
0.1
0.5
1.0
0
Entropy Weight
Entropy Weight
Figure 18: Sensitivity analysis entropy weight in rollout. Ratios of the entropy regularization weight
of the top-5% experiments per perturbations divided into groups with mean reward below and above 800.
To igves the role of meximum-entropy regularization during planning, we first trained random forest model on the complete set of conducted experiments, using the rollout parameters as input features and the final reward as the prediction target. Feature importance was then quantified using the Mean Decrease Impurity (MDI) metric (Breiman, 2002), with results presented in Figure 16. In a complementary analysis, we examined the distribution of entropy weights among the top-5% performing experiments for each perturbation and environment, with the resulting ratios shown in Figure 17. In order to differentiate between entropy regularization levels depending the mean reward of the top-5% experiments into two groups: mean reward below 800. The results are shown in Figure 18. Together, these evaluations highlight that entropy weight is the second most influential rollout hyperparameter, that moderale entropy regularization consistently improves planning performance, but also that excessive regularization degrades performance by inducing overly stochastic action distributions. Furthermore, the case of less severe perturbations, small entropy regularization is more beneficial. Conversely, for more severe perturbations, greater entropic actions more effective.
the severity of the perturbation, with mean reward above 800 and one with
separated the perturbations based
22

---

Published in Transactions on Machine Learning Research (12/2025)
E
Franka Panda Robot Simulation Details
Figure 19: Franka reach goal region. Visualization of the goal range in the Franka Reach MuJoCo simulation. The red dots mark the corners of the cube where the goals are sampled from.
As mentioned in the main text, the Franka Panda MuJoCo Menagerie model (Zakka et al., 2022) has been used to train the methods in simulation. The desired goal position g has been sampled for each trajectory uniformly from cube relative to the base of the robot arm:
0.2
0.5
~~
0.35
0.35
(16)
0.15
0.6
The borders the box are defined in meters. Figure 19 visualizes the goal cube in the simulation. Besides the goal position, a fixed straight downward-facing quaternion 94 = (0, 1, 0) has been used to penalize any other end-effector rotations. The reward for end-effector position and rotation Sq has then been computed as follows:
Sq -
-6 (s: 9i)² minils, 9ol6 1s, + 9
(17)
using the antipodal symmetry of quaternions.
23

---

Published in Transactions on Machine Learning Research (12/2025)
F
Additional Task Visualizations
Figure 20 provides some additional task visualization of different perturbation types and values for the Real World Reinforcement Learning benchmark.
0.125
0.2
0.25
0.3
0.4
0.15
0.25
0.35
0.45
0.55
7
(a) Walker Walk thigh length
(b) Walker Walk torso length
0.25
0.5
1
2
5
(c) Quadruped Walk shin length
Figure 20: RWRL tasks. Visualization of some RWRL perturbations.
24

---

Published in Transactions on Machine Learning Research (12/2025)
G
Additional Experiments Results
G.1
Toy Example
MELRO (No Entropy)
MELRO (Low Entropy)
MELRO (Optimal Entropy)
MELRO
Base Policy
(High Entropy)
709
710
697
668
613
Original
50
52
*
52
443
394
Perturbed
No entropy MBRL setting.
MELRO (No Entropy)
MELRO (Low Entropy)
MELRO (Medium Entropy)
MELRO
MaxEnt Policy
(High Entropy)
493
601
592
553
520
Original
217
205
226
251
263
Perturbed
(b) MaxEntRL MBRL setting.
Figure 21: Entropy regularization toy maze. 100 evaluation trajectories the no entropy base policy, MaxFnt base policy, and MELRO with different entropy regularization weights for the unperturbed training maze and the perturbed evaluation maze. The components of MELRO have been trained here in the standard MBRL setting in (a) and in maximum-entropy regularized MBRL sctting in (b). The mean reward for each method and environment is shown in the top left corner of every maze. The goal is visualized by the red star.
Impact of Entropy Regularization Weights for Planning Figure 21 compares the trajectories gen- crated by MELRO with different entropy regularization weights during planning. The components were trained in two frameworks: a slandard MBRL framework (Figure 21a) and a maximum-nntropp--rggallee MBRL framework (Figure 21b). In addition to the results presented in Section 4.1, these findings demon- strate that MELRO enhances performance low as well as in highly maximum--ntroo--rgglaaaia settings. Furthermore, the trajectories demonstrate that MELRO achieves superior outcomes in the original training maze, even without with low entropy regularization during planning. This finding indicates that
MBRL
25

---

Published in Transactions on Machine Learning Research (12/2025)
rollout itself can improve performance for specific perturbations if the model, base policy, and critic have been trained using Ma
Base Policy
MaxEnt Base Policy
MELRO
-3.0
1.4
33,5
1.2
1.2
4.0
1.0
1.0
mean entropy
4.5
(000 entro
0.8
5.0
0.6
0.6
- 5.5
- 6.0
0.2
0.2
-7.0
0.0
Figure 22: Entropy visualization toy maze. Hexbin heatmaps showing the mean entropy values at different positions in the toy maze of 100 evaluation trajectories of the low entropy base policy, MaxEnt base policy, and MELRO. The goal visualized by the red star.
Entropy Visualization As shown by (Eysenbach & Levine, 2022), maximum-entropp regularization ables dynamic adaptation of the action noise level according to the current state. This enables the policy to reduce the entropy level, provided that the reward can be increased accordingly. As illustrated Figure 22, this phenomenon is also evident in the entropy levels of the low entropy base policy in the toy maze. At the beginning of each episode and when the point mass collides with the blocks the middle of the maze, the entropy values are small. They then increase as the velocity increases. However, this is only possible the reward signal is sufficiently strong to compensate for the maximum-entroop regularization penalty. This is particularly challenging at the initial positions of the maze, due to the minimal change in reward resulting from the exponential reward function. Evidence of this phenomenon is apparent in the entropy heatmap of the MaxEnt base policy in Figure 22. At the start of each episode, the MaxEnt base policy consistently generates high entropy values, causing the pointmass to be stecred in all directions. is only when the pointmass is closer to the goal that the MaxEnt base policy adapts the entropy values.
As illustrated in Figure 22, MELRO enhances entropy regularization in distinct ways. Firstly, the overall entropy scale is increased to match the scale of the MaxEnt base policy, which introduces more diverse paths and ultimately enables the pointmass to overcome obstacles. Secondly, despite the overall increase in entropy, MELRO can maintain dynamic entropy adaptation in manner analogous to the low entropy base policy. This enables MELRO to steer the pointmass with greater precision towards the target, particularly at significant states such as the beginning of each episode, in the proximity of the blocks, and after passing them.
26

---

Published in Transactions on Machine Learning Research (12/2025)
G.2
Simulation
In this chapler, provide the complete results for all simulated experiments.
Disabled Joint Cheetah Run
Disabled Joint Walker Walk
1000
1000
Back
Front
Left
Right
800
800
Reward
600
Reward
600
400
400
200
200
0
0
ankle
hip
knee
ankle
hip
knee
ankle
hip
knee
ankle
hip
knee
Disabled Joint Quadruped Walk
1000
800
Reward
600
400
200
Right
Left
Right
Left
Back
Front
0
ankle
hip
knee
ankle
hip
knee
ankle
hip
knee
ankle
hip
knee
Disabled joint
Base Policy
Rollout
MaxEnt Base Policy
(ours)
Figure 23: Disabled joints results. Average reward and 95%-CI CT for all disabled joint perturbations. The horizontal brackets indicate the positioning of the joints.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Thigh Length Walker Walk
1000
800
Reward
600
400
200
0.125
0.15
0.175
0.2
0.25
0.3
0.35
0.4
Torso Length Walker Walk
1000
800
Reward
600
400
200
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
Perturbation value
Base Policy
Rollout
MaxEnt Base Policy
MELRO (ours)
Figure 24: Walker RWRL perturbations results. perturbations.
Average reward and 95%-CI for all Walker RWRL
Length Quadruped Walk
1000
800
Reward
400
0.25
0.5
0.75
1.0
1.25
2.0
3.0
5.0
Joint Damping Quadruped Walk
1000
800
Reward
600
400
0
3.0
5.0
7.5
10.0
20.0
30.0
40.0
50.0
55.0
60.0
65.0
70.0
85.0
Perturbation value
Base Policy
Rollout
MaxEnt Base Policy
MELRO (ours)
Figure 25: Quadruped RWRL perturbations results.
Average reward and 95%-CI for all Quadruped
RWRL perturbations.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Disabled Joint Cheetah Run
Disabled Joint Walker Walk
1000
1000
Back
Front
Left
Right
800
800
Reward
600
Reward
600
400
400
200
200
0
0
ankle hip kneeankle hip knee
ankle hip kneeankle hip knee
Disabled Joint Quadruped Walk
1000
800
Reward
600
400
200
Right
Left
Right
Left
Back
Front
0
ankle hip
knee ankle
hip
knee ankle hip knee ankle hip knee
Disabled joint
Base Policy DR
MELRO (ours)
Figure 26: Disabled joints results. Average reward and 95%-CI CI for all disabled joint perturbations. The
horizontal brackets indicate the positioning of the joints.
29

---

Published in Transactions on Machine Learning Research (12/2025)
Thigh Length Walker Walk
1000
800
Reward
600
400
200
0
0.125
0.15
0.175
0.2
0.25
0.3
0.35
0.4
Torso Length Walker Walk
1000
800
Reward
600
400
200
0
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
Perturbation value
Policy (ours)
Figure 27: Walker RWRL perturbations results. perturbations.
Average reward and 95%-CI for all Walker RWRL
Shin Length Quadruped Walk
1000
800
Reward
600
400
200
0
0.25
0.5
0.75
1.0
1.25
2.0
3.0
5.0
Joint Damping Quadruped Walk
1000
800
Reward
600
400
200
0
3.0
5.0
7.5
10.0
20.0
30.0
40.0
50.0
55.0
60.0
65.0
70.0
85.0
Perturbation value
Policy
Figure 28: Quadruped RWRL perturbations results. Average reward and 95%-CI for all Quadruped RWRL perturbations.
30

---

Published in Transactions on Machine Learning Research (12/2025)
Disabled Joint
Disabled Joint Walker Walk
Disabled Joint Quadruped Walk
Thigh Length Walker Walk
Cheetah Run
1000
Reward
500
250
1
2
3
2
3
3
2
3
le6
1e6
1e6
1e6
Length
Length Quadruped Walk
Joint Damping Quadruped Walk
Walker Walk
1000
hw wwnnnnn
Reward
500
250
2
3
0
2
3
2
3
le6
le6
1e6
Environment Steps
Domain Randomization
MaxEnt Base Policy
Figure 29: MaxEntRL domain randomization training curves. Reward training curves for the base policies trained using MaxEmRRL and domain randomization for all evaluated tasks individually.
31


Reviews:
[{"review_id": "BWqLAzBd0G", "reviewer": "Reviewer_c2f6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\n\nThis paper proposes MELRO (Maximum-Entropy Learning-Based Rollout), a control framework that couples maximum-entropy RL with online rollout planning. Offline, a variational state-space world model, a stochastic Gaussian policy, and a critic are trained with MaxEnt regularization. Online planning is performed in the space of the policy’s action-distribution parameters with an entropy-regularized objective and a truncated base-policy rollout before bootstrapping with the critic. Experiments on DMC tasks with RWRL-style and non-parametric “disabled joint” perturbations show MELRO improves over a MaxEnt base policy and over standard rollout. A Franka Panda reaching task reports lower goal distance than the MaxEnt base policy.\n\n## Strengths\n\n1. Methodological and design choices. Authors propose clear and principled integration of MaxEnt RL and rollout, including planning in $(\\mu,\\sigma)$ space and explicit entropy regularization during planning.\n    \n2. Broad empirical probe of robustness. Evaluations cover parametric (length, damping) and non-parametric (disabled joints) disturbances, with consistent gains vs. baselines and 95% CIs over 10 seeds.\n    \n3. Real-robot demo. A sim-to-real reaching task on a Franka Panda shows measurable performance gains and practical feasibility at 10 Hz command rate with a 1 kHz PD loop.\n    \n4. Clarity and presentation: The paper is overall crisp in exposition and well-organized.\n    \n\n## Weaknesses\n\n1. No analysis of the computational costs. Due to the online planning stage, authors should include the computational costs, inference time, and the hardware used.\n    \n2. Lack of a domain randomization baseline. The authors mention that DR is an orthogonal direction, but it is unclear what the benefits of MELRO are compared to DR. A better explanation would help readers better contextualize the proposed method.\n    \n3. Figure 9 is unclear: which is MELRO and which is Max Ent?\n    \n4. (minor) In some tasks, base policy or rollout alone can outperform MELRO (e.g., disabled joint walker walk, torso length walker for 0.5) from Fig. 4. Why could this happen?\n\n\nThis said, I believe the paper is an interesting read and of interest to the TMLR community, and I would recommend it for acceptance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I believe the authors did a great job supporting the paper claims by accurate, convincing and clear evidence.", "audience_yn": "Yes", "audience_explanation": "RL and robotics are hot topics and the authors present an interesting work. This work is of interest for the TMLR community.", "requested_changes": "1. Please include an analysis of the computational costs as per W1 (critical)\n2. Either include an experiment or expand on the differences in performance of domain randomization and MELRO as per W2 (critical)\n3. Figure 9 fix as per W3 (minor)\n4. Explanation of why MELRO sometimes underperforms (minor)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0zRvEn7k4X", "reviewer": "Reviewer_fcZj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to modify planning methods (à la Model Predictive Control) to include maximum entropy optimizing _during planning_, not just in the proposal policy itself (as MaxEntRL would do). Extending the intuition of MaxEntRL, this pushes agents to not just learn \"myopically\" robust policies, but to also robustly plan. The authors argue that this robustness is an important ingredient in _sim to real_ transfer, and demonstrate improved performance on a variety of perturbed tasks, and an actual sim to real transfer.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors claim that MELRO improves generalization performance in simulation and transfer, which are both backed by evidence. The authors also perform experiments to better understand where the performance increase comes from.", "audience_yn": "Yes", "audience_explanation": "The sim2real problem is an important one, and this is an interesting addition to the field of planning methods that combines ingredients known to work in other domains.", "requested_changes": "I think the main thing I find lacking in the paper is the ablation & introspection part. Section 4.1.2 could easily be 2 pages & have way more information than what's in Figures 5 (the only non-reward-based plot of the paper!). There's a lot to unpack in maximum entropy methods and the effect they have on action choices, but none of that is explored here, and so we're left with only hints of why planning maxent matters.\n\nRelatedly, the appendix seems to list sets of possible hyperparameters, but where are the results of these? As is, I'm not sure there's any explanation of the impact/sensitivity of those values in the paper, in particular the impact of the entropy parameters seems important to report.\n\nI may be misunderstanding the setup here, but IIUC the Rollout baseline has no MaxEnt at all. Isn't a missing baseline one where the policy  is _trained_ with the MELRO setup, but where the entropy term is turned off at inference? This would further isolate the impact of entropy in planning.", "broader_impact_concerns": "None", "additional_comments": "Minor: Maybe it's just my perspective on this, but I'm surprised to not see MPC mentioned by name. Is it no longer called that?", "extra_scores": {}, "extra_text": {}}, {"review_id": "EWeaDbCPdl", "reviewer": "Reviewer_uZk7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a sim-to-real method called MELRO complementary to the domain randomization methods. MELRO is claimed to be effective when domain randomization methods are too costly to exhaust all the parameters MELRO incorporates rollout online planning to the maximum entropy framework, replacing the robustness against environments brought by domain randomization with 1) maximum entropy robustness and 2) online replanning for residual uncertainties. On simulation and a real Franka robot arm the authors tested the utilities of MELRO.", "claims_and_evidence_yn": "No", "claims_explanation": "The main motivation of the paper lies in alleviating the burden of domain randomization yet keeping the robustness against environmental shift by the proposed MERLO. However, it is quite unclear to me that how or how much that is possible. The authors refer to the maximum entropy framework as \"inherently robust\". But how robust is that compared to domain randomization, especially when compared to DR that explicitly treats the shift as an instance of the env distribution? Does the MaxEnt framework, when formulating the entropy in Eq. (2), incorporates any idea related to such env shift? If the entropy itself contains no other information than state/actions, then why would it have the potential for sim2real and be a good alternative to DR? These concerns are not touched upon in the paper. While the online part of MERLO could be helpful by generating imaginary rollouts according to the world model, its answer to the aforementioned question is still unknown.\n\nEmpirically, the current experiments lack also evidence on 1) how MERLO is efficient and performing when compared to existing DR methods 2) the role played by MaxEnt and rollout alone.  While Figure 4 shows an ablation study, its scope is limited to the MERLO components, and in some cases MERLO shows almost the same performance as the MaxEnt base policy.", "audience_yn": "Yes", "audience_explanation": "While I believe the paper is not ready for publication, I think the idea of \"approximating the robustness of domain randomization for better sim2real with a computationally light method\" is interesting and useful. The method proposed by the authors is potentially one of them, but I do think significant expansion is required for the proposed MERLO to be seriously considered effective.", "requested_changes": "Currently, the paper has only 9 pages. This is significantly shorter than even the regular paper limit (12 pages) of TMLR. I believe this room can be utilized to do at least the following:\n- more explanations on why the maximum entropy framework would work in the offline setting, why it has the desired robustness to help a smooth sim2real transfer. If the vanilla MaxEnt does not work, then what extensions are needed? For example, trajectory-wise or even env-wise entropy?\n- more experimenst comparing against DR methods. It is necessary to show how much performance loss and computational loss are incurred\n- more explanations on the framework itself: there are many components in MERLO like world model, recurrent net, etc. Why these components are chosen? What would happen if any of them is replaced?", "broader_impact_concerns": "not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)