Strength Through Diversity: Robust Behavior Learning via Mixture Policies
Authors: Tim Seyde
Wilko Schwarting
Igor Gilitschenski
Markus Wulfmeier
Daniela Rus
Venue: CoRL2021 Poster (2021)
Keywords: Learning Control
Hierarchical Optimization
Sample Efficiency

Abstract: Efficiency in robot learning is highly dependent on hyperparameters. Robot morphology and task structure differ widely and finding the optimal setting typically requires sequential or parallel repetition of experiments, strongly increasing the interaction count. We propose a training method that only relies on a single trial by enabling agents to select and combine controller designs conditioned on the task. Our Hyperparameter Mixture Policies (HMPs) feature diverse sub-policies that vary in distribution types and parameterization, reducing the impact of design choices and unlocking synergies between low-level components. We demonstrate strong performance on continuous control tasks, including a simulated ANYmal robot, showing that HMPs yield robust, data-efficient learning.

Strength Through Diversity:
Robust Behavior Learning via Mixture Policies
Tim Seyde'
Wilko Schwarting MIT CSAIL
Igor Gilitschenski University of Toronto
Markus Wulfmeier" DeepMind
MIT CSAIL
Daniela Rus* MIT CSAIL
Abstract: Efficiency in robot learning is highly dependent on hyperparameters. Robot morphology and task structure differ widely and finding the optimal ting typically requires sequential or parallel repetition of experiments, strongly increasing the interaction count. We propose a training method that only relies on a single trial by enabling agents to select and combine controller designs condi- tioned the task. Our Hyperparameter Mixture Policies (HMPs) feature diverse sub-policies that vary in distribution types and parameterization, reducing the im- pact of design choices and unlocking synergies between low-level components. We demonstrate strong performance on continuous control tasks, including a sim- ulated ANYmal robot, showing that HMPs yield robust, data-efficient learning."
Keywords: Learning Control, Hierarchical Optimization, Sample Efficiency
Introduction
m(als)
Real-world autonomous robots require versatile con- trollers that continuously adapt behavior to chang- ing environmental conditions and task specifications. Reinforcement learning (RL) has driven success in modeling complex control strategies in games [1, 2]. simulated robotics [3] and real-world systems [4, 5]. However, efficient learning often conditioned on good parameter selection and may require tuning for each task or domain [6]. Common approaches to hyperparameter optimization leverage parallel or se- quential evaluation strategies. While parallel strate- gies are computationally costly and need improve on random search [7], sequential strategies [8, 9] are time-intensive with hybrid approaches trading-off one for the other [10]. Gradient-based methods en- able online adaptation at the cost of requiring differ- entiable objectives. Selecting parameters efficiently is essential to learning capable robot controllers.
(als. 1)
n (a(a)s)2)
300
ReL
400
200
10
Hyperparameter Mixture Policy (HMP) with diverse low-level distributions. The agent can seamlessly adapt its policy structure to the presented tasks by modulat- ing component activations in single trial.
Figure 1:
The nature of motion planning problems further dictates suitable controller designs for learning. Continuous control can represent intricate transitions in state-action space to yield highly-optimized behaviors through local exploration or generate smooth references for low-level tracking con- troller. Discrete control can leverage reduced resolution for coarse exploration and readily encodes
"Correpondencee to tseyde@mit.ed: edu. "Equal advising. 2 Plecase find additional details at htttps:/// //sttes//sssss/// pog.g//ecco///eeee/////ettttt//.
5th Conference on Robot Learning (CoRL. 2021), London, UK.

---

bang-bang responses to switching dynamics. Optimal controller selection does not have to be uni- modal and can vary with different phases task and stages of the learning process. It may then be advantageous to provide agents with a diverse set of controllers that differ in their parameteriza- tion. This enables agents to select designs suitable for the presented task and unlock compositional synergies. In order to support this type of compositionality, we extend the perspective of previous work on hierarchical reinforcement learning [11].
In this paper, we propose a hierarchical policy over a diverse mixture of low-level controllers to improve robustness and reduce the necessity for parameter tuning and related data requirements. The low-level controllers are diverse in architecture, hyperparameters, and distribution characteristics to provide the robot with a rich set of controller designs. Our approach then enables learning robots:
to optimize a set of diverse controller designs concurrently for increased data-efficiency, to self-select suitable controllers conditioned on the task for reduced human parameter tuning,
compose behaviors from multiple controller designs for exploitation of emergent synergies.
We evaluate performance on a variety of torque-control tasks from the DeepMind Control Suite [12]. Additionally, we investigate learning of PD-control targets for the ANYmal robot in RaiSim [13], which was the foundation for Sim2Real transfer in Lee et al. [4]. Throughout, we demonstrate that enabling agents to operate over a diverse set of controllers guards against individual failure modes and unlocks synergies between different controller designs. While the high-level selector introduces its own hyperparameters, demonstrate its robustness to loss of state information by modelling unconditional component selection and subjecting the selector to adversarial distractor components.
2 Related Work
The performance of deep RI. algorithms is strongly tied to hyperparameter choices [6. 14, 15]. Commonly, hyperparameter optimization performed multiple experiments via sets of agents or tasks [16]. Simple parallel strategies include expert selection grid-search [17, 18), and can be less efficient than random search [7]. Bayesian Optimization provides more structure at the cost of sequential evaluations [9, 19, 20). Evolutionary strategies [8, 21, 22] enable discontinuous optimization and can evolve parameters at different rates, but typically use sequential evaluation. Population-Basee -ased Training (PBT) [10] alternatively evolves parameter variations online in parallel. This requires populations of agents each with their own environment, leading to significant data and computational requirements. Our work is also related to neural architecture search [23]. In particular, similarities can be found to methods that render architecture search differential [24, 25], enabling direct optimization of the effectively architecture during a single experiment.
Optimizing hyperparameters during the lifetime of single agent reduces these requirements. Gradient-based optimization ([26, 27]) yields online adaptation when the objective is differentiable in the parameters. HOOF [28] extends towards non-differential aspects and enables gradient- free off-policy training with hyperparameter schedules. However, the optimized parameters need di- rectly affect the policy update, precluding e.g. application to architecture search. Related methods have been used to learn architecture schedules [29]. With HMPs, we consider a single agent lifetime to reduce data and computational requirements. We consider policies that vary in their parameters, architecture, and distributional characteristic. Formulating a mixture over these diverse components allows us to evolve multiple controllers in parallel while the agent modulates activation based on pected performance. The approach further enables combining controllers with different parameters.
The problem of hyperparameter choice can further be framed as a contextual bandit problem [30]. In comparison to mixture agents, this formulation does not natively share data across policies with different parameters. Mixtures of specialized motion controllers naturally arise in form of motor primitives [31, 32]. The application of dynamical movement primitives [33] to robot control via a mixture library has been shown in [34]. Mixture distributions have a long-standing history to model diverse and multi-modal data [35]. In RL, they have been applied as a mixture of linear Gaussians in which component specialization is achieved by introducing entropy bounds [36] and provide

---

an inference perspective to the options framework which models an agent via the separation into high-level controller and low-level behaviours [11, 37, 38). Similarly, quality diversity algorithms evolve repertoires of diverse low-level skills which a high-level selector may act on [39, 40, 41]. We build the training of mixture policies via weighted maximum likelihood optimization, which has previously been used in connection with information asymmetry to generate diverse, compositional skills [42]. Diversity in mixture policies has been explored to strengthen skill discovery by explicitly optimizing for diversity-related objectives [43, 44, 45, 46] with fixed architectures and single set of hyperparameters.
3 Preliminaries
We formulate controller optimization Markov Decision Process (MDP) defined by the tuple R, R) where S and A denote the state and action space, respectively, T : S x A + S represents the transition distribution, R S x A + R the reward mapping, and y E [0, the discount factor. We define 8, and a to be the state and action at time t, respectively. Let (a(a))) denote a policy distribution with parameters 9 and define the discounted infinite horizon return G, 7p= R ag)), where 8t+1 T(s++1 |st, de) and a; (o(ae Our goal is to learn the optimal policy maximizing G. under unknown dynamics and reward mappings. This is typically done by modeling To (a (8)) as a Gaussian distribution with a neural network predicting the mean and variance from 8t. In this work, consider more diverse class of policy distributions.
Hyperparameter Mixture Policies
We propose Hyperparameter Mixture Poli- cies (HMP) to train a hierarchical policy over diverse low-level controllers with dis- tinct hyperparameters and distribution char- acteristics. The resulting mixture is given by
Algorithm 1: Hyperparameter Mixture Policies
Initialize:
Nauphaga: (target) update steps, Nº: action samples per state, KL bounds
while k Nuup do
for le + 1 to Narger do Sample hatch of trajectories from memory B, N₂ actions from Sa, to estimate expectations Compute gradients over batch for me, 1j. Ap. Qu ((s.a,)
Ta(als) Σ
H 8B (ils)m (als,
(1)
8u + Vo EEND E
exp
with H (i]s) and T (aaass i) as the weight and probability density of component Thus, n is high-level selector and n sub-policy from a diverse set of controller designs. The agent then self-selects the most suitable controller for individual phases of task or stages of the learning process. This enables robust adaptation to a broad range of motion planning problems while reduc- ing the necessity of manual parameter tuning and sequential experiment design.
log TO (a, 1s)
+E, Ap (0. EEWD KL |TTE )]
(8)
6, V VTg(T) = -,, + ESnD log N.
Σ exp ((s,a;)
(5)
ôxp <- VA (4 EanD [L [TOO )]
(8)
ōq Ve (Qe.)))(Q0ss a) Qzat)²
(10)
Apply gradients update Tox 7, Ap. Qo Update target networks mi no. Q' Qo
4.1 Policy Improvement
We use an actor-critic algorithm where policy improvement relies on two stages as [47]. First, non-paun parameerettic policy q(a)s) is optimized samples from the state-action value function Q" under the constraint of remaining close in expectation to the current parametric policy æy. The parametric policy then updated to better approximate the non- non-parametric target. By performing the actual policy improvement with non-parametrie policy, we bypass the need for gradient estimation via likelihood ratio [48] reparametrization trick [49]. In addition, this perspective enables the optimisation of mixture distributions in reinforcement learning without continuous relaxation [50].
3

---

Step 1 Fitting the Non-parametric Policy
As we not have access to the ground-truth state-action value function Q we employ a learned approximation Qd. parameterized by é, and formulate the objective iteration & as
max J (g) Bg [Q
(2)
9
s.t. R KI [I(((a|)))] (als)) s,
(3)
where e defines a trust-region around the current parametric policy, Tea- This can solved to provide closed-form relation in terms of the current parametric policy
qu (als) noz (als) exp (u(s.a)
(4)
where y) is computed by minimizing the dual function
(nn) = 77 + tnEaND log Toe (als) exp
Qa(s.a)
da
(5)
Step 2 Updating the Parametric Policy
The parametric policy πe optimized to approximate the non- pur-aarmetticc policy by minimizing their KI. divergence as
min (( = LEDD [K)(g((l())|]=((()])))
(6)
Plugging in (4) and introducing additional KL constraint enable generalization beyond the sample distribution yields
FiTL exp (o(s.a) log (a(
max (() = EyDD
0 s.t. ED [KL( (als)][Ta
The update proceeds via a Lagrangian relaxation of the KL constraint enabling the application of gradient-based optimization. We further decouple the KI. constraint and define independent con- straints for each distributional parameter p in both the high-level selector and low-level control poli- cies [47, 42]. We define these separately per diverse component to accommodate differences in the distributional parameter dynamics between low-level controllers (in comparison to e.g. Wulfmeier et al. [42]). To enable training of diverse distributions with different constraints, we enforce the KL constraints per component. Additional changes enable training of mixed continuous discrete policies are described in Section 4.3. We obtain updated parameters 0x+1 solution of
max min L(O, Ap) = E.ND... [exp( (xxp(ellaa)) Qo(taa)) log ma(ass) A,50
(8)
Σ Ep BeND (L((T((a)s) ITa (als))]
where we over decoupled components, each only varying along its respective parameter p. We also introduce component specific Lagrangian multipliers Ao and KL bounds Ep- A two component mixture of e.g. a Categorical (o1) and Gaussian (12, Σz) would then yield p = (CHLL a1 (22, Σz)-
4.2 Policy Evaluation
In order to stabilize off-policy learning of the state-action value function Qs we leverage the Retrace algorithm [51]. Here, we truncate the infinite series after steps and bootstrap from the target state-action value network, with details provided in Appendix D. To increase efficiency, further consider two-step transitions by squashing consecutive timesteps before adding them to memory.

---

Swwing-pp
hight
Contact
Rhooe
Cartpoot
Nomaal Cataponcal
Narmal Contsorica
Trajectory
Trajectory
Trajectory
Chrerton
Thesestep
Timrattpp
1-SNE over Componentss
Figure 2: Component specialization within an HMP consisting of a narrow Gaussian (aini = 0.3) and coarse Categorical (7hin 2). On Cartpole, bang-bang control enables fast swing-up and fine-grained control stabilizes the upright. On Cheetah, fine-grained control coordinates the contact phase and bang-bang control retracts the limbs during flight phase. Providing an agent with diverse low-level controllers can unlock synergistic specialization that consistent across states (t-SNE).
4.3 Combining Continuous and Discrete Distributions
Continuous and discrete policies do not share the same support. Furthermore, actions are subject to numerical cut-off errors. In practice this can result in the action-likelhhood of discrete policies being 0 most of the time. To facilitate training with diverse mixtures, we approximate discrete components by piece-wise constant sneudo-densitic for backpropagation. Thus, mapped into the support for probability computation. For query action a and discrete mixture component i with finite support C; we obtain the corresponding piece- constant pseudo-density
samples are
Th (als, = Σ p:(āls) (a)a
REC;
where (ū]s)) is the probability of ä in the original discrete distribution, I is the indicator function, and (ā) is a ball radius around à (we use 5 = 0.1 here). This improves sharing of gradient information between continuous and discrete policies and enables discrete components to train on samples generated by continuous components accelerate learning.
Experiments
We benchmark the performance of HMPs continuous control in the DeepMind Control Suite [12], learning PD-control for ANYmal in RaiSim [52, 13], and manipulation tasks in Metaworld [53]. We further compare to the gradient- free hyperparameter optimizer HOOF [28] in OpenAl Gym [54]. To better isolate the effects of diversity, we consider standard Gaussian policy and evaluate single pa- rameter variations together with their composition into diverse HMPs. We furthermore investigate HMPs consisting of randomly sampled components and show that strong performance can be recov- ered. Overall, we find diversity to enable robust learning across tasks and to guard against failure modes of individual components. Our figures visualize performance mean and standard deviation.
5.1 Qualitative Example
We provide an illustrative example component specialization within a diverse policy in Figure 2. The agent combines localized Gaussian (omm = 0.3) with a coarse Categorical (Mbin 2) policy. On a Cartpole swing-up up task, the agent leverages bang-bang control for swing-up and continuous control for stabilization. On Cheetah locomotion task, continuous control coordinates the intricate contact phase and bang-bang control quickly retracts the limbs during the flight phase. Applying 1-SNE dimensionality reduction yields consistent clustering across trajectories, indicating consistent component specialization. This aligns with human intuition and highlights the promise of composi- tion, further motivating HMPs and analysis of synergies in heterogeneous mixture policies.
5

---

Cartpole Swing-p up
Cheutah Bun
Finger Spin
Walcer Walk
Quadraped Pun
Atifmmal Run
30
611 Episdd (K)
03 Episode [K]
08
87 Episode [K]
31. 94
67
83
07
23
43
80
10 26 16 48
Episode
[K]
Episode
[K]
Episode
[K]
(2:9)
30.3:0.51
Figure 3: Combining continuous and discrete distributions to unlock synergies. Pairing a narrow Gaussian with a bang-bang controller yields strong performance, guarding against component failure (bang-bang on ANYmal, Gaussian on Finger) and improving on individual performance (HMP on Cartpole). Coarse control can drive exploration, fine-gu grained control can enable accurate tracking.
Finger Spin
Reacher
Quadruped
Humanola Walk
Raisim Anymal
1900
Eshoode
64
30
63
53
34
10
13
84
60
10
10
MKCD
N
Finger Spin
Reacher Hard
Vibieer VIOl
Quadruped Run
Humanold Vilk
Raldim Anymal
Epiepolt
7M
NKCD
RHIP
DDPG
Assembly
Bin Picking
Drawer Open
Door Open
Hammer
Place
Episnde Reeurr
200
5000
2000
15 11
25
18
205
10
2.0
13 26
15
35
20
10
Episade [K]
Edlsode [K]
Episode [K]
Episode [K]
Episode [K]
Episade [K]
Figure 4: Top- HMP consisting of a Gaussian (N), Kumaraswamy (K), Categorical (C) and Discrete
Gaussian (D) heads with individual components for reference. Bottom HMP instance compared
baselines RHPO and DDPG. Our HMP robust sub-policy failure (e.g. K on Humanoid) and yields strong performance especially the real-world inspired ANYmal and manipulation domains.
5.2 Heterogeneous Distributions
Compositional Solutions We further evaluate synergies arising from heterogeneous mixtures by combining a narrow Gaussian policy (a 0.3) with Categorical policy (with Tain E {2, 9))- Figure 3 indicates that performance of the Gaussian significantly improves in combination with a bang-bang policy (riin = 2) for torque-control (panels 1-5) (see also [55]). Conversely, the Gaussian guards against the failure mode induced by bang-hang control on ANYmal (panel 6). We can further replace the Gaussian with more fine-grainer Categorical (Nain 9) to reach comparable performance on the Control Suite tasks. However, the resulting mixture cannot compensate for bang- bang signals on AN as discrete control is not well-suited for position reference generation.
Diverse Distributions We broaden analysis of diverse distributions and consider a Gaussian (oini = 1.0), Kumaraswamy (Cmi = 1.0), Categorical (réin 5) and Discrete Gaussian (Main 5), as well as their combination into an HMP which we refer to as NKCD. Figure 4 highlights that the HMP is able to solve all tasks, guarding against individual component failure (e.g. on Quadruped) or premature convergence (e.g. D on Reacher). On ANYmal, the HMP leverages the Kumaraswamy policy to outperform the Gaussian policy, while the Kumaraswamy is suppressed on the Humanoid to reach strong performance. This underlines the robust performance that diverse HMPs provide by evaluating multiple controller designs jointly, reducing environment--peccfii tuning and guarding from component failure. We compare the NKCD HMP to the RHPO [42] and DDPG [56] agents. RHPO leverages homogencous mixture policy consisting of 4 MPO-type Gaussians (aiii = 1.0).

---

Figure 4 shows that the HMP and RHPO outperform DDPG on all tasks. The HMP performs com- petitively with RHPO throughout and outperforms RHPO on the real-world inspired ANYmal and manipulation domains. This underlines HMP's ability to enable data-efficient learning by training multiple policy designs in parallel and transferring problem-speciine controller selection to the agent.
Random components We consider sampling sub-policies with random hyperparameters. This includes randomizing distribution type, initialization, architecture and activations of each component. Figure provides perfor- mance statistics across 10 random instances of an HMP with 10 random components. We ob-
Cartoolle Swingup
Finger Spin
Lpisade Return
3000
xxx
Reacher Herd
Welker Walk
Fploods Return
serve that random selection yields performance competitive with the optimized RHPO agent. Beyond random selection, the engineer may strict the space of available sub-policies to in- ject structural priors into the learning process.
Fpisade [K]
Fplcodi [K]
Random HMP (10)
RHPO
Figure 5: Random sub sub-policy parameterizations.
Gradient-free optimization We compare the NKCD HMP to HOOF [28], which introduced method for gradient-free hyperparameter op- timization and evaluated on OpenAl Gym [54]. Their results are provided in Figure 6 for refer- ence. We note that our diverse mixture displays competitive performance on these benchmarks without any fine-tuning for Gym domains.
HultChestab-22
Hoppero22
Antovt
2d122
Timestop [H)
Timesliop
Timesilop [M]
Timostap [M]
MKCD (Cum)
HOOF (TNPG)
-HOFF (A2C)
Figure 6: HMP vs. HOOF [28] on OpenAl Gym.
5.3 Hyperparameter Variations
In the following, we evaluate performance of different hyperparameter combinations on continuous control tasks from the DeepMind Control Suite and locomotion the ANYmal robot in RaiSim. We consider a standard MPO parameterization as the baseline and analyze the impact of combining potentially sub-optimal parameter variations. To account for increased model capacity in mixture policies, we compare to RHPO-type homogeneous mixture with standard MPO parameters.
Initialization We vary the initial standard deviation of Gaussian policy heads as this can signifi- cantly impact the rate of convergence. The bounded action space of the agent is [-1.0, -11 We consider the initial values am = {0.3, 1.0, 3.0} with the standard literature value of = 1.0. Fig- ure 7 (row 1) indicates that the Control Suite tasks favor large variance to drive exploration. while generating position targets on ANYmal requires low variance to avoid instability of the PD con- troller and subsequent falling. Generally, we find that a diverse policy improves performance over the weaker component, yielding robust controller that prevent individual failure modes. This is evident for ANYmal Run, where the high variance policy fails but the diverse mixture succeeds.
Architecture We vary the layer structure with ক E 20), [200], [200, 200 and standard value oं = [200]. Figure 7 (row 2) indicates that performance is robust to architecture variations with increased capacity slightly improving performance. The heterogeneous mixture performs slightly better on the Control Suite tasks while the homogeneous mixture is slightly better ANYmal Run.
Activations We vary the policy activations with n, (ELU, Leaky ReLU, Tanh} and कं ELU. Figure 7 (row 3) shows that the heterogeneous mixture improves performance over both the homo- geneous mixture and the individual components on the Control Suite tasks. On ANYmal, combining Leaky ReLU and Tanh activations initially causes quick episode termination and delayed learning. Based on individual performance this is surprising and could indicate that ELU activations better suited for composition on this task. This is the only instance where observe reduced performance.
7

---

Cartpole Swingup
finger Spin
Quadruped Run
Eplcode Recur
8
30
X0
9.. (0.3:3.0)
12 o.. 0.3
dx
3.0
NW: 0,.
[1.0:1.0]
N: 6. 1.0
Episode Retum 3
8
NN. [[20] [200. 200]]
N: -200) [20]
N: R- [200.200] 200]
MN: [[200] [200]]
N: ft- [200]
Eptcode Recum 8
%
30
NN: - (LeatyRLU) Tanh]
n2 - LoakyRoLU
Tanb
nu
(ELU) ELU]
n, ELV
Eptode Return
MM Ou 130
5:20 3]
11:
N.
4ux
m 20
5
N: - 2C
3
NWN: 12c
4.20
41 N:
20
53. 02
Episode
02
33
Epicode
Figure 7: Performance of diverse mixture (solid red), a homogeneous mixture (solid green), and individual components. The homogeneous mixture uses MPO parameters, while the diverse mixture combines potentially sub-optimal parameter variations. Generally, the diverse mixture performs competitively while guarding against sub-policy failure modes (e.g. aii = 3.0 on ANYmal, top).
Learning Rate We vary the policy learning rates such that aLR E 2 x (10-5,100 1003)----0 and oiR - 2 10-4, Figure 7 (row 4) indicates that smaller learning rates reduce efficiency. However, pairing fast with slow head yields competitive performance, significantly improving over the individual slow component, and outperforms a nominal mixture on the Finger and Quadruped tasks.
6 Conclusion
Finding the right hyperparameters has a considerable impact on performance when enabling robots to learn complex behaviors through interaction and recent progress in machine learning can often be traced back to better hyperparameter settings [57]. A sub-optimal algorithm with thoughtfully tuned hyperparameters easily outperforms state-of-the-ard approach that has not been tuned sufficiently. Tuning requires both domain knowledge and experience with the underlying algorithm. Even then, it still incurs considerable computational cost that is particularly limiting when relying on world data. Our work proposes the use of diverse mixture policies to effectively mitigate this challenge. Moreover, we demonstrate the benefits of combining different distribution types and policy param- eterizations from perspective of compositionality in skill learning. Our Hyperparameter Mixture Policies (HMPs) induce diversity that can help in component specialization during different phases of a task, e.g. where certain movements require either coarse or fine-grained control. It has also the potential to accelerate the learning process, e.g. where more extreme actions enable faster explo- ration. The approach is easy to use and yields competitive performance across a range of common torque-control benchmark tasks, well as for generating PD-control targets within a high-fidelity simulation of the ANYmal quadruped, without extensive parameter tuning. While learning algo- rithms can always benefit from additional tuning, our approach increases robustness and helps to accelerate research in reinforcement learning for complex dynamic robots, in particular when there no access to extensive computational resources.
8

---

Acknowledgments
Tim Seyde, Wilko Schwarting, Igor Gilitschenski, and Daniela Rus supported in part by the Office of Naval Research (ONR) Grant 19001--8-1-2830 Qualcomm, and Toyota Research In- stitute (TRI). This article solely reflects the opinions and conclusions of its authors and not TRI, Toyota, or any other entity. We thank them for their support. The authors further would like to thank Lucas Liebenwein for assistance with cluster deployment, and acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources, We would also like to thank the reviewers and program chairs for their helpful feedback and suggestions for improvement.
References
[1] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782)350-335, 2019.
[2] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):600) 609, 2020.
[3] O. M. Andrychowicz, B. Baker, M. Chociej, Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[4] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47). 2020.
[5] T. Li, R. Calandra, D. Pathak, Y. Tian, F. Meier, and A. Rai. Planning in learned latent action spaces for generalizable legged locomotion. IEEE Robotics and Automation Letters, 6(2): 2682-2689, 2021.
[6] Henderson, R. Islam, P. Bachman, Pineau, D. Precup, and D. Meger. Deep reinforce- ment learning that matters. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[7] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
[8] P. A. Castillo, J. Merelo, A. Prieto, V. Rivas, and G. Romero. G-prop: Global optimization of multilayer perceptrons using gas. Neurocomputing, 35(1-4):149-1b) 2000.
[9] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3999. 2009.
[10] M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, al. Population based training of neural networks. arXiv preprint arXiv: 2017.
[11] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for tem- poral abstraction in reinforcement learning. Artificial intelligence, 12:111211 1999.
[12] Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, and N. Heess. dm_control: Software and tasks for continuous control. arXiv preprint arXiv:2006. 12983, 2020.
[13] J. Hwangbo, J. Lee, and M. Hutter. Per-contact iteration method for solving contact dynamics. IEEE Robotics and Automation Letters, (12):895-002. 2018. URL www raisii saisim. com.
[14] Henderson, J. Romoff, and Pineau. Where did my optimum go?: An empirical analysis of gradient descent optimization in policy gradient methods. arXiv preprint 110.0:1810 2018.

---

[15] B. Zhang, R. Rajan, L. Pineda, N. Lambert, A. Biedenkapp. K. Chua, F. Hutter, and R. Ca- landra, On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning. In Proceedings of the International Conference on Artificial Intelligence and Statis- fics (AISTATS), 2021.
[16] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In 25th annual conference on neural information processing systems (NIPS 2011), volume 24. Neural Information Processing Systems Foundation, 2011.
[17] V. Mnih, A. P. Badia, M. Mirza, A. Graves, Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[18] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and Abbeel. Benchmarking deep reinforce- ment learning for continuous control. In International conference machine learning, pages 1329 1338. PMLR, 2016.
[19] Snock, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. arXiv preprint 12X6::200.2244: 2012.
[20] T. Springenberg, A. Klein, Falkner, and Hutter. Bayesian optimization with robust bayesian neural networks. In Proceedings of the 30th International Conference Neural Information Processing Systems, pages 4141-4149, 2016.
[21] H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. Hierarchical representa- tions for efficient architecture search. arXiv preprint arXiv:1711.00433, 2017.
[22] Young, D. C. Rose, T. P. Karnowski, S.-H. Lim, and R. M. Patton. Optimizing deep learning hyper-parameters through evolutionary algorithm. In Proceedings of the Workshop on Machine Learning in High-PPrforrance Computing Environments, pages 1-5, 2015.
[23] Elsken, J. H. Metzen, F. Hutter, et al. Neural architecture search: A survey, J. Mach. Learn. Res., 20(55):1-21, 2019.
[24] Saxena and J. Verbeek. Convolutional neural fabrics. arXiv preprint urXiv:1606.02440 2016.
[25] H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. arXiv preprint 1rXiv:1806.00083 2018.
[26] Y. Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889 1900, 2000.
[27] Z. Xu, H. van Hasselt, and D. Silver. Meta-gradient reinforcement learning. arXiv preprint 2018.
[28] S. Paul, V. Kurin, and S. Whiteson. Fast efficient hyperparameter tuning for policy gradient methods. 2019.
[29] W. Czarnecki, S. Jayakumar, M. Jaderberg, L. Hasenelever, Y. W. Tch, N. Heess, S. Osindero, and R. Pascanu. Mix & match agent curricula for reinforcement learning. In International Conference on Machine Learning, pages 1087-1095. PMLR, 2018
[30] T. Schaul, D. Borsa, D. Ding, D. Szepesvari, G. Ostrovski, W. Dabney, and S. Osindero. Adapt- ing behaviour for learning progress. arXiv preprint arXiv: arXivv:111..00 19/2.06900, 2019.
[31] S. F. Giszter, F. A. Mussa-Ivaldi, and E. Bizzi. Convergent force fields organized in the frog's spinal cord Journal of neuroscience, 13(2):467-491, 1993.
[32] C. B. Hart and S. Giszter. A neural basis for motor primitives in the spinal cord. Journal of Neuroscience, 30(4): 1022::1222111100 1336, 2010.
10

---

[33] A. J. Ijspeert, J. Nakanishi, and S. Schaal. Movement imitation with nonlinear dynamical systems humanoid robots. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), volume pages 1398-1403. IEEE, 2002.
[34] K. Mülling, J. Kober, O. Kroemer, and J. Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3):263- 279, 2013.
[35] C. M. Bishop. Mixture density networks. 1994.
[36] C. Daniel, G. Neumann, and J. Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pages 273-281. PMIR, 2012.
[37] M. Smith, H. Hoof, and J. Pineau. An inference-based policy gradient method for learning options. In International Conference on Machine Learning, pages 4703-4712. PMLR, 2018.
[38] M. Wulfmeier, D. Rao, Hafner, Lampe, A. Abdolmaleki, T. Hertweck, M. Neunert, D. Tirumala, N. Siegel, N. Heess, al. Data-efficient hindsight off- off-policy option learning. arXiv preprint arXiv:2007.15588, 2020.
[39] J.BB. Mouret and Clune. Illuiinaiing search spaces by mapping elites. arXiv preprint 1XXv:1554.049099 2015.
[40] M. Duarte, J. Gomes, S. M. Oliveira, and A. L. Christensen. Evolution of repertoire-based control for robots with complex locomotor systems. IEEE Transactions on Evolutionary Com- putation, 22(2):314-233 2017.
[41] O. Nilsson and A. Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 866-875,2021
[42] M. Wulfmeier, A. Abdolmaleki, R. Hafner, J. Springenberg, M. Neunert, T. Hertweck, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Compositional transfer in hierarchical 1006611220 1128,, 2019.
reinforcement learning, arXiv preprint arXiv: 1ttl::999..122.00
[43] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint arXiv: 1802.06070 2018
[44] A. Goyal, S. Sodhani, J. Binas, X. B. Peng, S. Levine, and Y. Bengio. Reinforcement learning with competitive ensembles of information-coostraaned primitives. arXiv preprint 1906. /0667, 2019.
[45] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 2019.
[46] A. Sharma, M. Ahn, S. Levine, V. Kumar, K. Hausman, and S. Gu. Emergent real-world robotic skills via unsupervised off- off-policy reinforcement learning. arXiv preprint arXh::2004.1297 2020.
[47] A. Abdolmaleki, J. T. Springenberg. Degrave, S. Bohez, Y. Tassa, Belov, N. Heess, and M. Riedmiller. Relative entropy regularized policy iteration. arXiv preprint 182.2218/. 2018.
[48] R. Williams. Simple statistical grllient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-22) 1992.
[49] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv:
arXiv preprint
2013.
[50] A. C. Li, C. Florensa, I. Clavera, and P. Abbeel. Sub-policy adaptation for hierarchical rein- forcement learning, arXiv preprint
2019.

---

[51] R. Munos, T. Stepleton, A. Harutyunyan, and M. G. Bellemare. Safe and efficient off-policy reinforcement learning. arXiv preprint arXiv: Fr.xiv:666...m 1606.02647, 2016.
[52] M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V. Tsounis, J. Hwangbo, K. Bodie, P. Fankhauser, M. Bloesch, et al. Anymal-a highly mobile and dynamic quadrupedal robot. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 38-44. IEEE, 2016.
[53] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100. PMLR, 2020.
[54] G. Brockman, V. Cheung, L. Pettersson, Schneider, Schulman, J. Tang, and Zaremba. Openai gym. arXiv preprint 16X.v:1606.01544 2016.
[55] T. Seyde, I. Gilitschenski, W. Schwarting, B. Stellato, M. Riedmiller, M. Wulfmeier, and D. Rus. Is bang-bang control you need?: Solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems, 35, 2021.
[56] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
[57] T. Sivaprasad, F. Mai, T. Vogels, M. Jaggi, and F. Fleuret. Optimizer benchmarking needs to account for hyperparameter tuning. International Conference on Machine Learning, pages 9036-9045. PMLR, 2020.
[58] A. Mitnik. New properties of the kumaraswamy distribution. Communications in Statistics- Theory and Methods, 42(5):741-755, 2013.
[59] G. Dulac-Amold, N. Levine, D. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges of real-world reinforcement learning, arXiv preprint arXiv:2003. / 1881, 2020.
[60] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Ab- dolmalcki, A. Cassirer, F. Yang, K. Baumli, Henderson, Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 2020.
12


Reviews:
[{"review_id": "ofk0HIQ5dHt", "reviewer": "Reviewer_1NVm", "summary": "This paper proposes to learn a “meta-policy” that selects the best policy among a set of policies, depending on the state. The idea of the authors is that each sub-policy could have a different structure or be trained with different hyper-parameters: the meta-policy can therefore be viewed as an hyper-parameter selection algorithm (we do not have to select the hyper-parameters, the meta-policy does it). The authors test their idea with the Deep Mind Control Suite and an experiment with the ANYmal simulation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths\n- Interesting idea\n- A lot of different experiments are reported\n\n### Weaknesses\n- the sub-policies have to be chosen and might not be actually diverse; a simple baseline/variant would have been to use a set of sub-policies with random hyper-parameters (so that we do not have to choose, like in standard hyper-parameter tuning/parameter control algorithms)\n- many important parts of the algorithms are unclear to me; in particular, it is never clearly stated that/if the sub-policies are trained simultaneously\n- only 4 replicates (!), which is far from being enough to perform any statistics\n- no real robotics experiments and most of the experiments (except maybe ANYmal) are actually not close to a robot\n- the authors mention related work that does automatic parameter control/tuning, but there is no empirical comparison to this kind of technique\n- no statistical analysis (p-value, etc.): on many plots, the variances seem to overlap, which strongly suggest that most differences are not statistically significant\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this is a promising paper that describes a large amount of work. However, while each paragraph is well written, it found it difficult to follow the main story and the main ideas. \n\nIn addition, I think the paper would be much stronger if the sub-policies were automatically generated (e.g., random hyper-parameters, grid search-like, etc.) instead of hand-picked.\n\nI initially suggested a \"weak accept\" but I moved to \"weak reject\" when I finally found in the supplementary material that only 4 seeds were used. This is too little to conclude anything from the empirical comparisons, especially when many results look very similar.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- The results critically depends on the choice of the sub-policies, which are hand-picked.\n\n- Only 4 seeds were used, which is too little to perform any statistics and draw any conclusion in empirical work (most statisticians would say \"at least 20, although more is almost always better). First, the number of seeds should be in the caption of each result figures (so that we have the elements to interpret the figures). Second, are the figures representing the mean/standard deviation? mean/standard error? median/95% CI? This should also be in the caption.\n\n\n- The authors argue that their approach is faster/more data-efficient than alternatives (e.g., grid search); is it because all the competing sub-policies use the same samples? it is never clearly stated that/if all the sub-policies are trained together with the same samples. \n\n- The number of samples might be lower than learning each sub-policy independently, but all the rest of the computation (e.g., activating the networks) is the same as learning each of them independently; this should at least be discussed. \n\n- How is 4.1 different from the ref. 44 and from any other actor-critic algorithm? what is specific to the hierarchical nature of the problem here? The authors should highlight the critical novelties/changes here.\n\n### Other issues\n- The experiments on architecture variations (section 5.3, figure 7b) are probably not statistically significant (hard to say since the authors do not perform any statistical analysis). Please remove the claim that the proposed method is “slightly better” or provide some p-values.\n- There is an important literature about “quality diversity” that aims at finding automatically a diverse set of policies, which seems to be very relevant for this paper (but overlooked by the authors). For instance, check:\n  * Duarte M, Gomes J, Oliveira SM, Christensen AL. Evolution of repertoire-based control for robots with complex locomotor systems. IEEE Transactions on Evolutionary Computation. 2017 Jun 30;22(2):314-28.\n   * Nilsson O, Cully A. Policy gradient assisted MAP-Elites. Proc. of GECCO 2021. \n"}}, {"review_id": "mK_VVWLHoWC", "reviewer": "Reviewer_ADtN", "summary": "This works aims to combat the need for extensive hyperparameter tuning by proposing a method that can utilize a single trial to combine different control designs conditional on the task. This reduces impact of hyperparameters and design decisions and shows more robustness on different simulated tasks. \n\nThe key contribution of this work seems to be reformulating the policy to be a mixture of experts style policy and then integrating this into the framework of Relative entropy regularized policy iteration\n\nThe key contributions of this work seem to be: \n\n1. Introducing a mixture policy distribution where each component has different parameterization and potentially hyperparameters. \n2. Modifying the KL regularized PI framework to add in the mixture policy. \n3. Enabling optimization of mixed continuous and discrete distributions by introducing a pseudo density function. \n4. Separating the constraints in KL regularized PI such that there is a separate constraint and corresponding lagrange multipliers per constraint. \n\nThere is a detailed empirical analysis combining different components and noting how the combination does better than individual pieces. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The approach seems simple\n2. It is nice to be able to reuse data across multiple hyperparameters and architectures for controllers, especially very heterogeneous controller modes. \n3. It is interesting that different controllers specialize to different parts of a trajectory. \n4. The range of empirical results on actually trying different hyperparameters is a plus\n\nWeaknesses: \n1. Limited robotics experiments, results and discussion. \n2. The improvement on the domains is somewhat marginal, especially over RHPO. \n3. The figures and results are somewhat hard to parse without better signposting and labeling. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall I think the paper has an interesting premise, but I am recommending for a weak reject because of the results not being very much better than the baselines in many domains, a lack of robotics results and not a very clear novel contribution over prior work. I think a combination of multiple of these factors makes me recommend a weak rejection, but I am willing to reconsider given more experiments across other domains showing bigger improvements, significantly more clarity in the writing or more discussion or presentation of robotics results. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The first and most glaring issue seems to be that this paper is entirely in simulation, with very little robotics. There is one domain which is the ANYmal robotics, but the rest is just standard mujoco tasks. This feels a bit underwhelming for a robotics conference, especially when the ANYmal task is just PD tracking. \n\nIn the introduction, the words “synergy” is used excessively and is not adding much value. The overall introduction is a bit too vague right now to add much value on what is actually happening, would correct this. \n\nThe paper describes the following phenomenon often: “This enables robust adaptation to a broad range of motion planning problems while reducing the necessity of manual parameter tuning and sequential experiment design” -> why is this?? I don’t see the clear reason why training a number of different models together makes this any  more robust? Could this be expanded more clearly? I’m assuming that it’s because multiple different hyperparameter models can be trained using the same data simultaneously and the best model is chosen automatically, but this could be spelled out a bit more for the reader. \n\nOverall, the presentation of the paper makes it very hard to distinguish what is novel and introduced in this paper versus just adopted from past papers. Now it is perfectly ok to take a lot from past papers, it just needs to be made more clear in Section 4, and the key contributions need to be highlighted. \n\nI think the methods section would also do well from a further discussion on why this framework helps with robustness and hyperparameter selections and such. \n\nIt’s also not clear why this objective would yield to actual specialization and not just collapse like many mixture of experts frameworks have. Can this be elaborated on in the paper?\n\nAlso a discussion on how to pick the different components would be very useful, currently that whole thing is left in the air. Also are the components trained with different hyperparameters?\n\nIt seems like the actual improvement in Fig 4 is quite marginal besides in the ANYMal domain. Can other domains be presented where this makes more of a difference? There is currently only one datapointn that this actually matters. \n\nFor the robustness plots, I think Fig 5 is actually a bit misleading. On closer inspection I can see that red is better than green on the quadruped, but the overall delta seems quite small and I was originally looking at just the error bars. I think more signposting should be done to indicate what to look for in the figures rather than “A is better than B”. \n\nI think the comparisons against HOOF are a bit moot because the base RL algorithm is quite different. Can HOOF be combined with the same base RL algorithm here for Fig 6?\n\nWhat about the robustness to hyperaparameters of the whole learning procedure and not the individual components??\n\nCan Fig 7 be labeled with what each different line means properly? Which is the homogenous, heterogenous and the individual components?\n\nI think Fig 7 and 8 have pretty good analysis but on individual components at a time. What happens when multiple are varied together? Also the difference in performance is not really going between working and not working altogether, but just small differences in return. Are there other domains where these robustness benefits are highlighted more clearly. \n"}}, {"review_id": "7_VBNqOtH_N", "reviewer": "Reviewer_PWdD", "summary": "The manuscript is proposing a method for hierarchical RL using a diverse set of sub-policies and training a mixture of them. The main idea is modeling a parametric policy as a mixture of sub-policies and simultaneously learning a non-parametric policy and fitting the parametric policy to the non-parametric one. The algorithm is later evaluated on various learning to control tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The consideration of the role of hyper-parameters and the hierarchical nature of the problem is appreciated. It is intuitive and fits the final application of RL for robotics better.\n\nThe proposed method is intuitive and mathematically sensible.\n\nThe results are very promising.\n\nThe major issue for me is the lack of proper coverage of related work. The paper clearly fits into the area of hierarchical RL as well as RL with options framework. However, the authors ignore this literature completely in their discussion of related work and empirical study.\n\nThe experimental study is not explained in detail. After reading the paper a few times, I am still confused about how many random trials are performed and what are the shaded areas in the figures. The baselines are also not justified. For example, the authors do not discuss why did they choose DDPG?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The method and the paper is sensible; however, I need a better justification for empirical study as well as an extended review of the relevant literature to recommend the paper for publication.\n\n----\n\nI thank the authors for their clarifications. It partially answers my concerns but the major issues remain. Hence, I will not change my score", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Related work covering Hierarchical RL and RL with options.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "4VAkoPYvCE", "reviewer": "Reviewer_ghjV", "summary": "This paper presents a robust hyperparameter selection method that learns diverse sub-policies with various distribution types and parameterization. The authors use relative policy optimization [44] for the policy improvement step with modifications that use a sum of constraints consisting of decoupled components, each only varying along its respective parameter. The authors also use the piece-wise constant pseudo-density during policy evaluation that combines continuous action distribution and discrete action distribution. For empirical analysis, the authors evaluated the method on deepmind DM control tasks as well as OpenAI gym tasks comparing to prior gradient-free optimization approaches that showed that the method can achieve robust and sample-efficient learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an important problem in robot learning, which is the hyperparameter optimization issue. The proposed framework is neat since it can automatically train multiple sub-policies with different configurations in a hierarchical way, which is novel. The algorithm is simple yet effective in terms of the experimental results in simulated locomotion tasks. The paper is also well written and easy to understand.\n\nHowever, I do have a few concerns, which I will discuss as follows.\n\nFirst, the applicability of the proposed method seems a bit limited since the authors only consider different distribution types and parameterization of the policy in RL for hyperparameter optimization, which only covers a very small portion of the hyperparameters that matter in RL. The experimental results also only focus on simulated locomotion tasks, which seem not comprehensive. I think it would be important to extend the method to handle a wider range of hyperparameters in RL, e.g. learning rate, architecture design, etc. It is also pivotal to test the method on robotic manipulation tasks and also real-world tasks, which are more realistic for robot learning.\n\nMoreover, the algorithm appears a bit incremental, it seems to be almost a direct adaptation of the prior work [44] to the setting mixture of policies. The theoretical insight of such a direct adaptation is also missing.\n\n================================\n\nAfter reading the rebuttal, I appreciate that the authors conduct additional more realistic robotic experiments and also clarified my concerns on novelty of the paper. However, my concern on automatically selecting other hyperparameters still remains. The authors claim that their approach can automatically select hyperparameters and only show the use case for the distribution types. I think such a claim cannot be fully substantiated in the paper, therefore limiting the contribution of the paper. I will keep my score.\n\n===============================\n\nMore updates: thank you for pointing me to the additional experiments on randomly sampled hyperparameters! I've increased my score to a 6. Though I still have a few questions as follows. It seems that NKCD does not outperform RHPO and is even a bit worse in several domains. It is not also clear to me if RHPO is a reasonable baseline for varying other hyperparameters. The authors should perhaps compare to the oracle, which is DDPG tuned with each of the varying hyperparameter. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Given my two points on weaknesses of the proposed method discussed in the Strengths And Weaknesses section, I would vote for a weak reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1. Extend the method to handle more diverse hyperparameters in RL such as critic and policy learning rates, architecture choices etc.\n2. Perform evaluations on robotic manipulation tasks and also real-world tasks.\n3. Provide some theoretical insight of the method."}}]

Author rebuttal:

Meta-review: After the discussion and rebuttal from the authors providing additional evidence for their method, some of the reviewers updated their scores to accept the paper. The remaining points of criticism are mostly regarding the related work section, which can be fixed in the final version of the paper. I encourage the authors to carefully read the reviews and address the points raised by the reviewers in the final version of the paper.
Decision: Accept (Poster)