DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems
Authors: Pierre Schumacher
Daniel Haeufle
Dieter Büchler
Syn Schmitt
Georg Martius
Venue: ICLR 2023 notable top 25% (2023)
Keywords: reinforcement learning
musculoskeletal
correlated exploration

Abstract: Muscle-actuated organisms are capable of learning an unparalleled diversity of dexterous movements despite their vast amount of muscles. 
Reinforcement learning (RL) on large musculoskeletal models, however, has not been able to show similar performance.  
We conjecture that ineffective exploration in large overactuated action spaces is a key problem.
This is supported by the finding that common exploration noise strategies are inadequate in synthetic examples of overactuated systems. 
We identify differential extrinsic plasticity (DEP), a method from the domain of self-organization, as being able to induce state-space covering exploration within seconds of interaction. 
By integrating DEP into RL, we achieve fast learning of reaching and locomotion in musculoskeletal systems, outperforming current approaches in all considered tasks in sample efficiency and robustness.

Published as a conference paper at ICLR 2023
DEP-RL: EMBODIED EXPLORATION FOR REINFORCEMENT LEARNING IN OVERACTUATED AND MUSCULOSKELETAL S SYSTEMS
Pierre Schumacher 1.2 Daniel E.B. Haeufle Dieter Büchler'
Syn Schmitt Georg Martius'
Max Planck Institute for Intelligent Systems, Tübingen, Germany
-Hertie-Institute for Clinical Brain Research, Tübingen, Germany
IInttitute for Modelling and Simulation of Biomechanical Systems, Stuttgart, Germany
ABSTRACT
Muscle-actuated organisms are capable of learning an unparalleled diversity of dexterous movements despite their vast amount muscles. Reinforcement learn- ing (RL) on large musculoskeletal models, however, has not been able to show similar performance. We conjecture that ineffective exploration in large overactu- ated action spaces is key problem. This is supported by finding that common exploration noise strategies are inadequate in synthetic examples of overactuated systems. We identify differential extrinsic plasticity (DEP), method from the domain of ceff-organzzatio, as being able to induce state-space covering explo- ration within seconds of interaction. By integrating DEP into RL, we achieve fast learning of reaching and locomotion in musculoskeletal systems, outperforming current approaches all considered tasks in sample efficiency and robustness
500
5..00000 600
1200
32
2 20
R
é 2
Figure 1: We achieve robust control on series of overactuated environments. Left to right: torquearm, arm26, humanreacher, astrich-foraging ostrich-run, human-run, human-hop
INTRODUCTION
remarkable how biological organisms effectively learn achieve robust and adaptive behavior despite their largely overactuated setting- wwith many more muscles than degrees of freedom. As Reinforcement Learning (RL) is arguably a biological strategy (Niv) 2009), it could be a valuable tool to understand how such behavior can be achieved, however, the performance of current RL algorithms has been severely lacking so far (Song et al. 202T).
One pertinent issue since the conception of RL is how efficiently explore the state space (Sutton & Barto 2018). Techniques like e-greedy or zero-mean uncorrelated Gaussian noise have dominated most applications due their simplicity and effectiveness. While some work has focused on exploration based on temporally correlated noise (Uhlenbeck & Ornstein] 1930 Pinneri et al., 22220), learning tasks from scratch which require correlation across actions have seen much less attention. We therefore investigate different exploration noise paradigms systems with largely overactuated action spaces.
The problem aim to solve is the generation of motion through numerous redundant muscles. The natural antagonistic actuator arrangement requires a correlated stimulation of agonist and antagonist muscles avoid canceling of forces and to enable substantial motion. Additionally, torques generated by short muscle twitches are often not sufficient to induce adequate motions on the joint level due to
See http:://siessgooog...co//vvv//// for videos and code.

---

Published as a conference paper at ICLR 2023
chemical low-pass filter properties (Rockenfeller et al. 2013). Lastly, the sheer number of muscles in complex architectures (humans have more than 600 skeletal muscles) constitutes a combinatorial explosion unseen in most RL tasks. Altogether, these properties render sparse reward tasks extremely difficult and create local optima in weakly constrained tasks with dense rewards (Song et al. 2021
Consequently, many applications of RL to musculoskeletal systems have only been tractable under substantial simplifications. Most studies investigate now-dimensional systems (Tahami al. 2014 Crowder et al., 2 or simplify the control problem by only considering a few muscles (Joos et al. 2020; Fischer et al. 2021). Others, first extract muscle synergies (Diamond & Holland 2014)).
concept closely related motion primitives, or learn a torque-simulation mapping (Luo et al. 2021 before deploying RL methods. contrast to those works, we propose a novel method to learn control of high-dimm dines-iimesiiona and largely overactuated systems the muscle stimulation level. Most importantly, we avoid simplifications that reduce the effective number of actions or facilitate the learning problem, such as shaped reward functions or learning from demonstrations.
In this setting, we study selected exploration noise techniques and identify differential extrinsic plasticity (DEP) (Der & Martius 2013) to be capable of producing effective exploration for muscle- driven systems. While originally introduced in the domain of oof--rraanzztiin, we show that DEP creates strongly correlated stimulation patterns tuned to the particular embodiment of the system at hand. It is able to recruit muscle groups effecting large joint-space motions in only seconds of interaction and with minimal prior knowledge. In contrast to other approaches which employ explicit information about the particular muscle geometry at hand, e.g. knowledge about structural control layers or daxi-ne correlation matrices (Driess et al. 2018 WWlter et al. (202I), introduce prior information on which muscle length is contracted by which control signal in the form of identity matrix. We first empirically demonstrate DEP's properties in comparison popular exploration noise processes before we integrate it into RL. methods. The resulting DEP-RL controller able to outperform current approaches on unsolved reaching (Fischer et al. 2021 and running tasks Barbera et al. (02T) involving up to 120 muscles.
only
Contribution (1) We show that overactuated systems require noise correlated across actions for effective exploration. (2) We identify the DEP (Der Martius 2015) controller, known from the field of self-organizing behavior, to generate more effective exploration than other commonly used noise processes. This holds for a synthetic overactuated system and for muscle-drived control- our application area of interest. (3) We introduce repeatedly alternating between the RL policy and DEP within an episode as an efficient learning strategy. (4) We demonstrate that DEP-RL. is more robust in three locomotion tasks under out-of-distributtin (OOD) perturbations.
To our knowledge, we are the first to control the 7 degrees of freedom (DoF) human arm model (Saul et al., 2015) with RL on muscle stimulation evel-that is with 50 individually controllable muscle actuators. We also achieve the highest ever measured top speed on the simulated ostrich (Barbera et al. 202T) with 120 muscles using RL without reward shaping, curriculum learning, or expert demonstrations.
2 RELATED WORKS
Muscle control with RL Many works that apply RL to muscular control tasks investigate low- dimensional setups TTeck et al. 2018 Tahami et al., 2014 Crowder et al., 202T) or manually group muscles (Joos et al. 2020) to simplify learning. Fischer al. (2021) use the same 7 DoF arm as we do, but simplify control by directly specifying joint torques a E R7 and only add activation dynamics and motor noise. Most complex architectures are either controlled by trajectory optimization approaches (Al Borno et al. 2020) or make use of motion capture data (Lee et al. 2019). [arberra et al.) (202I) also only achieved a realistic gait with the use of demonstrations from real ostriches; learning from scratch resulted in a slow policy moving in small jumps. Some studies achieved motions on real muscular rorets al. 20188 Buchler et al. 2016). but were limited to simple morphologies and small numbers of muscles.
Neur challenges Multiple challenges on musculoskeletal control KKdziński et al. 2018 Kidziński et al., 2019: Song et al. 2021) using OpenSim (Delp et al., 2007) have been held. The top-ten submissions from Kidziñski et al. (2018) resorted to complex ensemble architectures and made use of parameter- or OU-noise. High-scoring solutions in (Kidzıñski et al., 2019) were commonly using explicit reward shaping, demonstrations, or complex training currieula with selected checkpoints,
2

---

Published as a conference paper at ICLR 2023
all of which required extensive hand-crafting. In contrast, our RL agent uses standard two-layer architecture, no curriculum, no reward shaping, and no demonstrations.
Large action spaces Some studies (Farquhar et al.) 2020 Synnaeve et al. 2019) tackle large action spaces by growing them iteratively from smaller versions. This would, however, require a priori knowledge of which muscle groups correspond to each other-which DEP learns by itself. [Tavakoli] architecture that scales in principle to continuous action spaces, Again, as it is not clear which muscles should be grouped. the number of possible hypergraphs is intractable. Other works (Dulac- Aonold et al. 2016; Wang et al., 2016) deal with large virtual action spaces, but only for discrete actions. Finally, studies on action priors(Biza et al.) 2021 Singh
et al. (2021) present a hypergraph-
et al. 202T) learn initially from expert data to bootstrap policy learning. Our method scales to large continuous action spaces and does not require demonstrations.
Action space reduction with muscles Several works use architectures reducing the control dimen- sionality before deploying RL methods. [Luo et al. (202T) learn a muscle-coordination network that maps desired joint torques to muscle excitations to act on PD controllers. Jiang et al. (019)) establish correction network that imposes output limits on torque actuators, enabling them to reproduce muscle behavior. However, these methods require specific data collection strategies and prior knowledge on joint ranges, forces or desired behaviors, before applying RL. While they simplify learning and enable large-scale control, our approach is simpler in execution, does not require knowledge about the specific muscular architecture and works with simulators of varying detail, including elastic tendons. Some works use PCA or other techniques extract synergies and learn on them (Al Borno et al.) 2020; Zhao et al., (022)). but they either require human data or expert demonstrations. Our approach can readily be applied to large range of systems with only minimal tuning.
BACKGROUND
Reinforcement learning We consider discounted episodic Markov Decision Processes (MDP) M = (S, ... P. P.. 7), where A is the action space, S the state space, r S x A - IR is the reward function, p(s' (s.a) a) the transition probability, Po(s) the initial state distribution and y the discount factor. The objective is to learn policy {a]ss) that maximizes the expected discounted return J = Ex Σ. 7'rr, where r are observed when rolling out n. For goal-reaching tasks, we define a goal space G with r(8, a, being 0 when reached the goal g and - everywhere else. The policy is then also conditioned on the goal (a)s, g) following Schaul et al. 20155).
Musele modeling In all our MuJoCo experiments, we use the MuJoCo internal muscle model, which approximates musele characteristics in a simplified way that is computationally efficient but still reproduces many significant muscle properties. One limitation consists in the non-elasticity of the tendon. This reduces computation time immensely. but also renders the task fully observable, as muscle length feedback now uniquely maps to the joint configuration. Experiments with a planar humanoid were conducted in HyFyDy (Geijtenbeek] [2019] 2019)20211.. fast biomechanics simulator that contains elastic tendons and achieves similar accuracy to the widely used OpenSim software. See Suppl. A.4 for details on the MuJoCo muscle model.
Limitations of uncorrelated noise in (vastly) overactuated systems The defining property of an overactuated system is that the number of actuators exceeds the number of degrees of freedom. Let us consider a hypothetical I DoF system, where we replace the single actuator with maximum force F by actuators, each contributing maximally with force FM /n. The force for a particular actuator then F, (FM) /n) fu. with f. E [-1,1], while the total force is F = Σn F; Σ. Ps/n. We detine F; = FM f; to make the dependence on n clear. What happens if we apply random noise to such a system? The variance of the resulting torque is:
Var
Var P
(1:)+1
1 Σ Cov(F.
For i.i.d. noise with fixed variance Var(F:) (f:) with 1/n and the second term is zero. Thus, for large n the effective variance will approach zero. The logical solution would be to increase the variance of each actuator, but as realistic actuator can output an arbitrarily large force, the maximum achievable variance is bounded. Clearly, a vanishing effective variance cannot result in adequate exploration. For correlated noise, the second term in
0², typically used in RL, the first term decreases
3

---

Published as a conference paper at ICLR 2023
Eq. has n² - n terms and decays with 1 - 1/n, it can avoid vanishing and might be used to increase the effective variance.
4 METHODS
4.1 DIFFERENTIAL EXTRINSIC PLASTICITY (DEP)
Hebbian learning (HI.) rules are widely adopted in neuroscience. They change the strength of connection between neurons proportional to their mutual activity. However, when applied in a control network that connects sensors to actuators, all activity is generated by the network itself and thus the effect of the environment is largely ignored. In contrast, differential extrinsic plasticity (DEP), learning rule capable self-organizing complex systems and generating coherent behaviors (Der & Martius 2015), takes the environment into the loop. Consider controller:
ag = tanh(1 Cs2 + he),
with the action a, E IR" the state E IR" learned control matrix C' E R=*x,, time-dependent bias h, E I2" and an amplification constant k E R. The DEP state is only constituted sensors related to the actuators, in contrast to the more general RL state. We only consider the initialization Cij- = To be able to react to the enviromment, an update rule is required for the control matrix C. A Hebbian rule proposes Cg x ai, 8j,t, while a differential Hebbian rule might be Ciy x ât,t á3,e. The differential rule changes C' according changes the sensors and actions, but there is no connection to the consequences of actions, only information of the current time step is used.
DEP proposes several changes: (1) There is a causal connection to the future time step Ć x 6 At where ((x;) = ar-Ar is an inverse prediction model and Al controls the time scale of learned behaviors. (2) A normalization scheme for C' adjusts relative magnitudes to retain strong activity in all regions of the state space C,₁ = /tl/([[Ca; + E), with e < (3) The time-dependent bias term h, x -at prevents overly strong actions from keeping the system at the joint limits with 5, = O. This learning rule becomes TÔ f( C: the first term drives changes while the second one is first-order decay term. The factor tunes the time scale of changing the controller parameters C. The normalization rescales C' at each time step. The velocities of any variable ã are approximated by é; It 2t-1-
The inverse prediction model relates observed changes in s to corresponding changes in actions that could have led to those changes. In the case where each actuator directly influences one sensor through the consequences of its action, the simple identity model can be used, similar to prior work (Der & Martius 2015 Pinneri & Martius 2018): f(e)) == šØ This yields the rule:
- = -
As a direct consequence, the update of C driven by the velocity correlation matrix of the current and the previous state. This means that velocity correlations between dtate-dimensions that happen across time, for instance, due to physical coupling, are amplified by strengthening of corresponding connections between states and actions in Eq. 2 The choice of f 'implicilly embeds knowledge of which sensor connected to which actuator and is assuming an approximately linear relationship. This knowledge is easily available for technical actuators. In muscle-drive systems, such a relationship holds for sensors measuring muscle length with f((+) == -8,, as muscles contract with increasing excitations. If there or a more complex model that links proprioceptive sensors to actuators.
sensors per actuator, f can either be rectangular matrix
Pinneri & Martius (2018) has shown that in oow-dimensional systems DEP can converge to different stationary behaviors. Which behavior reached highly sensitive to small perturbations. We observe that with high-dimensional and chaotic environments, the stochasticity injected by the randomization after episode resets is sufficient to prevent behavioral deprivation. In practice, we use DEP in combination with an RL policy, such that the interplay will force DEP out of specific limit cycles, similar to the interventions of the human operator in Martius et al. (001)). Intuitively. DEP is able to quickly coerce system into highly correlated motions by "chaining together what changes together". More details can be found in Suppl. B33 and in Der & Martius (2013). while Suppl. C.77 an application of DEP to a simple 1D-system elucidating the principal mechanism.
4

---

Published as a conference paper at ICLR 2023
4.2 INTEGRATING DEP AS EXPLORATION IN REINFORCEMENT LEARNING (DEP-RL)
Integrating the rapidly changing policy of DEP into RL algorithms requires some considerations. DEP is an independent network and follows its own gradient-free update rule, which cannot be easily integrated into madieett-bssed RL training. Summing actions of DEP and the RL policy, as with conventional exploration noise, leads in our experience to chaotic behavior, hindering learning (Suppl. C.2 Therefore, we consider different strategy: The DEP policy is taking over control completely for a certain time interval, similar to intra-episode exploration (Pislar et al., 2022): ;
RL policy π and exploration policy Eq. 2 are alternating in controlling the system. While we tested many different integrations (Suppl. B.3 and C.2) we here use the following components:
Intra-episode exploration DEP and RL alternate within each episode according to a stochastic switching procedure. After each policy sampling, DEP takes over with probability Powitch- Actions are then computed using DEP for a fixed time horizon HEE, before we alternate back to the policy. In this way, the policy may already approach goal before DEP creates unsupervised exploration.
Initial exploration As DEP is creating exploration that excites the system into various modes, we suggest running an unsupervised pre-training phase with exclusive DEP control. The data collected during this phase is used to pre-fill the buffer for bootstrapping the learning progress in off- off-policy RL.
5 EXPERIMENTS
We first conduct synthetic experiments that investigate the exploration capabilities of colored noise (Timmer& & König, 1995 et al. 2020) (see Suppl. and A.3). Omstein-UUlenbert (OU) noise (Uhlenbeck & Omsttin 1930), and DEP by measuring state-space coverage in torque- and muscle-variants of a synthetically overactuated planar arm. OU-noise, in particular, was a common choice in previous muscular control challenges (Song al., 2021 Kidziński et al. 2018 Kidziński iet al. 2019). Afterwards, DEP-RL is applied to the same setup to assure that the previous findings are relevant for learning. Finally, we apply DEP-RL to challenging reaching and locomotion tasks, among which humanreacher and ostrich-run have not been solved with adequate performance so far. Even though DEP-RL could be combined with any off- off-policy RL algorithm, we choose MPO (Abdolmaleki et al. 2018) 8 as our learning algorithm and will refer to the integration as DEP-MPO from now on. See Suppl. B.T for details. We use 10 random seeds for each experiment if not stated otherwise.
5.1 ENVIRONMENTS
Ostrich-run and -foraging are taken from [arbbera et al. (202I), while all of its variants and all other tasks, including specifications of state spaces and reward functions, are constructed by from existing geometrical models. We use SCONE (Geijtenbeek] 2019 202T) for the bipedal human tasks and MuJoCo (Ikkala & Hämäläinn] 2022 Todorov et al. (2012) for the arm-reaching tasks. See
Suppl. B.2 for details.
torquearm A 2-DoF planar arm that moves in 2D plane and is actuated by 2 torque generators. arm26 torquearm driven by 6 muscles. The agent has to reach random goals with sparse rewards. humanreacher A 7-DoF arm that moves in full 3D. It is actuated by 50 muscles. The agent has to reach goals that randomly appear in front of it at "Tace"-height The reward function is sparse. ostrich-run A bipedal ostrich needs to run as fast possible in horizontal line and is only provided a weakly-constrainigg reward form of the velocity of its center of mass, projected onto the x-axis. Only provided with this generic reward and without motion capture data, learning agent is prone to local optima. The bird possesses 120 individually controllable muscles and moves in full 3D.
astrich-foraging An ostrich neck and head actuated by 52 muscles need to reach randomly appearing goals with the beak. We changed the reward from the original environment to be sparse.
ostrich-stepdown Variant of ostrich-run which involves single step several meters from the start. The height is adjustable and the position of the step randomly varies with Ar ~ N (A, 0.2). The task successful the ostrich manages to run for 2 10 meters past the step. The reward is sparse. ostrich-slopetrotter The ostrich runs across series of half-sloped steps. Conventional steps would disadvantage gaits with small foot clearance, while the half-slope allows most gaits to move up the step without getting stuck. The task reward is the achieved distance, given at the end.
human-run An mu-sle planar human runs in a straight line as fast as possible. The reward is the COM-velocity. The model is the same as in the NeurlPS challenge (Kidziński et al. 2018).
5

---

Published as a conference paper at ICLR 2023
human-hop The reward equals if the human's COM-height exceeds 1.08 otherwise it is O. human-stepdown The human runs across slopes before encountering a step of varying height. human-hopstacle The human has jump across two slanted planes and large drop without falling.
We observed a critical bug in ostrich-run which we fixed for our experiments. This explains the differing results for the baseline from [Barbera et al. (2021) in Sec. 5.4 See Suppl. B.2 for details.
5.2 EXPLORATION WITH OVERACTUATED SYSTEMS
Primarily, are interested in efficient control of muscle-driven systems. These systems exhibit large degree of action redundancy in addition to a myriad of nonlinear characteristics. Thus, we create an artificial setup allowing us to study exploration overactuated systems in isolation.
White
OU
DEP
White
Pink
Red
OU
DEP
2 actions hand-y
0.6
0.27
state covenge
Stite eoveragaggg
0.00
0.4
0.15
600 actions
hand-y -
0.0
0.10
0.05
0.0
hand-x
hand-x
hand-x
A of actions
actions
(a) torquearm trajectories
(b) torquearm state coverage arm26 state coverage
Figure 2: Only DEP reaches adequate state-space coverage for all considered action spaces. (a) Hand trajectories are collected during 50 episodes of 1000 iterations (At 10 ms) of pure exploration with different noise strategies. We show hand trajectories for the original action space E IR.2 (top) and the expanded action space R800 (bottom). (b) Endeffector-space coverage for torquearm. (c) Endeffector- coverage for arm26.
Overactuated torque-driven system In the default case, the robot (Fig. torquearm) can be controlled by specifying desired joint torques E R2. We now artificially introduce action space multiplier n, such that the control input grows to 2n. To apply this new control vector to the original system, we average the actions into the original dimensionality, keeping maximal torques identical:
a 72 Σ
Oj+n(k-1):
(4)
where à, is the inflated action and k E [1,2] for our system. We emphasize that we not only have more actions but also capture important characteristics of an overactuated system, the redundancy. As predicted in Sec. B] we observe that redundant actuators decrease the effective exploration when using uncorrelated Gaussian (white) noise (compare 2 vs. 600 actions in Fig. 2 (a. b)). Also, for correlated pink and red colored noise or OU-noise, the exploration decays with an increasing number of actions. Only DEP covers the full endeffector space for all setups. See Suppl. A.l for details on the used coverage metric and Suppl. C.I for more visualizations.
Overactuated muscle-driven system Consider now system with individually controlled muscle actuators (Fig. T arm26). This architecture is already overactuated as multiple muscles are connected each joint, the two biarticular muscles (green) even span two joints at the same time. In addition. we apply Eq. 4 to create virtual redundant action spaces. In Figure (c), we see that most noise processes perform even worse than the previous example, even though the number of actions is identical. Only DEP again reaches full endeffector-ppate coverage for any investigated number of actions. These results suggest that the heterogeneous redundant structure and activation dynamics of muscle-actuators require exploration strategies correlated across time and across actions to induce meaningful endeffector-space coverage, which DEP can generate.
We emphasize that all noise strategies for experiments in Sec. 5.2 were individually optimized to produce maximum sample-entropy for each system and for each action space, while DEP was tuned only once maximize the joint-space entropy of the humanreacher environment. We additionally observe that all strategies consequently produce outputs close to the boundaries of the action space, known bang-bang control, maximizing the variance of the resulting joint torque (Sec.

---

Published as a conference paper at ICLR 2023
RL with muscles While the previous results demonstrate that the exploration issue is significant, is imperative to demonstrate the effect in a learning scenario. We therefore train an RL agent for differently sized action spaces and compare its performance to DEP-MPO. Figure S shows that the performance of the MPO agent strongly decreases with the number of available actions, while the DEP-MPO agent performs well even with 600 available actions, which is the approximate number of muscles in the human body. DEP-MPO strongly benefits from the improved exploration of DEP. We repeated the experiment with sparse rewards and HER in Suppl. C.6 the results are almost identical.
6 actions
30 actions
600 actions
arm26 virtual
1.0
success
0.6
0.5
МРO
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
30
500
steps
x x10"
steps
x x100
steps
x107
actions
Figure 3: DEP-MPO outperforms MPO in sparse point-reaching for arm26 with all virtual action spaces. Left: Learning performance decays with a growing number of actions for MPO, DEP-MPO largely unaffected. Right: Training-averagee success rates for different action spaces.
5.3 SPARSE REWARD TASKS WITH UP TO 52 ACTUATORS
We now apply our algorithm to realistic control scenarios with large numbers of muscles. As many sparse reward goal-reaching tasks considered in this section, choose Hindsight Experience Replay (HER) (Andrychowicz et al. 2017; Crowder et al. 2021) as a natural baseline. Generally, DEP-MPO performs much better than vanilla MPÒ (Fig. 4 For the more challenging environments, the combination of DEP with HER yields the best results.
DEP-MPO also solves the human-hop task, which is not goal-conditioned. As the reward only given for exceeding threshold COM-height, MPO never encounters non-zero reward.
To the best our knowledge, we are the first to control the 7-DoF human arm with RL on a muscle stimulation level-thal that is with 50 individually controllable muscle actuators. In contrast, (Fischer et al.
2021 only added activation dynamics and motor noise to 7 torque joints a E R!.
DEP-MPO
MPO
DEP-HER-MPH
HER MPO
success rale
return
0.3
0.0
0.0
2.5
5.0
0.0
0.5
1.0
1.5
2
0.00
0.25
0.50
0.75
1.00
steps
x.10"
steps
x107
steps x107
steps x107
(a) arm26
(b) ostrich--foe forrici-
(c) humanreacher
(d) human-hop
Figure 4: Training performance for all sparse tasks. (a) DEP-MPO and DEP-HER-MET MPP quickly
solve the task. (b) While DEP-HER-MPO performs on par with HER-MPO. DEP-MPO allows the agent solve the task in contrast to MPO. (c) DEP-MPO achieves better data-efficiency over MPO, while the best agent is DEP-HER-MPO. We conjecture that HER enables more effective use of the unsupervised data. (d) DEP-MPP finds sparse rewards even in the absence of goal-conditioning, which makes the application of HER infeasible. MPO does not encounter any non-zero reward.
5.4 APPLICATION BIPEDAL. LOCOMOTION
While the results above show that DEP-RL performs well several reaching tasks where complete state-space coverage is desirable, it unclear how it handles locomotion. Useful running gaits only occupy minuscule volume in the behavioral space and unstructured exploration leads the agent falling down very often. To test this, we apply DEP-RL to challenging locomotion tasks involving an 18-muscle human and 120-muscle bipedal ostrich. As Barbera al. (021) also attempted train
7

---

Published as a conference paper at ICLR 2023
rumning gait from scratch, choose their implementation of TD4 baseline for ostrich-run, while we provide MPO and DEP-MPO agents. See Suppl. C.4 for additional baselines.
High velocity running When applying DEP-MPO the human-run task (Fig 5 left), we observe similar initial learning between MPO and DEP-MPO. At ze 10t steps, DEP-MPO policies suddenly increase performance after a plateau. This coincides with the switch an alternating gait, which is faster than an asymmetric gait. At the end of training, 5 out of 5 random seeds achieve alternating gait with symmetric leg extensions, while MPO only achieves this for out of seeds (Fig. 5 right).
In ostrich-run, we observe faster initial learning for DEP-MPO (Fig. 6 left). There is, however, drop in performance after which the results equalize with MPO. Looking at the best rollout of the 10 evaluation episodes for each point in the curve (Fig. 6 observe that some DEP-MPO trials perform much better than the average. If we consequenly record the velocity of the fastest policy checkpoint from all runs for each method, we observe that DEP-MPO achieves the largest measured top speed (Fig. 7 right). The DEP-MPO policy is also characterized by overall stronger leg extension and symmetric alternation of feet positions. See Suppl. [C. for visualizations.
In contrast, extensive hyperparameter tuning did not lead to better asymptotic performance for MPO (Suppl. B.6), nor did other exploration noise types (Suppl. C.4).
rollout
best rollout
DEP-MPO
MPO
avg: relative x (m)
0.5
2000
0.0
DEP-
00.5
left foot
right foot
left foot
right foot
steps
x10"
steps
x107
Figure 5: DEP improves performance in the presence of local optima,
in human-run.
Left: DEP-MPO initially performs identically to MPO, before a sudden increase in performance can be observed for all trials. Right: The final gaits learned by DEP-MPO possess high symmetry, as measured by the averaged relative pelvis-deviation of the feet. Only out of 5 random seeds for MPO achieves alternating gait, which coincides with larger velocities and task returns.
rollout
hest rollout
mean rollout
top speed
6000
8000
velocity (m/s)
10.0
SOCE
DIXX return
6000
7.5
return
4000
2000
2000
2.5
0.0
1.0
MPO MPO
TD4
DEP MPO MPO
TD4
steps
X00
steps
x10
Figure 6: Learning curves and maximal returns for ostrich-run. Left: TD4 learns fast at first, but only achieves suboptimal "hopping" gait. DEP-MPO outperforms the final MPO performance initially, but decays during late training. Even though the returns seem close, DEP-MPO achieves the fastest ever recorded top speed on the ostrich. Top speeds are averaged over 50 test episodes, using the fastest checkpoint for each method, marked by diamond markers. Statistical significance marked with (**) for < 0.01 and (***) for < 0.001 using a student t-test. TD4 return is significantly lower than both other strategies (**3 not shown for clarity of the figure).
Robustness against perturbations To investigate the obtained policies, we evaluate their robustness to cuitt--distribution (OOD) variations the environment. The policies are trained on a ground (Sec.[5.4) 5.4) and then frozen, which call in-distribution (ID). Afterwards, various OOD perturbations in the form of a large step of varying heights, a series sloped steps or two inclined planes are introduced and the performance is measured probe the robustness of the learned policies.
DEP-MPO yields the most robust controller against stepdown and sloped-step perturbations for all considered tasks (Fig. 7). As MPO is unable achieve good behavior without DEP for human-
8

---

Published as a conference paper at ICLR 2023
Table 1: Training averaged performance metrics for the considered tasks, marks success rates for reaching tasks, while (R) marks returns for all other tasks. Tasks for which HER is not applicable are marked with n.a. We perform student-t tests between the best DEP- augmeentd policy and the best non-DEP policy with significance levels: (*): 0.05; (**); p < 0.01; (***); p 0.001
DEP-MPO
DEP-HER-MPE
MPO
HER-MPO
arm26 (S)
0.95 + 0.01 (***)
0.93 + 0.01 0.90 + 0.03 (***)
0.62 0.23
0.85 = 0.09
ostrich-foraging (S)
0.42 = 0.09
0,00
0.88 = 0.03
humanreacher (S)
0.72 + 0.16
0.82 + 0.12 (*) n.a.
0.50 0.23
0.65 + 0.24
human-hop (R)
472 (***)
0+
n.a.
ostrich-run (R)
4432 + 702 (**)
n.a.
4064 + 732
n.a.
human-run (R)
1601 320
n.a.
1395 + 325
n.a.
hopstacle, we compare the performance of DEP-MPO with (OOD) and without (ID) perturbations. Interestingly, DEP-MPO is very robust, except for one random seed. This policy learned not to hop, but move of its legs above its shoulders, increasing its COM-height. Although hard achieve, this behavior is sensitive to perturbations. Final policy checkpoints are used for all experiments.
1.0
sUccess rate
slopetretee stenddwn
Xic Nerdoown
height (cm)
step beight gight (cm)
40
1.5
hoppsaake
distance
1.0
20
0.5
0.0
DEP-MPO
ramddom seed
Figure 7: DEP-MPO is the most robust against all considered perturbations. Ostrich: DEP-MPO performs best under stepdown perturbations for varying step heights. The starting distance of the step was randomly varied. For the slopetrotter task, the average achieved distance is largest for DEP-MPO, although with considerable variability. Human: DEP-MPO also performs better for stepdown perturbations. For human- hopstale, 4 out of seeds of DEP-MPO achieve robust hopping. The remaining seed found non-hopping solution achieving good returns that not robust.
To our knowledge, we are the first to produce robust running gait of this speed with RL applied to system with 120 muscles, and we achieve this without reward shaping, curriculum learning or expert demonstrations. Without demonstrations, Barbera et al. (2021) only achieved a "hopping" behavior.
6 CONCLUSION
We have shown that common exploration noise strategies perform inadequately on overactuated sys- tems using synthetic examples. We identified DEP, controller from the domain of self-organization, of being able to induce state-space covering exploration in this scenario. We then proposed way integrate DEP into RL to apply DEP-RL unsolved reaching (Fischer al. (202T) and locomo- tion (Barbera et al. (02TT) tasks. Even though we do not use motion capture data or training curricula, we were able to outperform all baselines. With this, we provide ample evidence that exploration is key issue in the application of RL to muscular control tasks.
Despite the promising results, there are several limitations to the present work. The muscle simulation in MuloCo is simplified compared to OpenSim and other software. While we provided results in the more biomechanically realistic simulator HyFyDy, the resulting motions are not consistent with human motor control yet. In order for the community to benefit from the present study, further work integrating natural cost terms or other incentives for natural motion is required. Additionally, the integration of DEP and RL, while performing very well in the investigated tasks, might not be feasible for every application. Thus, a more principled coupling between the DEP mechanism and an RL policy is interesting future direction.

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
We provide extensive experimental details in Suppl. B.1 descriptions of all the used environments in (uppl.[B.2jam all used hyperparameters together with optimization graphs Suppl. [B.6] The used RL. algorithms available from the TonicRL package (Pardo 2020). The ostrich environment (Barbera 2022) and the human-run environments are publicly available. The latter was simulated using default model in SCONE (Geijtenbeek] 2019), an open-source biomechanics simulation software. We employed a recent version which includes a Python API, available at: https: titu scone- core Additionally, we made use of the commereial SCONE plug-in HyFyDy (Geijtenbeek] (021). which uses the same muscle and contact model as vanilla SCONE, but significantly faster. We further include all environments and variations as well the used learning algorithms in the submitted source code. All remaining environments are simulated in MuJoCo, which is freely available. curated code repository will be published. We also mention hardware requirements in Suppl. B.5
ACKNOWLEDGEMENT
The authors thank Daniel Höglinger for help in prior work, Andrii Zadaianchuck, Arash Tavakoli and Sebastian Blaes for helpful discussions and Marin Vlastelica, Marco Bagatella and Pavel Kolev for their help reviewing the manuscript. A special thanks goes to Thomas Geijtenbeek for providing the scone Python interface. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS for supporting Pierre Schumacher. Georg Martius is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This work was supported by the Cyber Valley Research Fund (Cy CC-VRFF-2000- 1 to DH and GM). We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: OIIS 18039B)
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL
wtwpaarrwwwweeeeeeooooooeeeee... Sesiid-sSAAANON 11NxXWWbb
Mazen Al Borno, Jennifer I.. Hicks, and Scott I.. Delp. The effects of motor modularity on per- formance, learning and generalizability in upper-extremity reaching: A computational analysis. Journal The Royal Society Interface, 171167):0020011, 2020. doi: 10. 10.1089/sf...222.00000
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, Curran As- httprr/psscccpppe/////////..
sociates, Inc., 2017. URL http::tttt/p/ooooott URL/nttp:::/prrceediiine .ew.seeeediinnnn..ee........ 453fadbd8 ott.aaaaaa..............3333133.
Vittorio La Barbera. OstrichRL, 2022.
URL
ostrichr/
Vittorio La Barbera, Fabio Pardo, Yuval Tassa, Monica Daley, Christopher Richards, Petar Kormushev, and John Hutchinson. OstrichRL: musculoskeletal ostrich simulation to study bio-mechanical locomotion. In Deep RL Workshop NeurIPS 2021, 2021. httptppss//oppereeieeee net/
Ondrej Biza, Dian Wang, Robert Platt, Min-Willem van de Meent, and Lawson L.S. Wong. Action priors for large action spaces in robotics. In Proceedings of the 20th International Conference on Autonomous Agents and MulriAgemt Systems, AAMAS '21, pp. 205-213, Richland, SC. 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 9781450383073.
Dieter Buchler, Heiko Ott, and Jan Peters. A lightweight robotic arm with pneumatic muscles for robot learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 4086-4092. IEEE, 2016. ISBN 978-1---7771 118-1---77--02--33 doi: 10. 1100/ICRAA A. 2016.7487599
10

---

Published as a conference paper at ICLR 2023
Douglas C. Crowder, Jessica Abreu, and Robert F. Kirsch. Hindsight Experience Replay Improves Reinforcement Learning for Control of a MIMO Musculoskeletal Model of the Human Arm. IEEE Transactions on Neural Systems and Rehabilitation Engineering, pp. 1016-1025, 2021. doi: 10.1 109900SRE .3081055
Scott I.. Delp, Frank C. Anderson, Allison S. Arnold, Peter Loan, Ayman Habib, Chand T. John, Eran Guendelman, and Darryl G. Thelen. Opensim: Open-source software to create and an- alyze dynamic simulations of movement. IEEE Trans. Biomed. Engineering, 54(11):1940- 1950, 2007. URL http::/ttpplp uni-trier de/ /tt//bb/oourrrlllbbbeeebbbbeebee htm.mm44..mmmm..m
DelpAALHJGTO7
Ralf Der and Georg Martius. Novel plasticity rule can explain the development of sensorimotor intelligence. Proceedings of the National Academy of Sciences, 112445)66224--6222, 2015. doi: 10.1073/pnas. 15084001 1508400112
http:t/:: www. www.::paaaaaagaaaaeooooo http://pw..org/...aaaa///.......
A. Diamond and O. E. Holland. Reaching control of a full-torso, modelled musculoskeletal robot using muscle synergies emergent under reinforcement learning. Bioinspiration & Biomimetics, pp. 016015, 2014. doi: (0.1088/774833112///1000
Danny Driess, Heiko Zimmermann, Simon Wolfen, Dan Suissa, Daniel Haeufle, Daniel Hennes, Mare Toussaint, and Syn Schmitt. Learning to Control Redundant Musculoskeletal Systems with Neural Networks and SQP: Exploiting Muscle Properties. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 6461-6468 IEEE, 2018. ISBN 978-1-5386-3081- doi: 10.1 10110///A.A000 8463166666
Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep Reinforcement Learning in Large Discrete Action Spaces, 2016. URL http: //arxivvvvvv///////////////a/ai/ wwt.r//wwwwg/./a///o///a///o org/abs/
07679
Gregory Farquhar, Laura Gustafson, Zeming Lin, Shimon Whiteson, Nicolas Usunier, and Gabriel Synnaeve. Growing action spaces. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3040-3051. PMLR, 2020. URL https::/proceeeee (atsss/v19//aaqquuue
mlr
html
Florian Fischer, Miroslav Bachinski, Markus Klar, Arthur Fleig, and Jörg Müller. Reinforcement learning control of a biomechanical model of the upper extremity. Scientific Reports, 11(1):14445, Jul 2021. ISSN 2045-2322. doi: 11033//41198-001-99933
URL htt.ttttt//....oomm
1038/ 0038/s415988 021 93760
Thomas Geijtenbeek. Scone: Open source software for predictive simulation of biological motion.
Journal of Open Source Software, 4(38):1421, 2019. doi: 0122 URL https
01421
Thomas Geijtenbeek. The Hyfydy simulation software, 11 2021. URL ht:::::/hyyyyyyy......m com
https://hyfydy:
Elena Glassman and Russ Tedrake. A quadratic regulator-based heuristic for rapidly exploring state space. In 2010 JEEE International Conference on Robotics and Automation, pp. 5021-5028, 2010. doi: 10. 10.100//0B01222005550777
DFB B Haeufle, M Günther, A Bayer, and Schmitt, S. Hill-type muscle model with serial damping and eccentric force-velocity relation. Journal of Biomechanics, pp. 1531-6, 2014. doi: 10.1016/j. #biomech.2014.02.000
Jakob Hollenstein, Auddy Sayantan, Matteo Saveriano, Erwan Renaudo, and Justus Piater. How do Offline Measures for Exploration in Reinforcement Learning behave? In Knowledge Based Reinforcement Learning Workshop at IJCAI-PRICAI 2020, Yokohama, Japan 2021. URL https://iis. wtt..k............o.oo httpp//b........ppppp///p//////a wemolllennttiiim 2 .pd
ac.

---

Published as a conference paper at ICLR 2023
Aleksi Ikkala and Perttu Hämäläinen. Converting biomechanical models from opensim to mujoco. In Diego Torricelli, Metin Akay, and Jose L. Pons (eds.), Converging Clinical and Engineering Research Neurorehabilitation IV. pp. 277-281, Cham, 2022. Springer International Publishing. ISBN 978-3-030-70316-53
Yifeng Jiang, Tom Van Wouwe, Fried! De Groote, and C. Karen Liu. Synthesis of biologically realistic human motion using joint torque actuation. ACM Trans. Graph,, 38(4), jul 2019. ISSN 0730-0301. doi: 10.1 145/3306346.33322666
hhttps://doi.. org 10. 1145/ 01055/333346664 3322299463336266 3322966
Emanuel Joos. Fabien Péan, and Orcun Goksel. Reinforcement learning of musculoskeletal control from functional simulations. In Anne L. Martel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou, Daniel Racoceanu, and Leo Joskowicz (eds.), Medical Image Computing and Computer Assisted Intervention - MICCAI 2020, pp. 135-145, Cham, 2020. Springer International Publishing. ISBN 978-3-030-59716-0..
Eukasz Kidziński, Sharada Prasanna Mohanty, Carmichael F. Ong, Zhewei Huang, Shuchang Zhou, Anton Pechenko, Adam Stelmaszczyk, Piotr Jarosik, Mikhail Pavlov, Sergey Kolesnikov, Sergey Plis, Zhibo Chen, Zhizheng Zhang. Jiale Chen, Jun Shi, Zhuobin Zheng. Chun Yuan, Zhihui Lin, Henryk Michalewski, Piotr Milos, Blazej Osinski, Andrew Melnik, Malte Schilling, Helge Ritter, Sean F. Carroll, Jennifer Hicks, Sergey Levine, Marcel Salathé, and Scott Delp. Learning to run challenge solutions: Adapting reinforcement learning methods for neuromuseuloskelett environments. In Sergio Escalera and Markus Weimer (eds.), The NIPS '17 Competition: Building Intelligent Systems, pp. 121-153, Cham, 2018. Springer International Publishing. ISBN 978-3- 319-94042-7.
Lukasz Kidziński, Carmichael Ong, Sharada Prasanna Mohanty, Jennifer Hicks, Sean F. Carroll, Bo Zhou, Hongsheng Zeng, Fan Wang, Rongzhong Lian, Hao Tian, Wojciech Jaskowski, Garrett Andersen, Odd Rune Lykkebø, Nihat Engin Toklu, Pranav Shyam, Rupesh Kumar Srivastava, Sergey Kolesnikov, Oleksii Hrinchuk, Anton Pechenko, Mattias Ljungström, Zhen Wang. Xu Hu, Zehong Hu, Minghui Qiu, Jun Huang, Aleksei Shpilman, Ivan Sosin, Oleg Svidchenko, Aleksandra Malysheva, Daniel Kudenko, Lance Rane, Aditya Bhatt, Zhengfei Wang, Penghui Qi, Zeyang Yu, Peng Peng. Quan Yuan, Wenxin Li, Yunsheng Tian, Ruihan Yang, Pingchuan Ma, Shauharda Khadka, Somdeb Majumdar, Zach Dwiel, Yinyin Liu, Evren Tumer, Jeremy Watson, Marcel Salathé, Sergey Levine, and Scott Delp. Artificial Intelligence for Prosthetics challenge solutions. 2019.
Dinant A. Kistemaker, A. J. K. Van Soest, J. D. Wong, I. Kurtzer, and P. L. Gribble. Control of position and movement is simplified by combined muscle spindle and Golgi tendon organ feedback. Journal of Neurophysiology, pp. 1 1126 1139, 2013. doi: 10. 15111//.007511.2011
Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lec. Scalable muscle-actuated human simulation and control. ACM Transactions on Graphics, pp. 1-13, 2019, doi: 10. 10.1145/3306346, 3322972.
Shuzhen Luo, Ghaith Androwis, Sergei Adamovich, Erick Nunez, Hao Su, and Xianlian Zhou. Robust walking control of lower limb rehabilitation exoskeleton coupled with musculoskeletal model via deep reinforcement learning, 2021. URL
rs- 1212542/v
Georg Martius, Rafael Hostettler, Alois Knoll, and Ralf Der. Compliant control for soft robots: Emergent behavior of a tendon driven anthropomorphic arm. In 2016 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pp. 767-773, 2016. doi: 10.1109/IROS. 2016.7759138.
Andrew William Moore. Efficient memory-based learning for robot control. Technical report, University of Cambridge, 1990.
Yael Niv. Reinforcement learning in the brain. Journal of Mathematical Psychology, pp. 139-154, 2009. doi: jo.p.00... 12.000.
Fabio Pardo. Tonic: A deep reinforcement learning library for fast prototyping and benchmarking. http:://gghhhaooocco///aa/ro///p 2020.
12

---

Published as a conference paper at ICLR 2023
Cristina Pinneri and Georg Martius. Systematic self- self-exploration of behaviors robots in dynamical systems framework. In Proc. Artificial Life XI. pp. 319-326 MIT Press, Cambridge, MA, 2018.
doi: 10.1162/isal 00062. URL https: www. mitpress journals
jiipresssouunalss...... org/
10. 11 6221221112 10.1112/ia0 111l://aa____0_00__ 3_062
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinck, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learning (CoRL) 2020, 2020. URL BtKLLttss///ooccom ger..ggtthh...... io/
cor (cor12000pappe _apppe___77]]______
Miruna Pislar, David Szepesvari, Georg Ostrovski, Diana I. Borsa, and Tom Schaul. When should
agents explore? In International Conference on Learning Representations, 2022. URL https
//openreview.oo net A www.:rummidddwwwxxiddde
Christopher T. Richards and Enrico A. Eberhard. In vitro virtual reality: an anatomically explicit mus- culoskeletal simulation powered by in vitro muscle using closed-loop dassue-softeel interaction. Journal of Experimental Biology, 223(10), 05 2020. ISSN 0022-0943 doi: 10.1242/jeb.210054 URL
1242/ 21 200054 jeb210054.
Robert Rockenfeller, Michael Günther, Syn Schmitt, and * Götz, Thomas. Comparative sensitivity analysis of muscle activation dynamics. Computational and Mathematical Methods in Medicine, pp. 1-16, 2015. doi: 10. 1..1155/015/588449
Jonas Rubenson, Denham B. Heliams, David G. Lloyd, and Paul A. Fournier. Gait selection in the ostrich: Mechanical and metabolic characteristics of walking and running with and without an aerial phase. Proceedings of the Royal Society of London. Series B: Biological Sciences, pp.
1091-1099, 2004. doi: 10. h..1098/sspb.2004.222
Katherine R. Saul, Xiao Hu, Craig M. Gochler, Meghan E. Vidt, Mclissa Daly, Anca Velisar, and Wendy M. Murray. Benchmarking of dynamic simulation predictions in two software platforms using upper limb musculoskeletal model. Computer Methods in Biomechanics and Biomedical Engineering, pp. 1445-1458, 2015. doi: 10. 1011080/112558222204444
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312-1320, Lille, France, 7 2015. PMLR. URL https: httpss:/ppooceedingg
mlr ptt..
html
T Siebert and Rode, C. Computational modeling of muscle biomechanics. In Zhongmin Jin (ed.), Computational Modelling of Biomechanics and Biotribology in the Musculoskeletal System, pp. 173-204. Woodhead Publishing, Elsevier, 1 edition, 2014. ISBN 978-0-85709-661-1 doi: 10.1533197805700679922211
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinchart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning
Representations, 2021. URLL/tttps://ppenreiiee wtw.:eerrreeeeeeeeeeeerreree wtt.rrrrmmm......-000000....
Seungmoon Song, Lukasz Kidziński, Xue Bin Peng, Carmichael Ong, Jennifer Hicks, Sergey Levine, Christopher G. Atkeson, and Scott L. Delp. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. Journal of NeuroEngineering and Rehabilitation, 18(1):126, Aug 2021. ISSN 1743-0003. doi: 2221186//122844400000999999 URL
https://doi ow.. 1186/11860001 (tt.://s1844440 02111111228--0000111--1111111111
Richard S. Sutton and Andrew Barto. Reinforcement Learning: An Introduction. Bradford Book, 2018. ISBN 0-262-03924-9.
Gabriel Synnaeve, Jonas Gehring, Zeming Lin, Daniel Haziza, Nicolas Usunier, Danielle Rothermel, Vegard Mella, Da Ju. Nicolas Carion, Laura Gustafson, and Daniel Gant. Growing Up Together: Structured Exploration for Large Action Spaces. 2019. URL https: htpp:////p/ee//wwww net/ [formm? id 175gkke
13

---

Published as a conference paper at ICLR 2023
Ehsan Tahami, Amir Jafari, and Ali Fallah. Learning to Control the Three-Link Musculoskeletal Arm Using Actor-Critee Critic Reinforcement Learning Algorithm during Reaching Movement. Biomedical Engineering: Applications. Basis and Communications, pp. 1450064, 2014. doi: 10.4015/
$101623721450064
Arash Tavakoli, Mehdi Fatemi, and Petar Kormushev. Learning to represent action values as a hypergraph on the action vertices. In International Conference on Learning Representations, 2021. net forum id=Xv 364FiXIv
URL httpps:://ppeeeeeeeeeeeeee http://wwwee///or...te.
Juan Camilo Vasquez Tieck, Marin Vlastelica Pogančić, Jacques Kaiser, Arne Roennau, Marc-Oliver Gewaltig, and Rüdiger Dillmann. Learning continuous muscle control for multi-joint arm by extending proximal policy optimization with liquid state machine. In Véra Kürková, Yannis Manolopoulos, Barbara Hammer, Lazaros Iliadis, and Ilias Maglogiannis (eds.), Artificial Neural Networks and Machine Learning ICANN 2018, pp. 211-221, Cham, 2018. Springer International Publishing. ISBN 678-3-030-01411-6.
J. Timmer and M. König. On generating power law noise. A&A, 300:707-710, 1995.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model- based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10. (09/IROS.2012..6866 2012.6381199
G. E. Uhlenbeck and L. S. Omstein. On the theory of the brownian motion. Phys. Rev. 36:823 841, 1930. doi: 10.1 103/Physsee.36.825 URL httpp:::/inx...ps.orrg/o/oo PhysRev. 36.823
J.M. Wakeling, C. Tijs. N. Konow, and * A.A. Biewener. Modeling muscle function using experi- mentally determined subject-specific muscle properties. Journal of Biomechanics, pp. 110242, 2021. doi: hi..eeh...oommcc..2.......1..
Johannes R. Walter, Michael Günther, Daniel F. B. Haeufle, and Syn Schmitt. A geometry- and muscle-based control architecture for synthesising biological movement. Biological Cybernetics, pp. 7-37, 2021. doi: 10. 00007//000422002-00858555
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML 16, pp. 1995-2003. JMLR.org org, 2016.
Kunkun Zhao, Haiying Wen, Zhisheng Zhang, Manfredo Atzori, Henning Müller, Zhongqu Xie, and Alessandro Scano. Evaluation of Methods for the Extraction of Spatial Muscle Synergies. Frontiers in Neuroscience, 16, 2022. ISSN 1662-453X. URL https: /www. frontiersin org/articles 3389/ fnins 2022. 732156
14

---

Published as a conference paper at ICLR 2023
Supplementary Material
Videos of the observed exploration patterns and learned policies are available here 2 A curated code repository will be published upon acceptance.
A
THEORETICAL BACKGROUND
A.1 STATE-SPACE COVERAGE
Following (Glassman & Tedrake 2010 Hollenstein et al. 2021). a possible measure for joint- space coverage in low-dimensional spaces can be obtained by projecting all recorded joint values ui 12: ...., zi. with E {1, 2} into discrete grid. For minimum and maximum joint values and b, and grid of size N2, let there discretized vector of values 24 = a k Az, with Ar (b-a)/N and & E {I, N - 1}. We can then compute matrix Stj such that:
1
if 3t such that 2; < ol 21+1 and 2j < q 2j+1
S
(5)
0,
otherwise
The coverage then given by:
DigSu Siy
$
(6)
N2
which is the number of visited grid points divided by the total grid size. In practice, the metric will reach 100% as the arm cannot reach every point in space due to its geometry.
A.2 ORNSTEIN- UHLENBECK NOISE
The OU-process (Uhlenbeck & Ornstein, 1930) is stochastic process that produces temporally correlated signals. It defined by:
Z+41 21 0(p x4 rt) + a We.
where w, WV100, 1) is noise term sampled from standard Gaussian distribution that drives the process, 0 controls the strength the drift term, a the strength of the stochastic term and is the mean. For our experiments, we set zo f s.t. and remain as tunable parameters. In practice, actions computed by the OU-process might exceed the allowed range E [-1, 1] for large a; actions are subsequently clipped the minimum and maximum values.
A.3 COLORED NOISE
The color of random noise is defined by the frequency dependency of its power spectral density (PSD):
PSD(f) & f
(8)
where 3 the frequency exponent of the pawer-law, sometimes colloquially referred as the color of the noise. For uncorrelated, or white, noise 3 = 0 and the PSD is constant. In general, larger values of 33 lead to noise signals with slower frequency contributions. While colors such as white (5 0), pink (33 = 1) and red (3 2) were investigated in Sec. Se2. we allowed any value 3 > 0 for the optimization Sec. C.4 In practice, we use an identical implementation of colored noise to (Pinneri et al. (020), which is based on efficient Fourier transformation. The tuneable parameters are the color of the noise 3 2 0 and scaling parameter a 2 0 which is multiplied by the noise values. Actions exceeding the allowed ranges clipped, identical to the previous section.
A.4 MUSCLE MODELING
The force production in biological muscles is quite complex and state state-dependent (Wakeling et al. 2021 Siebert & Rode 2014 Haeufle et al., (2014). In contrast to most robotic actuators, the force depends non-linearly on muscle length, velocity, and stimulation. The muscle also has low-pass filter characteristics, making the control problem hard for classical approaches-ai addition to the typical redundancy of having more muscles than DoF. While these properties are all reproduced in the
hutps://stesggogll.com//eieie/
15

---

Published as a conference paper at ICLR 2023
MuJoCo internal muscle model, which has been used for other muscle-based studies (Barbera et al. 2021] Fischer et al. [022]] Ikkala & Hämäläinen 2022 Richards & Eberhard 2020). MuloCo uses certain simplifications. It employs phenomenological force-length and force-velocity relationships which are modeled as simple functions. An additional choice, is that the tendon that connects muscles and bones is inelastic. While in more realistic systems the tendon length might vary independently of the muscle length, in MuJoCo we have:
lental
lmuuclle + Itendon
constant
where lmuscll is the length of the muscle fiber and hendon the length of the tendon. Knowledge of the muscle fiber lengths muscle consequently allows the unique determination of hotal and to infer the current joint configuration. While this choice is sensible considering the immense data requirements of RI., it simplifies the control problem compared to realistic biological agents which must infer hotal from other proprioceptive signals, which certain studies suggest to be possible in biological systems (Kistemaker al. 2013).
We also point out that the chemical muscle activation dynamics in MuloCo induce low-pass filter on the applied control signals, such that temporally uncorrelated actions might not cause significant motion. The muscle activity am (t) is governed by the dynamics equation:
a(t) am (t)
dm (t)
(10)
T(0m (t), (t))²
where is the action computed by the RL policy and is an action and activity dependent time scale, given by:
Fact + (1)) /deact/ /0..5 + 1.5 1.5 am (t))
if af > am if af())) S am am(()
T( (t)(())a)(t))
(11)
The constants Tact and Tdeact are set to 0.01 and 0.04 by default, such that activity increases faster than it decreases.
See Barbera et al. (202T) and the MuJoCo documentation for more details on the muscle model and its parametrization.
Real biological systems also suffer significant delays for sensors and actuators, well as being restricted to certain input modalities. These constraints are currently not modeled in MuloCo.
B EXPERIMENTAL DETAILS
B. GENERAL DETAILS
For the state coverage measures (Fig. and 12 we recorded 50 episodes of 1000 iterations each. The environment was reset after every episode. The state coverage metric was computed over episodes a time, after which we reset the internal state of DEP.
All experiments involving training (Fig. and 6) were averaged over 10 random seeds. Each point in the learning curves corresponds to 10 evaluation episodes without exploration that were recorded regular intervals during training.
For the maximum speed measurements, the fastest checkpoint out of all runs in Fig. 0 was chosen for each method. We then executed 50 test episodes without exploration and recorded the fasted velocity within each episode.
For the robustness evaluations, the last training checkpoint of each run in Fig. 6 is chosen, as the robustness of the policies generally increases with training time in our experiments. We then record 100 episodes each with ostrich-stepdown and ostrrch-slopeerrotter perturbations, without any exploration. For the former, binary success is recorded if the ostrich is able to pass the step and run for 10 additional meters afterwards. For the latter, we record the average traveled distance, as large number of obstacles prevents most rollouts from successfully running past of them.
For the gait visualizations (Fig. [55 anjand the same policies for the speed measurements used, as they exhibit the most natural gats.. We then record a single episode and visualize the last 5 seconds to ensure converged pattern.
16

---

Published as a conference paper at ICLR 2023
B.2 ENVIRONMENTS
All tasks except OstrichRL (Barbera et al. 2021) and the human environments Geijtenbeek (2019 9
202I) were constructed from existing geometrical models in MuJoCo (Ikkala & Hämäläinen 2022
Todorov et al., 2012) from which we created RL environments. We additionally created variants of ostrich-run involving perturbations, i.e. ostrich-stepdown, and ostrich setriih--loppetrotter
torquearm 2-DoF arm that moves in a 2D plane and is actuated by 2 torque generators. is not used for RL. but as a comparative tool. Its geometry is identical to arm26, but different joint positions are reachable as it is not restricted by the geometry of the muscles. We manually restrict the joint ranges to of E [-120, 120] (degrees) to prevent self-collisions.
arm26 A 2-DoF planar arm driven by muscles. The model was adapted from the original one in (Todorov et al. 2012), we modified the maximum muscle forces and shifted the gravity such that the arm fully extends ("down" on Fig. D. The agent has to reach goals that are 5 cm in radius. They randomly appear in the upper right corner in a 35 cm by 15 cm rectangular area. The arm motion is restricted compared to torquearm by the passive stretch of the muscle fiber. The reward is given by:
{
if d(s) < 0.05 1-1, otherwise,
10.
r(8,8')
(12)
where d(ss) is the Euclidean distance between the hand position and the goal. The episode terminates the goal is reached, i.c. d((s) < 0.05. The negative reward incentivizes the agent to reach the goal as quickly possible. Exploration in this task is difficult not only because of the overactuation, but also because the activation dynamics of the museles require temporal correlation for effective state coverage. An episode lasts for 300 iterations, with At 10 ms.
humanreacher A 7-DoF arm that moves in full 3D. It is actuated by 50 muscles. The agent has to reach goals of 4 cm that randomly appear in front of it at "face" "face""heiight in a 15 cm by 30 cm by 25 rectangular volume. The reward function and termination condition are identical to arm26, except for the goal radius. In addition to the issues detailed in the previous paragraph, singular museles are not strong enough to effect every joint motion. For example, pulling the arm above the shoulder requires several muscles be stimulated at the same time, while opposing, antagonistic, muscles should not be active. The muscular geometry is also strongly asymmetric. An exploration strategy has to compute the right correlation across connected muscle groups, and across time, for each motion. The joint limits and the bone geometry create cul-de-sac states, e.g. at some point the agent might not be able to extend the clbow further to reach a goal, it has to move back and change the pose. The initial pose of the arm is fully extended and points downwards. We randomly vary the joint pose by ginit + (0, 0.01) and each joint velocity by Ginit + (0, 0.03) after each episode reset. This helps the RL agent and HER to make progress the task as it causes the arm to slightly self-explore. As DEP is fully deterministic, it also prevents it from generating the same control signals during each episode in the initial unsupervised exploration phase. An episode lasts for 300 iterations, with Al = 10 ms.
ostrich-foraging This task is unchanged from (Barbera et al.) 202T), ) except for the rewards which we modified to be sparse, identical to arm26. The termination condition is also identical. An ostrich neck and head actuated by 52 muscles need reach randomly appearing goals with the beak. The goals appear in a uniform sphere around the beak, but only goals with goal-beak distances d(s) E [0.6, 0.8) are allowed. The goals have a radius of 5 cm. The initial pose is an upright neck position (see Fig. D. but following the original task (Barbera et al. (021) the pose is not randomized after episode resets, the last pose of the previous episode is simply kept as the first pose of the new episode. It is thus very unlikely for an agent with inadequate exploration to ever encounter single goal. The neck itself is very flexible and offers almost no casily reachable cul-de-sac states, which we conjecture to explain the good performance of HER-MPO in Fig. 4 An episode lasts for 400 iterations, with At 25 ms.
The bipedal ostrich, from (Barbera et al. 2021), needs to run fast as possible in a horizontal line and is only provided a weakly conatly-onsttaiiingg reward in form of its velocity. Only provided with this generic reward and without motion capture data, learning agent is prone to local
ostrich-run
17

---

Published as a conference paper at ICLR 2023
optima. The bird possesses 120 individually controllable muscles and moves in full 3D without any external constraints. The reward is given by:
((s - C.CM(Ms)
(13)
where Uz (s) is the velocity of the center of mass projected to the x-axis. An ideal policy will consequently run in a perfectly straight line as fast as possible. The episode terminates the head of the ostrich is below 0.9 m, the pelvis is below 0.6 or the torso angle exceeds -0.8 0rrrso (s) < 0.8 (radians). The leg positions are slightly randomized at the end of each episode, which lasts for a maximum horizon of 1000 iterations with A = 25 ms.
We point out that the author's implementation of ostrich-run (Barbera 2022) has set a default stiffness to all joints in the simulation. While this generally ensures the stability of the model, that only applies to joints that connect different parts of the system. In this case, the stiffness was also set for the root joints of the ostrich, essentially creating a spring that weakly pulls it back to the starting position. As the absolute x-position is withheld from the agent create a periodic state input, the non-observability of the spring force destabilizes learning. We therefore explicitly set the stiffness of all root positional and rotational joints joints to 0. This explains why our TD4 baseline reaches significantly higher scores than in the work by (Barbera et al.) 2021). Our measured maximum return for TD4 lies Az 4044, while the reported returns without the change did not seem to exceed 2000.
ostrich-stepdown A step is added to the original ostrich-run task. The height of the step is adjustable and its position randomly varies with Ar (-10, 0.2). The ostrich is initially top of the step and has to run across the drop in height without falling over. The task is successful the ostrich manages to run for 10 meters past the step. The episode is terminated if the x-position exceeds 10 m, the torso angle exceeds -0.8 < (sorso(s) < 0.8 rad, the head is below 0.5 m or the head is below the pelvis height. We relaxed the termination conditions to allow for suboptimal configurations that are used to bring the ostrich back into running pose.
ostrich-slopetrotter A series of half-sloped steps is added the orig- inal ostrich-run task. The obstacles are sloped on the incoming side while there is perpendicular drop similar a conventional step on the outgoing side. Rectangular stairs would disadvantage gaits with small foot clearance, while the half-slope allows most gaits move up the step without getting stuck. There are seven obstacles spaced at 5 m intervals in total. The episode terminates the x- x-position exceeds 50 m, the re- The
maining termination conditions are identical to ostrich-stepdowe obstacles are wide enough to prevent slightly diagonal running gaits from simply avoiding the obstacles. The task reward is the achieved distance, given at the end.
Figure 8: Sloped step for ostrich-slopetrotter.
human-run This task uses the planar human model from the NeurIPS competition (Kizzinski et al.) 2018)) simulated in HyFyDy instead of OpenSim. HyFyDy uses the same muscle and contact models as OpenSim, but is significantly faster. The model has 18 leg muscles and no arms. The task reward is the COM-velocity in x-direction, identical to the ostrich. The episode length is 1000. The initial position is slightly randomized from standing position. The episode terminates if the COM-height falls below 0.5 m.
human-hop In this task, the reward of human-run is changed to be sparse. The agent receives a reward of if its COM-height exceeds 1.08 m and 0 otherwise. Periodic hopping will maximize this reward. The task is particularly challenging as there is no condiiiodii to improve exploration in early training. The agent has to figure out a single hop from scratch. The initial state is slightly randomized from squatting position.
human-stepdown
The human-run task modified by including a parcours of varying slope with large drop at the end. We record a success if the agent is able to navigate the entire parcours without falling to the ground.
human- hhostacle
The human-hop task is modified by including two inclined slopes. The task is marked as a success if the agent is able periodically hop for 1000 time steps without falling the
18

---

Published as a conference paper at ICLR 2023
Table 2: Number of joints, state and action dimensions for all considered tasks. The ostrich-run variants ostrich-stepdone and ostrich-slopetrotter share identical state and action spaces, as the policies are not retrained. The planar reaching tasks torquearm and arm26 are used with virtual action spaces. For a given action multiplier we also multiply all muscle-related state data by i.e. muscle lengths, velocities, forces and activity.
torquearm
arm26
humanreacher
ostrich-foraging
ostrich-run
human-run
# of joints action dimension
2
2
21
36
56
9
2...600
6...600
50
52
120
18
state dimension
16
34
248
289
596
194
episode length
300
300
300
400
1000
1000
time step
10 ms
10
10 ms
25 ms
25 ms
10 ms
ground. Due the arrangement of the slopes, the agent will either hop inside the funnel, or jump out to the sides, where it will experience a steep drop.
Additional information regarding state and action sizes are summarized in Table 2 The observations are summarized in T
B.3 DEP IMPLEMENTATION
We use window of the recent history to adapt the DEP controller during learning. DEP requires as input proprioceptive sensor per actuator. We use joint angles for the torque-driven example in Fig. I while all muscle-driven tasks use the sum of muscle lengths and muscle forces, normalized with recorded data lie in [-1,
SDEP Iiusslle c. fuuscles
(14)
where (muscle is the length the muscle fibre, muscle the force acting on it and is scaling constant. Note that SDEP E IR' and IR" with m n. Even though the rest of the state information is discarded, DEP computes action patterns that achieve correlated sensor changes. When alternating between DEP and the policy in DEP-RL, we also feed the current input to DEP and perform training updates. We observed performance benefits in locomotion tasks as DEP's output strongly influenced by the recent gait dynamics induced by the policy. DEP is implemented to compute a batch of actions for batch of parallel environments such that there a separate history-dependent controller for each
Table 3: State information for all environments. The variants estrich-stepdone and ostrich- slopetrotter use identical observations to ostrich-run. This allows the evaluation of the robustness of the trained policies against OOD perturbations.
environment
observations
torquearm
joint positions, joint velocities, actuator positions, actuator velocities, actuator forces, goal position, hand position
arm26
joint positions, joint velocities, muscle lengths, muscle velocities, muscle forces, muscle activity, goal position, hand position joint positions, joint velocities, muscle lengths, muscle velocities, muscle forces, muscle activity, goal position, hand position joint positions. joint velocities, muscle activity, muscle forces, muscle lengths, muscle velocities, beak position, goal position, the vector from beak position the goal position
arm750
ostrich-foraging
ostrich-run
head height, pelvis height, feet height, joint positions (without x), joint velocities, muscle activity, muscle forces, muscle lengths, muscle velocities,
COM-x-velocins
human-run
joint positions (without x), joint velocities, muscle lengths, muscle velocities, musele forces, muscle activity, y-position of all bodies, orientation of all bodies, angular velocity of all bodies, linear velocity of all bodies, COM-x- velocity, torso angle, COM-y-positionn
19

---

Published as a conference paper at ICLR 2023
Table DEP hyperparameters for the learned policies. The test episode value signifies that an episode without DEP is recorded every N episodes. The value for arm-reaching was so large that was effectively never used. The force scale value is used scale the force input and the muscle length input.
(a) Arm-reaching settings.
(b) Ostrich settings.
Parameter
Value
Parameter
Value
DEP
1000
DEP
K buffer size
20
т
80
buffer size
600
90
bias rate
0.00002
bias rate
0.03
s4avg
s4avg
time dist (At)
60
time dist (A1)
integration
Pswitch HDEP
0.01
integration
Pswitch HDEP
0.0004
20
test episode
n.a.
test episode
force scale
0.0003
force scale
0.0003
(c) Human-run run settings.
(d) Human-hop settings.
Parameter
Value
Parameter
Value
DEP
K
1896
DEP
1288
T
26
T
35
buffer size
200
buffer size
200
bias rate
0.004154
bias rate
0.0926
s4avg time dist (At)
s4avg
0
time dist (At)
6
integration
Pawitch HDEP
0.01
integration
Pewitch
0,005
HDEP
30
test episode
n.a.
test episode
force scale
0.000054749
force scale
0.000547
environment. As the control matrix is very small, e.g. C E 120 even in ostrich-run, the most computationally intensive environment, this can done with minimal overhead.
B.4 RL IMPLEMENTATION
Our RL algorithms are implemented with slightly modified version of TonicRI. Pardo 2020).
B.5 HARDWARE
Training of each DEP-MPO agent for ostrich-run, the most computationally intensive environment, was executed on an NVIDIA V100 GPU and 20 CPU cores. Training for 10 iterations requires about 48 hours in real-time. Note that in general, we do train for 30 iterations for every 1000 environment interactions, which speeds up training with regard to reported learning steps. See Sec, B.6 for details.
B.6 HYPERPARAMETERS
We first detail the optimization choices made in the main part, before give the specific hyperpa- rameters that were chosen.
Optimization for ustrich-run We performed extensive hyperparameter optimization for the ostrich- run task with baseline MPO, but could not achieve a better final performance than default MPO parameters. The best performing set identical in performance the best run with default parameters
20

---

Published as a conference paper at ICLR 2023
Table S: RL parameters for MPO and TD4. The TD4 parameters are identical to BBrrbera et al.
2021
Non-reported values are left to their default setting in TonicRL. (Pardo 2020).
(a) MPO settings.
(b) TD4 settings.
Parameter
Value
Parameter
Value
buffer size
le6
buffer size
le6
batch size
256
batch size
100
steps before batches
3e5
steps before batches
5e4
steps between batches
1000
steps between batches
50
number of batches
30
number of batches
50
n-step return n parallel
3
n-step return learning rate
1
20
le-4
n sequential
10
TD3 action noise scale
0.25
parallel
15
sequential
8
exploration
OU
Action noise scale
0.25
Warm up random steps
le4
Table 6: Baseline parameters. For HER, 80% of the time a relabelled transition is added in addition to the original one.
(a) OU-noise settings.
(b) Colored noise settings.
Parameter
Value
Parameter
Value
humanreacher
0-drift
0.004
humanreacher
3-color
0.04
asccle
0.02
a-scae
0.1
ostrich-run
O-drift 0.1
ostrich-run
f-color
0.008
o-scale
0.07
o-scale
0.3
(c) Hindsight experience replay settings.
Parameter
Value
strategy
final
% hindsight
80
21

---

Published as a conference paper at ICLR 2023
iteration
iteration 5
iteration
iteration 9
best
6000
75
30
frequency
40
50
40
retum
5000
25
10
4000
0
5000
5000
0
5000
5000
2
4 5 6
return
return
return
retur
iteration
Figure 9: Hyperparameter optimization for ostrich-run. We performed iterations of meta- optimization for MPO with 100 sets of parameters in each round, for a total of 900 different combinations. The final best did not outperform the best run with the default parameters of MPO. The rightmost figure shows the best performing set for each iteration. The best return achieved over 10 evaluation episodes by MPO with default parameters out of 10 seeds shown with dashed black line.
in Fig. 6 In total, we computed iterations of neta-optiiization with 100 different sets of parameters in each round. The evolution of the performance histograms is shown in Fig.
Exploration experiments For the experiments in Sec. 5.2 all noise strategies, with the exception of DEP, were tuned grid search to maximize the end effector state space coverage for each task and for each action space separately. DEP was tuned once to maximize sample joint-pace entropy measure of the humanreacher task; its hyperparameters were then kept constant for all arm nem-reeching tasks in all sections of our study.
DEP-RL We identify three groups tunable parameters for DEP-RL: the RL agent parameters (Ta- ble 5). the DEP parameters (Table 4 and the parameters controlling the integration of DEP and the policy. We initially optimized only for the parameters of DEP and the integration. When we afterwards ran an optimization procedure for all sets of parameters the same time, we could not outperform our previous results. A pure MPO parameter search did also not yield better performance, such that we kept the parameters of MPO identical to the default parameters the TonicRL. library, except for minor changes regarding parallelization and batch sizes. The DEP parameters for DEP-RL in the reaching tasks were kept identical to the previous paragraph, while we heuristically chose the integration parameters. We, therefore, had set of values for all arm-reaching tasks in the entire study, The DEP and the integration parameters for ostrich-run were optimized for performance, we kept them identical for ostrich- ostrich-foraging.
Baselines The additional baselines for humanreacher and ostrich-run, see Suppl. C.4 were tuned individually for each task to maximize performance. TD4 is used with identical parameters (Barbera et al. 202T). The values are detailed in Table
22

---

Published as a conference paper at ICLR 2023
ostrich-run
humanreacher
meem rollout
best rollout
mean rollout
best rollout
0230
cundl saices rale
7r0m
4020
9030
de dech
aer wwbfforradiin
Gree
Adtyy
seeh Siree aech bechfoooradiii
seed
dew
Altyy
strgle
eoch bebefonnaadie
mecun rollout
best rollout
mean rollout
best rollout
6050
TROK gpcces Pen
16
Figure 10: Ablation experiments for DEP-RL. The stoch-force-ve forci-variant (bold) was used for all experiments in the main part. All ablations were trained for 5 10? iterations and averaged over 10 random seeds.
C ADDITIONAL EXPERIMENTS
C.I STATE-COVERAGE
We show additional visualizations of the trajectories generated by different noise processes on torquearm and arm26 in Fig. 11 and Fig. 12 respectively.
White
Pink
Red
OU
DEP
While
Pink
Red
OU
DEP
2 actioos
hand-y
0.6
600 actions hund-y
0.2
30
600
hand-x
hand-x
hand-x
hand-x
hand-x
Figure 11: Only DEP reaches adequate state-space coverage for all considered action spaces in torquearm. Hand trajectories collected during 50 episodes of 1000 iterations (Ar = 10 ms) of pure exploration with different noise strategies. Left: Hand trajectories for the original action space a E R2 (top) and expanded action space a 600 (bottom). Right: Endeffictor-sppac coverage.
C.2 ABLATIONS
We experiment with several implementations of DEP-RL and show cumulative and maximum performances for a selection of them in Fig. 10 The ablations are:
init DEP is only used for initial unsupervised exploration. The collected data is used to pre-fill the replay buffer. This component is active in all other ablations.
avg DEP actions and policy actions are combined a weighted average. The DEP weight is much smaller than the policy weight.
det DEP and the policy control the system in alternation. They are deterministically switched s.t. DEP acts for HDEP iterations and the RL policy for HgL iterations. The current state is used train and adapt the DEP agent, even if the RL action is used for the environment. DEP actions are also added to the replay buffer of the RI. agent.
23

---

Published as a conference paper at ICLR 2023
White
Pink
Red
OU
DEP
White
Pink
Red
OU
DEP
actions
hund-y
0.13
600 actions
hund-y
0.30
hand-x
hand-x
hand-x
hand-x
hand-x
A of actions
Figure 12: Only DEP reaches adequate state coverage for all considered action spaces in arm26 Hand trajectories collected during 50 episodes of 1000 iterations (At 10 ms) of pure exploration with different noise strategies. Left: Hand trajectories for the original action space a E E (top) and expanded action space a E T0:000 (bottom). Right: endeffector-space coverage.
stoch Identical to the previous ablation, but the alternation is stochastic s.t. there is probablitiy Pswitch that DEP takes over for HDEP iterations.
We additionally introduce force-variants of all these ablations where the state input of DEP is not only composed of the muscle lengths, but also the forces acting on the museles. We observed that this causes DEP to seek out states that produce more force variations, which are generally interesting for locomotion. Lengths and forces are normalized from recorded data and then added together with certain weighting, in order to not change the number of input dimensions of DEP.
Lastly, we show the performance for a noback-variant, where DEP is not learning in the background while the RL agent taking over control. Even though the variantriant achieves a fast gait for locomotion, we chose the seath-forre-variiaee for all the results in the main section, as it achieves good performance on all tasks. All ablations were averaged over ten random seeds.
C.3 ACTION CORRELATION MATRIX
We recorded action patterns generated from different noise strategies applied to ostrich Even though we only recorded 50 s of data, and DEP was learning from scratch, strong correlations and anti-com anniicorrelationn across muscle groups in order to observe the full bandwidth of motion generated by DEP. For this particular task, the ostrich was lying the ground while moving the legs back and forth in an alternating pattern. Uniform, colored and OU noise are unable to produce significant correlations across actions.
be observed in Fig. [3] We deactivate episode terminations
C.4 ADDITIONAL BASELINES
We combine MPO with colored (3-MPO) and OU-noise (OU-MPO) by summing them to the action computed by the baseline MPO policy. We then apply these new algorithms to humanreacher and ostrich-run, as they constitute challenging reaching and locomotion tasks. We also tested an implementation of DEP-TD4 the OU and the colored noise were optimized to achieve the best performance, see Suppl. B.6 It can be seen in Fig. 14 (left) that DEP-MPO achieves the largest returns in ostrich-run, while OU-MPO intermittently outperforms vanilla MPO. Similarly, OU-MPO and 3-MPO perform better than MPO in the humanreacher task,
ostrich-run. The base agents were identical for all baselines, while
in Fig. 14 but DEP-MPO achieves the best performance.
C.5
OSTRICH GAIT VISUALIZATION
We show the achieved foot movements and footstep patterns of the ostrich for the different algorithms in Fig. 15 The leg deviation is strongest for DEP-MPO, while also achieves the most regular foot pattern. This suggests that DEP improves exploration, as it allows for policies that utilize the embodiment of the agent to a greater extent, while also achieving larger running velocities. MPO manages less leg extension, while the TD4 gait is irregular and asymmetric. The step lengths of x Im achieved by DEP-MPO are also quite close to real ostriches (Rubenson et al. 2004), while MPO and TD4 only achieve small step lengths of ss 0.5m and 0.3m respectively. We provide additional visualizations of the relative r and z trajectories of the feet during locomotion for each algorithm in Fig. 16
24

---

Published as a conference paper at ICLR 2023
DEP
uniform noise
colored noise
OU noise
muscle
60
60
121
60
121
60
121
1
60
121
muscle
muscle
muscle
muscle
Figure 13: Action correlation matrix for different exploration strategies. We recorded 50 of data (1000 transitions) from ostrich-run and computed the correlation matrix from the action trajectories. Note that even though DEP was initialized with Cij = O, strong correlation and anti-correlation patterns can be observed for antagonistically opposed muscle groups. Colored and OU noise do not exhibit strong correlations across actions, as they are designed to produce temporally correlated signals.
ostrich-run
humanreacher
mean rollout
best rollout
rollout
1.00
6000
success rate
0.75
retur
4000
2000
steps
x 107
steps
x10
steps
x107
rollout
best rollout
mean rollout
cumul. success rate
0.8
6000
max return
0.6
2000
0.2
0.0
DEP-MP
MPO
DEP-TD4
TD4
OU-MPO
R-MPO
DERMPO
MPO
DEP-TD4
TD4
OU-MPO
8-MP
MPO
MPO
OU-MPO
BAPO
Figure 14: Left: Additional baselines for ostrich-run. We provide OU and 8-MPO agents by summing them to the action computed by MPO with regular exploration noise. Right: Identical baselines for humanreacher. OU and colored noise processes were optimized for the present tasks, while the base MPO agent was identical for all experiments in this figure.
25

---

Published as a conference paper at ICLR 2023
DEP-MPO
MPO
TD4
LP
RF
rel. foot-x (m)
0.5
LP
0.0
RF
-0.3
LF
RP
20
211
22
23
24
25
20
21
22
23
24
23
20
21
22
24
28
20
21
22
24
time (8)
time (s)
time
time (5)
Figure 15: DEP-MPO achieves the most widespread and symmetric gait. Left: The relative sagital foot position w.r.t. the torso visualizes the leg extension during locomotion. DEP-MPO creates symmetric gait with s Im step length. For MPO the step length much shorter. TD4 has completely shifted gait, the left foot is often in front of the right foot. Right: Foot contact pattern for all gaits. The shaded areas mark the time during which the respective foot (LF: left foot or RF: right foot) is in contact with the ground Visualized are the last 5 seconds of an evaluation episode to ensure converged pattern.
1.0
fool z (m)
0.5
0.0
0
0
foot (m)
foot (m)
foot - (m)
Figure 16: Foot gait patterns for ostrich-run during the final s of episode. While DEP-MPO
portrays slight asymmetry in the z-direction, MPO and TD4 are noticeably less symmetric.
C.6 ADDITIONAL EVALUATION WITH DENSE AND SPARSE REWARDS LARGE VIRTUAL ACTION SPACES
We present in Fig. 17 the results for experiments with virtual action spaces, shown in Fig. but here in addition with HER and for the case of sparse and dense rewards. For dense rewards (top), an increasing number of actions requires more environment steps for vanilla MPO solve the task, while the performance collapses for 600 actions. DEP-MPO reaches good performance for each considered action space. MPO performs similarly in the sparse task, albeit a larger variation across runs can be observed. While HER elicits faster learning, it still requires significantly more environment steps for and 120 actions than DEP-MPO and does not achieve good performance for 600 actions. DEP-MPO and HER-DEP-MPO quickly solve all tasks.
C.7 DESCRIPTION OF A UNE-DIMENSIONAL SYSTEM
In this section, we give an outline how DEP works a theoretical scenario. We will first describe simplified DEP rule and detail how its dynamics might excite the mountain car system (Moore) 1990) to cover the state. We will then apply the simplified rule on the mountain car environment and present the results, see Fig. 19
Mountain car
The original DEP controller is described by:
a, tanh(C's h+),
(15)
with the state 8, E R", a time-dependent bias h, E Rm,, the action a, E R and the learned control matrix C e pmxt The update rule is now defined
r f
(16)
where f(-) R
R is inverse model, relating future changes in the state to the change in action that caused them. First, we state the assumptions for this section:
1. We do not consider the normalization scheme for C.
26

---

Published as a conference paper at ICLR 2023
DEP-MPO
MPO
DEP-HER-MEL
HER-MPO
actions
120 actions
600 actions
dense
return
400
400
400
600
600
600
success rate
sparse
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
steps
X10
steps X X10'
steps
X107
Figure 17: DEP-MPO outperforms MPO in sparse and dense reward point-reaching for arm26 with all virtual action spaces. Top: Learning performance for dense rewards. DEP-MPO strongly outperforms MPO, no significant movement learning could be detected for MPO with 600 actions. Bottom:
Success rates for sparse reward reaching. While HER seems to increase the performance of the MPO baseline in most cases, the success rate only increases marginally for 600 actions, even after almost 1.5 x 10? steps. DEP-MPO solves all tasks with without the addition of HER.
2. We choose f(At) = &
3. A bias is not considered h; =
4. The nonlinearity is approximated by the first term of a Taylor expansion tanh(x) x.
5. We consider C' to instantaneously fulfill the update rule, without considering update dynam- ics.
Combining all the assumptions, obtain:
C 8,,8-11
(17)
We will now consider a simple system with 1 sensor: the continuous action mountain car.
The original environment defines the RL state sRL (z,, -z). From this we extract the sensor information for DEP s; = It, with s E R. In this 1-sensor formulation, the velocity correlation matrix in Eq. 17is is scalar, there is only sensor. Consequently, also the C' matrix is scalar. The resulting control equation is:
Cs. $181-181.
(18)
Let us assume an initial state of the car slightly to the right side of the valley, as in Fig. 18 with positive initial velocity. States the left. The initial velocity will cause the car to move up the mountain. After some time, the velocity correlation will be 8,8,-1 > with >> 0, leading to C's; = Ot > Logically, the car then starts pushing to the right, reinforcing the movement pattern and trying to increase its velocity. However, the task is set up such that the force insufficient to directly go up the mountain. will thus change the movement direction at some point and reverse due to gravity.
are positions the right of the valley bottom, while
to
After changing direction, 8,š;-1 will reverse sign as the previous velocity still points to the right, while the current velocity points to the left. Thus, 8, < with > and Cs. = a; < 0. The will consequently try push into the negative direction, accelerating downwards. If the past sensor derivative $;-1 1 defined as only one time step away, this trend will immediately reverse and the car will decelerate.
If, however, & is chosen to be not one time step apart from $;-1, 1. but At E N steps, then will take several time steps until the sign reversal of š At happens. In this intermittent regime, š,8,-At with > 0, and Cs, = a; < 0, pushing the car to the left, until the velocity correlation changes sign again.
If At is appropriately chosen, however, by the time the reversal happens, the will have moved into the negative state region s < O. this new region, Š > with s < O, and Cs; = a, < 0, which pushes the further to the left and up the mountain, until the phenomenon repeats.
27

---

Published as a conference paper at ICLR 2023
Figure 18: Continuous-action mountain car environment. To solve the task, the car must be brought the top of the right hill. However, the motor is too weak for a direct approach. The solution involves pushing the car from left to right and vice-versa with the correct frequency to gather momentum and climb up the mountain.
We offer simulation results of the described dynamics in Fig- For this simulated example, consider again the nonlinearity of tanh (Eq. [5). as it allows us to multiply C' by large constant and still satisfy the action limits of the environment. This is a necessary step as omitted the normalization scheme for C. We thus consider C = tanh ( with > 1.
To demonstrate the influence of At, we plot the results for two examples (Fig. 19 For = 27, the system trajectory at time step x 95 shows a reversal point. Before this reversal point, the position positive and the correlation Ś1وt- which leads positive actions. Due to gravity and the weak motor, however, the car starts to move into the opposite direction. As there was velocity sign change, we have 8 av while the position 8, is still positive, which yields negative action: The car is accelerating downwards and builds up momentum, which, after few repetitions, allows it to explore the environment.
The values of A/ for this example were chosen to yield good visualization. We observe full exploration of the mountain car system for At ?5.
This might seem like an overly simplified example, but it shows how DEP can increase the variance in a sensor value, even a large number of actuators is associated to it. Empirical evidence additionally demonstrates that in high- tigh-dimensional systems, if all other DEP components becomes less sensitive to the exact system dynamics and parameter specifications, generalizing easily muscle-driven systems with over 120 muscles. Note that the mountain task contains harmonic potential well, ubiquitous in many physical systems, the analysis might hold for wide range of models, even considering elastic muscles with nonlinear spring elements.
considered, DEP
C.8 SENSITIVITY TO CHANGES Pswitch
We performed ablation study the parameter Pewitch. that controls the probability of switching from the RL policy to the DEP controller. Figure[0 22 shows that the training averaged success rates for DEP-MPO much less sensitive to this hyperparameter for the humanreacher task than the average returns for ostrich-run. We conjecture that locomotion is inherently more unstable and that higher DEP probabilities cause the agent to fall down often, which hurts learning performance.
28

---

Published as a conference paper at ICLR 2023
At 27
At 50
random actions
0.05
0.00
0.05
3t3t- At
0.000
0.001
action
20
60
80
100
120
140
160
time steps
A simplified DEP rule in the mountain car environment (Moore 1990 The red line marks the position threshold at which the task solved. The black lines mark the zero point. Negative values of 6,3, the system into coherent motion. For At = 27, the reversal of š,Št-A happens on a time scale that is able to excite the system, while the setting At 50 is not able to induce sufficient exploration. The random actions are drawn from a standard Gaussian (0,, 1). We do not show the proposed actions for the Gaussian to keep the figure readable. The position for the setting At 27 clearly oscillates with larger and larger amplitudes over time, increasing the effective variance the sensor value. These two values of At were chosen to yield good visualizations, observe full exploration of the mountain car system for values A/ E (5.... 28].
Figure 19:
mark intermittent regimes where the controller output reverses, eventually bringing
DEP-MPO
MPO
humanreacher
ostrich-run
1.00
5000
cumul. success rate
0.75
avg. return
4500
0.50
4000
0.25
3500
0.00
3000
10
10
2
10
10
Pswitch
Pswitcc
Figure 20: Ablation over the DEP probability with DEP-MPO for two tasks. Left: Humanreacher performance is overall robust for different settings of Pswitch- while the benefit of DEP disappears for very small values. Right: Ostrich-run more sensitive to the parameter, as locomotion generally more unstable. Large DEP probabilities cause the agent to fall down very often. The horizontal line marks the average MPO performance without DEP. All values are averaged over seeds. Note the log-axis.
29


Reviews:
[{"review_id": "pnUgGNQ8VUl", "reviewer": "Reviewer_Bv9J", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines differential extrinsic plasticity (DEP) and reinforcement learning for more efficient learning of how to control overactuated systems. Demonstrations are done on simple hand design systems as well as complicated muscle-driven locomotion control. ", "summary_of_the_review": "This provides a new and intuitive explanation of why it is hard to learn controllers for overactuated systems with standard RL approaches using Gaussian noise. DEP is introduced to effectively resolve this issue. A wide range of examples is used to demonstrate the effectiveness of the proposed method,  including state coverage of a simple overactuated system and challenging locomotion control with bipedal systems with many muscles. I believe this paper makes a big step forward in solving muscle-driven control problems.", "strength_and_weaknesses": "Pro:\n\n1. A novel combination of RL and DEP for effective learning for muscle-driven systems. Experiments demonstrate the proposed approach outperforms existing baselines.\n\n2. The method is well motivated by issues surrounding the difficulty to explore with overactuated system. I especially appreciate the simple example that is introduced to provide intuition.\n\nCon:\n\nNot much. To nitpick a little, the motion quality for the ostrich control is still not natural. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "e-PGQEyxdUD", "reviewer": "Reviewer_PfGN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies a novel and interesting problem: muscular control with overactuated action spaces. It first demonstrates that the main issue with overactuated action spaces is how to effectively and efficiently perform exploration using a clear torquearm example with 2 and 600 DoF theoretically and empirically. It further proposes a framework that integrates differential extrinsic plasticity, a method from the domain of self-organization, into the exploration stage of reinforcement learning. The experimental results across torquearm, arm26, human reacher, ostrich-foraging, ostrich-run, human-run, and human-hop show that the proposed framework can learn a better policy with overactuated action spaces compared to various baselines. Moreover, this work also shows that the learned policy is robust to some forms of environmental changes.", "summary_of_the_review": "This work proposes integrating DSP into the traditional RL framework to tackle the inefficient exploration issue with overactuated action spaces. The idea of utilizing DSP for a more effective exploration is self-motivated. In addition, the studied problem, proposed idea, and the final results in the paper are well-presented. The quantitative results are impressive, and the qualitative results show that the proposed framework can successfully learn a robust policy with overactuated action spaces.", "strength_and_weaknesses": "+) The idea to integrate differential extrinsic plasticity (DSP), a method from self-organization, into the traditional RL framework is interesting and self-motivated. Although the way of integration (alternating exploration policy between DSP and learned policy) is simple, it tackles the core problem causing ineffective policy learning with overactuated action spaces. Moreover, the studied problem in this work is also essential and appealing.\n\n+) The paper is well-written and easy to follow. It clearly demonstrates its task, challenges (theoretically and empirically), and the proposed idea (with necessary background reviews). The figures clearly illustrate the results qualitatively and quantitatively. The supplementary videos also show the effectiveness of the learned policy with many qualitative examples.\n\n+) The proposed method works well across different agents (torquearm, arm26, ostrich, human arm, human walk) on different tasks (reaching, foraging, walking, hopping). The robustness experiments further show that the learned policy is able to maintain its performance better than baselines.\n\no) As mentioned by the authors in the conclusion section, the integration of DEP and RL in this work is firmly straightforward, and it might not directly provide much impact on other applications. I am looking forward to seeing more general or sophisticated frameworks, including ideas inspired by neuroscience or biology.\n\n-) I am curious about the sensitivity of the policy to the switch probability p_switch in the Intra-episode exploration stage. In the supplementary (table 4), the p_switch seems hand-picked for different agents for different tasks. Do authors conduct an ablation study regarding it?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "V7wjBRLwTL", "reviewer": "Reviewer_sbak", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Conventional exploration strategies for RL scale poorly to high-dimensional action spaces. \nDifferential Extrinsic Plasticity (DEP), integrated with RL, is proposed as a mechanism for coping\nwith highly-redundant actuation spaces, such as that of multiple musculotendon units that span joints\nin biomechanical models. The key idea of DEP is to learn a state-to-action mapping that is largely defined\nby a linear state-to-action matrix, C, which is updated by the velocity correlation matrix of the current\nand previous state.  This induces its own dynamics, without a goal task, and has been shown in prior\nwork on self-organizing behavior, to produce converge to useful stationary behaviors.  In this work,\nit is proposed to integrate this with RL, via alternating DEP and RL motion phases during learning.\nThe results are tested on a wide variety of scenarios and over-actuated systems. Various baselines\nand ablations are presented, including Uhlenbeck-Ornstein noise, colored noise, MPO and TD4 results, \nintegration with HER, and more.\n", "summary_of_the_review": "The paper makes significant progress on tackling an important and difficult problem, \nnamely that of high-dimensional action spaces that arise in over-actuated systems\nsuch as musculoskeletal models. It makes a very valuable contribution in this regard.\n\nThe underlying assumptions needed to apply this framework could be articulated a bit more clearly,\nwhen coming from the common RL mindset, where there is generally great freedom in the state specification.\nThe paper is in a good position to also further state caveats and differences in terms of bridging from RL to\nthe biomechanics community, which may have different evaluation criteria, related to motion quality,\nexperimental comparisons to ground-truth data, and predictive capabilities.\n\n-------------------------------------------------------------\nDetailed comments follow below.\n\nGiven the importance of matrix C, perhaps give it a name that connects with its implied role?\nConceptually, I now see that it represents the weights of a one-layer feedf-foward network.\n\nThe original DEP paper discusses the role of updating the threshold terms, \"to\ndrive the neurons away from their saturation regions.\"  Is this still important in this work, to stay\naway from converging to static poses, or does the integration with RL obviate this need?\n\nPerhaps some of the intuition provided in the original DER paper could be reiterated in this paper, to help\nthe reader, i.e., “chaining together what changes together.” \n\nIs the value of Delta t kept fixed for the bulk of the results?\n\nThe coverage of the related work is already excellent in the current paper, although\nthere is additional early work on muscle-based locomotion that also comes to mind, e.g.,\n\"Optimizing Locomotion Controllers Using Biologically-Based Actuators and Objectives\" (2012)\n\"Flexible Muscle-Based Locomotion for Bipedal Creatures\" (2013),\nwhich are both objective-based optimizations, albeit with strong priors on the control architecture.\n\nTo what extent can the final structure of the matrix C be interpreted?\nThis would be useful to here about. Is there an emergent low-dimensional structure?\nIs this structure consistent across multiple runs?  \n\nFor muscles spanning multiple joints, the inverse mapping, f, presumably becomes much less clear,\nas it now also depends on the motion of more than one joint.  Does this have any impact on the\nability to achieve good results?  Relatedly, I'm also curious to better understand how critical\nit is to have knowledge of the 1:1 correspondence between sensors and actuators.\n\n4.1 DEP introduction\n\nIt is not clear in the description at this point if the state s can be very general, as in RL,\nor if it refers much more specifically to the length of the muscle actuators. \nI believe that it is the latter, so it is worthwhile clarifying. \n\nText after eqn (2): \"f is an inverse model\"\nThis begs the question \"an inverse model of what?\".\n\nUpdate rule for C dot, with \"-C\" being a damping term. Perhaps call this a first-order decay term,\nwhich presumably acts as a regularizer?\n\nWhat forms of perception are available for the last three ostrich tasks, i.e., foraging, stepdown, slopetrotter?\nPerhaps it is in the appendix, but providing basic knowledge of \"is the their knowledge fo the upcoming terrain\"\nwould be useful.\n\nAppendix A \"... chemical muscle activiation dynamics\"\nIt's great to see this discussed, are these actuation dynamics being modeled or not?\nPresumably not (which is fine as a simplification), but state this explicitly.\n\nDEP-MPO for the human hopping is, subjectively speaking for this reader, the highest quality result. Very nice.\n", "strength_and_weaknesses": "Strengths:\n\n- important and difficult problem:  high-D action spaces, redundant actuation, musculoskeletal control\n- impressive results and progress on the above problem\n- progress towards understanding how overactuation might be resolved in natural systems\n\nWeaknesses:\n- missing possible intuition about how the redundant actuation is ultimately resolved \n- writing: The underlying assumptions needed to apply DEP could be articulated more clearly,\n  when coming from the common RL mindset, where there is generally great freedom in the state specification.\n- mixed motion quality of some of the results, subjectively speaking, e.g., Ostrich running, Ostrich head-and-neck during gaits\n- unclear how more classical biomechanics criteria, i.e., metaboloic energy consumption, might come into play,\n  or if these results are predictive in any way of what can be observed in natural systems\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RqSYFPipfZ", "reviewer": "Reviewer_x7yt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an exploration strategy for reinforcement learning. This exploration strategy is based on the differential extrinsic plasticity rule in neuroscience which outputs temporal correlated and state-dependent exploration noise. The exploration strategy can also be parameterized as a neural network and this paper update it by using a gradient-free method. During training, it is used as an intra-episode exploration and the overall method achieves large improvement over baselines on overactuated and musculoskeletal systems in simulation.", "summary_of_the_review": "In summary, I think this paper is a good paper and worth reading for the community. The exploration can be helpful for varies applications such as RL from image observations or RL on more complex systems.", "strength_and_weaknesses": "Strength:\n1) This paper is well-motivated and the writing is clear. \n2) The idea of using the differential extrinsic plasticity strategy is interesting and it’s novel to adapt it in the reinforcement learning tasks.\n3) The authors promised to release the code, which could be a benefit to the whole community.\n\nWeakness:\n1) Is there a specific reason of using MPO as the baseline, instead of some other methods like TD3, SAC, PPO? \n2) Can the authors provide some qualitivative visualization about what is the learned exploration strategy?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%