Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks
Authors: Isabell Wochner
Pierre Schumacher
Georg Martius
Dieter Büchler
Syn Schmitt
Daniel Haeufle
Venue: CoRL 2022 Poster (2022)
Keywords: reinforcement learning
model predictive control
actuator morphology
morphological computation

Abstract: Humans are able to outperform robots in terms of robustness, versatility, and learning of new tasks in a wide variety of movements. We hypothesize that highly nonlinear muscle dynamics play a large role in providing inherent stability, which is favorable to learning. While recent advances have been made in applying modern learning techniques to muscle-actuated systems both in simulation as well as in robotics, so far, no detailed analysis has been performed to show the benefits of muscles in this setting. Our study closes this gap by investigating core robotics challenges and comparing the performance of different actuator morphologies in terms of data-efficiency, hyperparameter sensitivity, and robustness.

Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks
Isabell Wochner*
Pierre Schumacher 22,
Georg Martius
Dieter Büchler
Syn Schmitt
Daniel E.B. Hacufle 1,3
Institute for Modelling and Simulation of Biomechanical Systems, University of Stuttgart, Germany Max Planck Institute for Intelligent Systems, Tübingen, Germany Instti-Instiiuue for Clinical Brain Research, University of Tübingen, Germany
Abstract: Humans are able to outperform robots in terms of robustness, versatility, and learning of new tasks in a wide variety of movements. We hypothesize that highly nonlinear muscle dynamics play a large role providing inherent stability, which is favorable to learning. While recent advances have been made in applying modern learning techniques
systems both simulation as well as in robotics, so far, no detailed analysis has been performed to show the benefits of muscles when learning from scratch. Our study closes this gap and showcases the potential of muscle actuators for core robotics challenges in terms data-efficiency, hyperparameter sensitivity, and robustness 2.
muscle-actuane
Keywords: reinforcement learning, model predictive control, actuator morphology
Marphologs
Tiaa
Marphology
Aetivarion dynamier
Mursele
FXTVU
pla)
mando
ixtida velockty
Figure 1: Key differences between torque actuator morphology and muscle actuator morphology.
1 Introduction
Recent developments new learning methods allow the generation of complex anthropomorphic motions such as grasping, jumping or hopping in robotics. However, current systems still struggle with real-world scenarios beyond the narrow conditions of laboratory experiments. Humans, on the other hand, are capable of quickly adapting to uncertain, complex, and changing environments in sheer endless variety of tasks. One key difference between biological and robotic systems lies in their actuator morphology: robotic drives are mostly designed to yield a linear relation between control signal and output torque. In contrast, muscles have complex nonlinear characteristics.
has already been demonstrated, that muscular nonlinearities may provide a benefit for stability and robustness, especially under environmental uncertainties or perturbations [1, 2, 3]. A benefit linear torque actuator morphology has been observed in computer simulations by exchanging the actuator morphology (similar Fig. in otherwise identical anthropomorphic tasks like reaching
"Equal contribution. Equal contribution.
"See https:
muscles for code videos.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

or locomotion [5. 6, 7, 8]. Similarly, reduced demand on the information processing capacity has been shown for muscles when compared to torque actuator morphology [9, 10, 11. 12, 13]. This opens the question whether muscular morphology could also be beneficial for robustness and data-efficiency in the process of learning movement control.
Recent ad vances in deep learning facilitated the generation of complex movements like point-reaching [14, 15, 16, 17] and locomotion [18, 19, 20, 21, 17] simulations with muscular actuator morphology. In the real world, optimization and learning approaches can also find controllers for robotic systems with pneumatic muscles exhibiting somewhat muscle-like actuator morphology [22, 23]. These examples demonstrate that learning and optimization methods can control muscle-driven systems and may enable benefits such as safe learning and robustness [23]. However, investigating advantages of nonlinear muscular actuator morphology over linear torque actuator morphology requires a direct comparison of both, which is is-to our wwwllleggemiiiinne in the literature.
While Peng et al. [24] performed a comparative analysis of different actuator morphologies, their work was focused on replicating reference trajectories. In contrast, we learn behaviors without demonstrations, provide extensive hyperparameter ablations and not only employ RL, but also other optimization methods applied to complex 3D models.
The purpose of this study is to test if learning with muscular actuator morphology is more data- efficient and results in more robust performance as compared to torque actuator morphology when learning from scratch. We investigate this in a very broad approach: we employ different learning strategies multiple anthropomorphic models for multiple variants of reaching and locomotion tasks solved in physics simulators of differing levels of detail. This provides new and comprehensive evidence of the beneficial contribution of muscular morphology to the learning of diverse movements.
2 Morphological difference between torque and muscle actuators
In contrast to idealized torque actuators, where torque is simply proportional to the control signal Uhoqque E [-1, 1].
(I) muscular force output nonlinearly depends on the muscle control signal Umuseles the muscle length
Timx Ugrque
MTU and contraction velocity /MTU- These biologically observed dependencies can be predicted by so-called Hill-rype muscle models [25]. In a nutshell, the model captures biochemical processes
transforming muscle stimulation U'muscle [0, 1] to the force-gener calcium ion activity a. This can be modeled by a first-order differential equation of the form [26]
= fa (2 a)
(2)
which induces low-pass filter characteristics (Fig. 1). The model further captures the nonlinear force-length and force-velocity relations [25]. The force-length relation is characterized by a positive slope (increasing force with increasing muscle fiber length) in the typical operating range of biological muscle fibres (Fig. 1). The force-velocity relation is characterized by decreasing force for faster shortening velocities and increasing force if the muscle is externlly stretched against its contraction direction (Fig. 1). A lever arm p(a) translates joint angle a into muscle-tendon-un length /MrU and muscle force into joint torque
Σ p.(a)f+ (anru,
(3)
for N muscles which span a coint-typically at least two in an antagonistic arrangement.
In practice, we employ two different muscle models: A detailed one with more physiological details, contained in demoa [27), and simpler model that efficiently adds muscular properties to existing MuloCo [28] simulations without sacrificing computational speed. See Suppl. A for details.
3 Methods
3.1
Learning approaches for movement control
We test if muscle actuator morphology facilitates learning by applying state-of-the-an learning algorithms covering an extensive range of approaches currently used in robotics. The common thread of the selected algorithms lies in their independence of the actuator morphology: this allows us to easily exchange idealized torque actuator morphology with muscle actuator morphology. We choose optimal control, model-predictive control and reinforcement learning learning approaches.

---

Table Overview of all models and tasks
Model
Task
Control
Environment
ArmMuJoCo
precise reaching
RL
MuJoCo
ArmMuJoCo
fäst reaching
RL opt. control, MPC
MuJoCo
ArmDemoa
smooth point-reaching
demoa
ArmDemoa
hitting ball with high-velncity hopping
opt. control, MPC
demoa
Biped
RI.
MuJoCo
FullBody
squatting
opt. control, MPC opt. control, MPC
demoa
FullBody
high-jumping
demoa
(a) ArmMlooo
(b) ArmDemoa
(c) Biped
(d) FullBody
Figure 2: Models used for the experiments.
Optimal control (OC)
The control problem with horizon N can be defined as:
N
min min me
(x((k, (lk)), k), subject æ<k + 1) = f(x(k)) (t)),)).
u(k) = TT< (.() x(:)).
(4)
where æ<k) E R denotes the current state at time k, and ukk) I. is the applied input at time Furthermore, i specifies the cost function, and f denotes the system dynamics. To optimize for the best control policy, we use the covariance matrix adaptation evolution strategy (CMA-ES) [29] in the optimal control case (open-loop strategy). CMA-ES is derivative- algorithm and widely used in machine learning. It combines different learning mechanisms for adapting the parameters of multivariate normal distribution. Note, that choose the same optimization parameters for both actuator morphologies to allow for a fair comparison even though the number of decision variables n, is always larger in the auscle-actuated case due the antagonistic setup.
Model predictive control (MPC) In MPC, we solve the control problem in a receding-horizon fashion with varying prediction horizons and recursively apply only the first element of the predicted optimal control sequence u0)) (closed-loo) strategy). We employ a warm start procedure using the CMA-ES optimizer and afterwards start the MPC routine with the local optimizer BOBYQA [30]. The parameters of the optimizers are either varied (see Sec. 4) or given in Suppl. B.
Reinforcement learning (RL) RL allows learning of reusable feedback controllers. Instead of minimizing a cost function (see Eq. 4), conventionally the discounted expected reward is maximized:
N-1 Σ K-1 r(k) k=
max J maax
(5)
where r(k) is the reward at time k, и is control policy and [0, 1] is discount factor such that long-term rewards are weighted less strongly. RL consequently solves a similar problem to MPC, able
but the resulting controllers prediction model. For the point-reaching tasks, we additionally employ goals characterizing the desired hand position, which then constitutes additional dependence of the reward function. The aim of the learning process is to learn controller policy 7 (u(k)) (=))) that takes as input the current sensor values, or state x(k), and outputs a control signal, or action, u(k) such that task is solved. In practice, we use the RL algorithm MPO [31], implemented in TonicRL [32].
in closed-loop fashion without being given explicit
3.2 Models
Arm The Arm model (Fig. a, b) consists of two segments connected with hinge joints (2 joints total) moving against gravity. The ArmMuJoCo [28, 17] model contains four muscles, two for each

---

joint. In the muscle-actuated case in ArmDemoa [33, 34], six Hill-Type muscles generate forces: two muscles for the shoulder and two for the elbow joint, plus two biarticular muscles acting on both joints. All joints are individually controllable.
Biped We converted the geometrical model of an OpenSim bipedal human without arms [19] for use in MuloCo. The model, consisting of 7 controllable joints (lower back, hips, knees, ankles) moves in 2D plane. Each joint is actuated by two antagonistic muscles one torque actuator.
FullBody The FullBody model [35, 36] consists of two legs and an upper body including arms based on human skeletal geometry. It consists of 8 controllable joints (ankles, knees, hips, lumbar and cervical spine) in 3D, and 14 movable joints in total including the arms. Each controllable joint was either actuated by two antagonistic muscles (muscle-actuated case) or by one idealized torque actuator (torque-actuater case). For more details, we refer to Suppl. C.
All models and their respective physics differential equations were solved with variable time step (max. time step 0.001s) in demoa and fixed time step (0.005s) in MuJoCo.
3.3 Objectives and rewards
We choose anthropomorphic movement objectives which are highly relevant for robotic applications. We expect that muscular actuator morphology provides benefits for such tasks. All task formulations allow application in musele and torque actuator morphologies with an identical reward or objective function. For a precise formulation of the used functions and conditions, Smooth point-reaching This task encourages smooth point-reaching. Therefore, the objective minimizes the L2-error between the desired and current joint angle while penalizing the angle velocity and jerk to ensure smooth motion. The desired angle is 90 for both the shoulder and the elbow joint, as this requires a large motion.
Suppl. D.
Precise point-reaching Similar to [13], we incentivize reaching random hand position in pre- determined rectangle, while minimizing the distance of the end effector to the goal. We specifically add a reward term that gives a much larger reward for precise motions that reach the center of the larget area. The episode does not terminate until a time limit of 1000 steps elapses.
Fast point-reaching The same objective as for precise point-reaching is used, but the episode terminates if the target is reached, incentivizing reaching speed over precision.
High-velocity ball serve A ball is dropped in front of the Arm model and the controller learns to hit the ball to achieve maximum ball velocity.
Squatting This squatting objective is taken from [35] and encourages desired hip, knee, and ankle angles for squatting position.
Maximum height jump The objective for the high-jumping task is taken from [37] and maximizes the position and velocity of the centre of mass of the human body model at the time of lift-off. The model is initialized to start from squatting position.
Hopping We developed an objective based on the z-axis velocity of the center of mass (COM)
of the system that encourages periodic hopping with maximum height. The episode terminates if extreme joint angles are exceeded.
4 Results
In the following, we present three major results for the investigated approaches and environments: (1) Muscle-like actuators general improve data-efficiency compared to torque- (2) The investigated learning and optimization algorithms exhibit greater robustness to hyperparameter variations when applied to muscle-driven systems. (3) The motions and controllers obtained from the muscular morphology are more robust against force perturbations that were not present during learning. We average results over 5 and 8 random seeds for OC/MPC and RL respectively.
4.1 Data efficiency: Learning with limited resources
Robotics applications in real-world scenarios often suffer from limited resources, which holds true for training and inference time. Therefore, we investigate the advantages of muscle-like actuator morphology in terms of overall learning efficiency and temporal control resolution.
Ad Advantages of muscular morphology Smooth and precise point-reaching generally require more
data with dorque-driven systems, as seen in Fig. 3. The performance of the muscle actuator, in contrast to torque morphology, varies very little for different settings of the temporal control resolution c. Precise reaching with RL also results in stable performance with fewer training iterations, and very

---

(a) smooth point reaching (0C)
(b) ball hitting (OC)
(c) squatingr(C)
(d) high jumping (OC)
-10
-10*
-10
-10
-0.6
10
100
80
100
genenttions
generstions
penerations
geneentions
lomqus, 0.05
longue, 0.15
tompue,
musele,
0.15
poiiz-reaching
point-reaching (KL.
hopping
-10
distoroo-rrtim motimmm
- -0.4
5000005 cate
-0.6
-0.8
-10
100
-10"
0.5
0.5
stops
steps
steps
musele
Figure Cost value or returns for different tasks. Plotting the mean and standard deviation (shaded area) for (OC/MPC) (RL) repeated runs for the two actuator morphologies (muscle in red, torque in blue). Additionally, the temporal control resolution e was varied in the OC cases.
smooth point-reaching (OC)
precise moint-reaching (RI.)
10
distance- ----rrumm -esture-retur
10
-0.50
2
-1.00
20
40
60
80
100
0.0
0.5
1.0
generations
steps
X10
socque, tongue
tonque,
ausele,
muscle,
c01.55
muscle
muscle, RL
tarque, RL
PD contol. OC and RL
Figure Cost value or return in comparison with PD-control for torque. outperform all other considered morphologies with OC, while PD-control achieves lower cost than torque actuation with large control resolution e 0.3. Right: PD-control does not yield an advantage over torque actuators with RL when applied to the precise point-reaching task.
Left: Muscles
small standard deviation across runs. Similar findings are seen for the squatting and hopping task. where musele-actuators achieve better data-efficiene and smaller variation across runs and are able find good-enough optimum with fewer iterations.
Advantages of torque morphology In tasks requiring fast and strong motions, without emphasis on stabilization, we find torque actuators to hold certain advantages. ball hitting and fast reaching, the torque cases show similar smaller variance, even though both actuators perform well for singular runs. The high-jumping task, where only strong, swift motion required to launch the system upwards, is solved much faster in the torque case. We also observe in the hopping task that, although only after a considerable number of training iterations and exhibiting a large variance, some torque-actuated runs achieve larger overall return than the best muscle-actuated runs.
We additionally investigated a controller for the torque actuator morphology, see Fig. 4. While the PD controller slightly improves the data-efficiency for some cases, for both OC as well as for RL, muscle actuator outperforms baselines. See Sec. F.2 for more experiments.
4.2 Robustness to hyperparameter variations
Tuning growing number of hyperparameters for learning models typically requires considerable time and computational resources. By analysing hyperparameter sensitivity, we test if tuning with torque or muscle actuator morphologies requires less resources.
5

---

0.35
c 0.155
C= 0.05
10a
C001
CONt
10
102.3
10
10
20
40
100
generitians
generations
generations
muscle, 0.05
muscle, 0.1
muscle, 0.2
muscle, 0.5
musele, 0.7
0.05
tonque,
lungue.
Iuque 0.5
loeque, 0.7
Figure 5: Muscle morphology is more robust towards hyperparameter variation (σr in point reaching. The cost value of the best observation is shown. The mean and standard deviation (shaded area) are plotted for five repeated runs for the two actuator morphologies (muscle in red, torque in blue) with different control resolutions c.
iteration
iteration 2
iteration
frequency
Ionque muscle
10
-1.0
0.8
-0.6
0.4
-1.0
-0.8
-0.6
0.4
-1.0
-0.8
-0.6
0.4
distance-return
x10
Distance-return
x10
distance-returm x10²
Figure 6: Muscle actuators need less parameter tuning for good performance. Hyperparameters optimized for precise point-reaching following iterative sampling scheme, each run trains for 2 x 10 iterations. Fifty sets of parameters are sampled randomly from pre-determined distributions, the final performance is evaluated and used to adapt the sampling distributions for the next iteration. We plot the return distributions over the sampled parameters at different iterations.
Figure 5 shows the cost curves for smooth roint-reaching for the evolutionary optimization algorithm CMA-ES for different values of a. which is the principal tuneable parameter for this algorithm. The performance curves vary much more for torque actuators for all considered cases. Furthermore, all muscle-actuated cases find a good-enough optimum with fewer iterations and a smaller variance, independent of the hyperparameter and the control resolution c.
The same task was repeated using MPC while varying the main hyperparameter prod which represents the prediction horizon in moving horizon strategies. The performance curves and the final cost vary much more for torque actuators (Fig. 7a, note, the cost is plotted logarithmically).
Finally, we performed an extensive hyperparameter optimization for precise point-reaching. For each iteration, 50 sets of parameters are randomly chosen and the final task performance is evaluated after 2 x 10th learning iterations. The sampling distributions for the parameters are then fit to the best performing runs and 50 additional sets are evaluated for the next iteration. We optimize the learning rates of MPO, as well as gradient-clipping thresholds, as these have strong influence on learning speed and stability. Muscle actuators already outperform torque-actuators in the first iteration, with a greater number of well performing parameter sets (Fig. 6). Almost no low- oww-erforminn runs remain for iteration 7, while a large torque-pete perque-performacce only achieved by a small subset of parameter settings. See Suppl. E for more hyperparameter variations.
4.3 Robustness to perturbations
In this section, we probe the robustness of the obtained policies against unknown perturbations. In precise point-reaching, we evaluated the RL reaching policies for two modifications that were not present during training: First, the hand-weight of the model is increased by 1.5 kg (dynamic load), and secondly a free spherical weight is attached to the end effector with a cable (chaotic load). We can see in Fig. that the muscle-based policy does not suffer significant changes in performance, except for small circular motion (3 cm) around the goal position in the chaotic load case. In contrast, the torque actuator morphology leads to unstable reaching and strong oscillations. Both morphologies seem to handle the dynamic load well. See Suppl. for more goals.

---

Vary pyperparameeer
Unknown perturbation weight
musele Sped - 0.2
10
10
muscle fpre 0.3
musele Spred
miscle Syed
tanque puat 0.2
loqque Fpad 0.3
10-8 88
10
juad 0.4
time [5]
time
tanque 0.5
Figure 7: Muscle morphology more robust towards variation hyperparameter pred and unknown perturbations. Plotting the development of cost over time for the two actuator morpholo- gies (muscle red, torque blue) while varying the hyperparameter /prest denoting the length the prediction horizon. Left: The unperturbed case. Right: The prediction model is not aware of the added weight to the lower arm.
musele dynamic load
muscle chantic load
muscle dynamic load
musele chaotic Inad
[m]
joints [ma)]
torque dynamic load
lorque chaotic load
tocque dynamic load
torque chaotic load
0.75
joints (rac]
hand
0.50
hand
[m]
hand
[m]
Time [6]
time [6]
Figure Trajectories for dynamic (1.5 kg weight) and chaotic (attached ball) load. Left:
Three random goals are exemplarily shown, the respective goal position is marked as circle. The
unperturbed baseline for each goal is shown with a dashed line. Right: Joint trajectories for the same
experiment, the unperturbed baselines are shown with dashed lines. Vertical bars mark episode resets.
For the MPC controller, evaluated robustness by introducing environment changes that are un- known to the prediction model. One example is the lifting of an object with unknown weight, typical robotics task. When adding kg the lower of the ArmDemoa model (Fig. 7), the performance in both actuator cases is worse than in the unperturbed case (left); the movement is also slower. However, the variance and absolute value of the final cost in the muscle-actuated case are still much lower com- pared to the torque-actuated case (plotted logarithmically), See Suppl. F.3 for more weight variations.
For periodic hopping with the Biped model, we evaluated trained RL policies with random forces that were drawn from a Gaussian distribution F ~ (N-00, or) and applied to the hip, knee, and ankle joints with a probability of 0.05 at each time step. We in Fig. 10 that the torque actuator morphology is stronger affected in relative performance than the muscle morphology. In the robustness investigation with MPC in the FullBody squatting task, force is applied the hip joint after the system has reached its desired position. Figure (left) shows that the desired joint angles are much less affected by the perturbation when muscle actuators are controlled. Furthermore, the cost value associated with the movement recovers much slower for torque actuators (Fig. 9 right).
hip anple (deg)
musole Fpor 50 muscle Fpes 50
-10
COSE
-15
torque Fpor
torque Fpst
-20
10-3
0.4
0.5
0.6
0.7
0
0.2
0.4
0.6
0.8
time [s]
time [s]
Figure 9: Hip and cost trajectory for squatting with unknown perturbation forces. The muscle morphology shown in red, the torque morphology in blue while varying the perturbation force Fpent [N] (applied between f >= 0.45 and f <= 0.5 s).

---

muscle
torque
unperturbed
10N
COM -2 [m]
1.4
w
m w Ww
relative performance
1.2
1.0
0.8
COM > [m]
VV W
0.0
0.4
0.2
30 15
15
Time [5]
time [s]
OF [N]
Figure 10: Muscle actuators lead more robust hopping, At each time step. there is 5 chance
of a random force being applied to the hip, knee, and ankle joints. Left: COM motion over time. Vertical bars mark episode resets due to falls of the Biped. Right: Relative performance for different
standard deviations of the random force pr. Performance is scaled by the unperturbed mean return.
Discussion
We investigated if muscle-like actuators have beneficial effects for modern learning methods in terms of data-efficiency, hyperparameter sensitivity and robustness to perturbations, A multitude of varia- tions across physics simulators, learning algorithms and task domains was considered in order show- case the potential of the considered morphologies independently of the underlying implementation. We showed that muscles yield benefits in tasks requiring stable motion, even when compared ideal- ized torque actuators, which can be considered an upper performance bound. Indeed, the used torque actuators are able to instantaneously output any desired force at any point of the trajectory, while muscles only slowly change their output due to activation dynamics and can only produce kinematics- dependent force output. Despite these limitations, the considered learning algorithms learn more effi- ciently with muscle actuation in all tasks, except for extreme motions where objectives require strong force application without stability considerations, such as ball-hitting and high-jumping. In bipedal hopping, it was found that muscles result in more efficient learning, even though some torque-runs achieve higher asymptotic performance. Finally, we observe muscle actuation to result in increased ro- bustness to perturbations and hyperparameter variations, which can facilitate learning real robotic systems that not only present sensor and motor noise, but also prohibit extensive parameter searches.
Outlook for real-world robotics We see two use-cases of our findings: (1) Muscular force-length- velocity and low-pass filter characteristics can be implemented as low-level actuator control for torque-controlled robotic systems (e.g., [38, 39, 40, 41]). This could allow us to exploit the improved data efficiency and robustness observed in our study for RL on a real robotic system. (2) Novel soft robotic actuators, such as artificial muscles [42, 43, 44, 45], promise to revolutionize specific application scenarios of robotics, e.g., wearable rehabilitation devices [46]. While soft actuated systems are hard to control from classical control theory point of view, our results and other works [24] suggest that RL may even benefit from their properties. In our study, the simplified MuloCo muscle model is applicable as low-level controller the sense the first use case, while the results with the complex scries-clastic muscle model in Demoa highlights the second use-case, making both cases strong arguments to consider RL and muscle properties a promising combination.
Limitations Although we have reported results for wide variety of algorithms and tasks, we cannot give theoretical statements about the general applicability of our findings. Additionally, some of the tasks we employed were limited in complexity and might also be solvable with classical control algorithms. The MuJoCo muscle model, while computationally efficient, only captures rudimentary properties of biological systems. The demoa implementation. on the other hand, includes visco-elastic, passive tendon characteristics and muscle routing as joint angle-dependent lever arms to account for many physiological details-at substantial additional computational cost. Finally, learning with intermediate control signals given to impedance or position controllers, instead of direct torque commands, might also improve learning performance, while muscle-like properties could have been introduced by learning priors or additional cost terms.
8

---

Acknowledgments
We thank Daniel Höglinger for help during the development of the hopping reward function. Further- more, we like thank Mare Toussaint, Danny Driess and David Holzmüller for initial discussions regarding the topic of learning with muscles. This work was supported by the Deutsche Forschungs- gemeinschaft (DFG, German Research Foundation) under Germanys Excellence Strategy EXC 2075 - 390740016 (SimTech). We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting all authors. Georg Martius is member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This work was supported by the Cyber Valley Research Fund (CyVy-RF-2020- 11 DH and GM). We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI
Center (FKZ: 18039B....
References
[1] H. Wagner and R. Blickhan. Stabilizing function of skeletal muscles: an analytical investigation. Journal of Theoretical Biology, 199(2):133-179, 1999.
[2] M. Eriten and H. Dankowicz. A rigorous dynamical-systee based analysis of the self- stabilizing influence of muscles. Journal of Biomechanical Engineering, 131(1):01001, 2009.
[3] S. Brändle, S. Schmitt, and M. A. Müller. Mathematical Biology A systems-theoretti analysis of low-level human motor control application to a single-joint arm model. Journal of Mathematical Biology, 11 2019.
F. [4] K. Stollenmaier, W. Ilg. and D. F. B. Haeufle. Predicting Perturbed Human Arm Movements in Neuro-Mussullsskeletal Model to Investigate the Muscular Force Response. Frontiers in Bioengineering and Biotechnology, 8(308), 4 2020. wt..3389/bbooee2220.oo
[5] A. van Soest and M. F. Bobbert. The contribution of muscle properties in the control of explosive movements. Biological Cybernetics, 6953);195-204, 1993.
[6] D. Haeufle, S. Grimmer, and A. Seyfarth. The role of intrinsic muscle properties for stable hopping Saannng-staale stability is achieved by the force-velocity relation. Bioinspiration & Biomimetics, (1):016004, 2010. doi: 10t:11...1111111 10011100/018131111111 01///4888112////00000000
[7] K. G. Gerritsen, J. van den Bogert, M. Hulliger, and R. Zernicke. Intrinsic muscle properties facilitate locomotor controla computer simulation study. Motor control, 2(3):206-222 220, 1998.
[8] C. T. John, F. C. Anderson, J. S. Higginson, and S. L. Delp. Stabilisation of walking by intrinsic muscle properties revealed in a three-dimensional muscle-driven simulation. Computer methods in biomechunics and biomedical engineering, 16(4):451-462 2013.
[9] R. J. Full and D. E. Koditschek. Templates and anchors: neuromechanical hypotheses of legged locomotion on land. Journal of Experimental Biology, 202(23):3325 3332, 1999.
[10] P. Holmes, R. J. Full, D. Koditschek, and J. Guckenheimer. The dynamics of legged locomotion:
Models, analyses, and challenges. SIAM review, 48(2):207-304, 2006.
[11] R. Blickhan, A. Seyfarth, H. Geyer, Grimmer, H. Wagner, and M. Günther. Intelligence by mechanics. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 365(1850): 220, 2007.
[12] D. Haeufle, M. Günther, G. Wunner, and S. Schmitt. Quantifying control effort of biological and technical movements: an 012716. 2014.
entropy-based approach. Physical Review E, 89(1):
[13] D. F. Haeufle, I. Wochner, D. Holzmüller, D. Driess, M. Günther, and S. Schmitt. Muscles reduce neuronal information load: quantification of control effort in biological vs. robotic pointing and walking. Frontiers Roboties and AI, 7:77, 2020.
[14] F. Fischer, M. Bachinski, M. Klar, A. Fleig, and J. Müller. Reinforcement learning control of a biomechanical model of the upper extremity. Scientific Reports, 11(1):14445, July 2021. 10.: (0388/441588-021-99933
[15] E. Joos, F. Péan, and O. Goksel. Reinforcement learning of musculoskeletal control from func- tional simulations. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 135-145. Springer, 2020.

---

[16] V. Caggiano. H. Wang, G. Durandau, M. Sartori, and V. Kumar. Myosuite a contact-rich simulation suite for musculoskeletal motor control, 2022. URL abs/2205. 13600.
[17] P. Schumacher, D. Häufle, D. Büchler, S. Schmitt, and G. Martius. Dep-rl: Embodied exploration for reinforcement learning in overactuated and musculoskeletal systems, 2022. URL https:
//arxiv. org/ /bw///////////// 2206 6. 00484.
[18] V. L. Barbera, F. Pardo, Y. Tassa, M. Daley, C. Richards, P. Kormushev, and J. Hutchinson. OstrichRL: A musculoskeletal ostrich simulation to study bio-mechanical locomotion. In Deep
RI. Workshop NeurlPS 202/, 2021. URL ht / aaaiiv.vivv org/ abs/ 21 2112. 06061.
[19] t.. Kidziński, S. Mohanty, C. F. Ong. Z.. Huang, S. Zhou, A. Pechenko, A. Stelmaszezyk, P. Jarosik, M. Pavlov, S. Kolesnikov, et al. Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments. In The NIPS'17 Competition: Building Intelligent Systems, 2018. URL http: aariiv.vvv...//..///a.a//aaaavva org haag//bb/// abb//a//// 00361.
[20] E. Kidziński, C. Ong. S. Mohanty, Hicks, S. Carroll, B. Zhou, Zeng, F. Wang, R. Lian, H. Tian, W. Jaskowski, G. Andersen, O. R. Lykkebø, N. E. Toklu, P. Shyam, R. K. Srivastava, Kolesníkov, O. Hrinchuk, A. Pechenko, M. Ljungström, Z. Wang, X. Hu, Z. Hu, M. Qiu, J. Huang. A. Shpilman, I. Sosin, O. Svidchenko, A. Malysheva, D. Kudenko. L. Rane, A. Bhatt, Z. Wang. P. Qi, Yu, Peng. Q. Yuan, W. Li, Y. Tian, R. Yang. P. Ma, S. Khadka, S. Majumdar, Z. Dwiel, Y. Liu, E. Tumer, J. Watson, M. Salathé, S. Levine, and S. Delp. Artificial intelligence for prosthetics: Challenge solutions. In S. Escalera and R. Herbrich, editors, The NeurlPS '18 Competition, pages 69-128, Cham, 2020. Springer International Publishing. ISBN 978-3-030- 29135-8.
[21] S. Song, L. Kidzińssii, X. B. Peng, C. Ong. Hicks, Levine, C. G. Atkeson, and L. Delp. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. Journal of NeuroEngineering and Rehabilitation, Aug 2021.
[22] D. Driess, H. Zimmermann, S. Wolfen, D. Suissa, D. Haeufle, D. Hennes, M. Toussaint, and S. Schmitt. Learning to control redundant musculoskeletal systems with neural networks and sqp: exploiting muscle properties. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2018.
[23] D. Büchler, R. Calandra, and J. Peters. Learning to control highly accelerated ballistic move- ments muscular robots. Robotics and Autonomous Systems, 2019.
[24] X. B. Peng and M. van de Panne. Learning locomotion skills using deeprl: Does the choice of action space matter? In Proceedings of the ACM SIGGRAPlI/Eurogrphiccs Symposium on Computer Animation, pages 1-13, 2017.
[25] T. Siebert and C. Rode. Computational modeling of muscle biomechanics. In Computational Modelling of Biomechanics and Biotribology in the Musculoskeletal System, chapter 6, pages 173 173-204. Woodhead Publishing, Elsevier, edition, 2014.
[26] R. Rockenfeller, M. Günther, S. Schmitt, and Götz, Thomas. Comparative sensitivity analysis of muscle activation dynamics. Computational and Mathematical Methods in Medicine, 2015: 1-16, 2015. doi: 10. ///5/2011/585400
[27] S. Schmitt. desoa-base: A Biophysics Simulator for Muscle-driven Motion, 2022. URL https: ottp://oooorg//////////o / darus-2550.
[28] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RS. International Conference on Intelligent Robots and Systems, pages 6026-5033 Oct.
2012. doi: 10. 109/IROS.20 2OTSS.2212..666 2.6386109.
[29] N. Hansen, S. D. Müller, and P. Koumoutsakos. Reducing the time complexity of the derandom- ized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary computation, 11(1):1-18, 2003.
[30] M. J. Powell. The bobyqa algorithm for bound constrained optimization without derivatives. Cambridge NA Report NA2009/06, University of Cambridge, Cambridge, 26. 2009.
[31] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maxi- mum posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL httttpp://axxi..oo/ org abs / 1806 06920.
10

---

[32] F. Pardo. Tonic: A deep reinforcement learning library for fast prototyping and benchmarking.
arXiv preprint arXiv:2011.07537, 2020.
[33] I. Wochner and S. Schmitt. arm26: A Human Arm Model, 2022. URL https::/d/ii.orgg
1 10. 1.841 1.499 9/ darus- 2871.
[34] I. Wochner, D. Driess, H. Zimmermann, D. Hacufle, M. Toussaint, and S. Schmitt. Optimality principles in human point-to-manife reaching accounting for muscle dynamics. Frontiers in Computational Neuroscience, 14:38, 2020.
[35] R. Walter, M. Günther, D. Haeufle, and Schmitt. A geometry-and muscle-based control architecture for synthesising biological movement. Biological Cybernetics, 115(():7-37, 2021.
[36] J. R. Walter, I. Wochner, M. Jacob, K. Stollemmaier, P. Lerge, and S. Schmitt. allmin: A Reduced Human All-Body Model, 2022. URL hwwp:: / org/ 0 1841 9/////arrr 2.922.
[37] M. G. Pandy, F. E. Zajac, E. Sim, and W. S. Levine. An optimal control model for maximum- height human jumping. Journal of Biomechanics, (111112::1111111
1990.
[38] Carcia-Cordovv, A. Guerrero-Gonzalez.
Pedreno-Molina, and Moran. Emulation of the animal muscular actuation system in an experimental platform. In IEEE International Confer- ence on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics Cyberspace. IEEE, 2001. doi: 10.1109//csmcc200000 20011999799.
[39] A. Seyfarth, K. T. Kalveram, and H. Geyer. Simulating Muscle-Reflex Dynamics in Simple Hopping Robot. In Proceedings of Fachgespräche Autonome Mobile Systeme, pages 294-300. doi: 10. 0000/99778--54-77777665 com/ 10. 1 1007 3-0---5550---- 74000--7744-2220
Springer, 2007.
URL http: //i:nk springer.
[40] J. Knüsel, A. Crespi, J.-M. Cabelguen, J. J. Ijspeert, and D. Ryczko. Reproducing five motor behaviors in a salamander robot with virtual muscles and a distributed epg controller regulated by drive signals and proprioceptive feedback. Frontiers in Neurorobotics, 14, 2020. ISSN 1662-5218. doi: articles /10. 3389 fnbot......... 2020. 64442.
URL wwtps:://////.m /ttp::://wwww// /www www.tiersinnnooooomm org/
[41] A. Rai, R. Antonova, S. Song, W. Martin, H. Geyer, and C. Atkeson. Bayesian optimization using domain knowledge on the atrias biped. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1771-1778. IEEE, 2018.
[42] I. Boblan, R. Bannasch, A. Schulz, and H. Schwenk. A human-like robot torso zar5 with fluidic muscles: Toward common platform for embodied ai. In 50 Years of Artificial Intelligence, pages 347-357. Springer, 2007.
[43] G. K. Klute, J. M. Czerniecki, and B. Hannaford. Mckibben artificial muscles: pneumatic actuators with biomechanical intelligence. In 1999 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (Cat. No. 99TH8399), pages 221-226. IEEE, 1999.
[44] B. Vanderborght, A. Albu-Schäffer, A. Bicchi, E. Burdet, D. G. Caldwell, R. Carloni, M. Cata- lano, O. Fiberger, W. Friedl, G. Ganesh, al. Variable impedance actuators: review. Roboties and autonomous systems, (111):161011110
2013.
[45] S. Wolfen, J. Walter, M. Günther, D. Hacufle, and S. Schmitt. Bioinspired pneumatic muscle spring units mimicking the human motion apparatus: benefits for passive motion range and joint stiffness variation in antagonistic setups. In 2018 25th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), pages 1 1-6. IEEE, 2018
[46] M. Zhu, S. Biswas, S. I. Dinulescu, N. Kastor, F. W. Hawkes, and Y. Visell. Soft, wearable roboties and haptics: Technologies, trends, and emerging applications. Proceedings of the IEEE, 1 110(2)24462744 2022. doi: 0099/PROC.2201.31100000
11


Reviews:
[{"review_id": "rtrw8tUqmUT", "reviewer": "Reviewer_SnhT", "summary": "This paper evaluates the impact of using a muscle actuator model as an action space for learning-based control. Data efficiency, hyperparameter sensitivity, and robustness of the learned behavior are evaluated on a set of simulated robotics tasks. The results indicate some empirical benefits of muscle actuation over direct torque control.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The analysis is thorough and the premise of bio-inspired action space design is interesting. I learned about some empirical trends for optimization with MTUs vs torque control.\n\nWeaknesses: The paper should address its relationship to some related prior work. Also, the takeaways for robot learning are ambiguous without comparing to the commonly used PD control action space.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is technically sound, and the question of action space design for learning-based control is interesting. However, uncited previous work seems to have provided a similar set of results. The authors should clarify how their work differs. Also, to make a valuable paper for robot learning practitioners, the authors should add the PD control action space as a baseline.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- A related work is notably omitted: “Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?” (Peng et al. 2017 / https://arxiv.org/pdf/1611.01055.pdf). Here, the authors compared Torque, MTU, position, and velocity action spaces regarding data efficiency and robustness on several RL locomotion tasks. This paper may have substantial differences from the results presented here but the distinction is currently unclear to me. Please add information to elucidate the differences, so that I can better assess the impact and originality of this work.\n- It's much more common to use proportional-derivative control than direct torque control in learning motor skills for robots. This is because position targets are a more efficient, robust, action space than torque control. Why not make this comparison in your experiments? It's likely to be more interesting to roboticists than a comparison to torque control.\n- In future work, I would also be curious about how model-based RL performs in the learning with muscles setting, given that it potentially needs to learn a more complex forward dynamics model. This could make for some interesting analysis. But this is just a suggestion and by no means needs to be addressed here.\n- Another good reference to consider for the impact of environment design on algorithm performance is Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning (Reda et al 2020 /  https://www.cs.ubc.ca/~van/papers/2020-MIG-envdesign/2020-MIG-envdesign.pdf)"}}, {"review_id": "n_fWSMNw6DR", "reviewer": "Reviewer_KLzq", "summary": "This paper argues that humans (and other animals) are able to perform a diverse range of motions *thanks to their muscle’s unique dynamical properties.* In particular, the authors hypothesize that \"highly non-linear muscle dynamics” provide inherent stability that makes learning tasks easier. \n\nTo verify this, the authors test if calculating robot torques ($\\tau$) from a control signal ($u \\in [0,1]$) using a muscle activation model [1] (See *Eq 3*) results in more stable learning, when compared to a standard (linear) idealized model: $\\tau = \\tau_{max} u$. This experiment is repeated across different tasks, robot morphologies, learning/control algorithms, and simulated environments. The muscle morphology has better data efficiency and is more robust to hyper-parameters changes.\n\n\n[1] Siebert, T., and C. Rode. \"Computational modeling of muscle biomechanics.\" Computational modelling of biomechanics and biotribology in the musculoskeletal system. Woodhead Publishing, 2014. 173-204.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* This paper demonstrates how muscle based actuation models can help learn better controllers in a variety of different scenarios. This is a non-trivial insight that could guide further hardware development.\n\n**Weaknesses** \n* It’s unclear if the proposed benefits are actually unique to muscle morphology. The key issue is the baseline used in their study (ideal actuator) is very naive. Most robots already use impedance controllers [2], position controllers [3], and even learned controllers [4] that may provide similar benefits (e.g. “smooth control” constraints) without requiring custom hardware.\n* The study doesn’t verify that key muscle properties (e.g. non-linearity) are actually important for better learning. For example, why couldn’t a well tuned low pass filter achieve the same level of smoothness? Thus, some of the claims in the paper (e.g. that non-linearity is a strength) are not properly backed up.\n* Finally, this paper is purely demonstrated in simulation, and it’s unclear if its conclusions will generalize to the wild. Indeed, it’s likely that real muscles have properties that are difficult to simulate and/or deviate from the paper’s activation model in significant ways. This limits the study’s potential impact on practical robot learning applications. \n\n[2] Hogan, Neville. \"Impedance control: An approach to manipulation: Part II—Implementation.\" (1985): 8-16.\n\n[3] Rocco, Paolo. \"Stability of PID control for industrial robot arms.\" IEEE transactions on robotics and automation 12.4 (1996): 606-614.\n\n[4] Martín-Martín, Roberto, et al. \"Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks.\" 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper investigates a fascinating topic and provides interesting insight on where muscle morphology may outperform idealized torque actuators. However, the practical takeaways are muddled because the baseline is very naive, and the simulated results may not be indicative of real world performance/behavior. As a result, I don’t think this paper has quite reached the acceptance threshold, but I will keep my mind open throughout the rebuttal period.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "In general, i would ask the authors to respond to my points above. Some specific issues are outlined below: \n\n* I would encourage the authors to implement stronger baselines (e.g. impedance controller [2]) as part of their study. In other words, investigate if muscles provide any benefit compared to the current state of the art in robotics. \n* The authors should discuss the real world applicability of their findings. What should roboticist take away from this study? "}}, {"review_id": "6M5Ox_GYQ7w", "reviewer": "Reviewer_vcXi", "summary": "This is a benchmarking paper evaluating muscle-like control against vanilla torque control in simple, anthropomorphic tasks. There are two kinds of muscle models deployed - the demoa model and a much simpler model written in Mujoco. There are three types of control algorithms being tested, optimal control (trajectory optimization) with CMA-ES, MPC, and DRL with MPO. The paper shows that vanilla torque control often leads to a very large variance across random seeds, therefore hurting its mean performance, and robustness against perturbation; while muscle-like control often manifests a much smaller variance and better robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The authors seem very honest by reporting a lot of details in their experiments, and providing code (though I don’t have the chance to take a look at the code)\n\n- Though number of tasks evaluated is not great, and the tasks are somewhat simplistic, the paper covers two models and three control modalities which I believe is a plus.\n\nWeaknesses:\n\n- Looking at the result figures, torque control is not always bad, just has a really large *variance* across random seeds. But the paper does not give much insight on why this is the case - for example, state-of-the-art DRL algorithms seem to be quite random-seed-robust on simple tasks such as hopping or reaching even with torque control. Why is such *robustness across seeds* absent in the experiments here?\n\n- Related to the point above, I’m not sure if the difference in performance actually mainly comes from *simulation & numerical stability*. The authors mentioned that they have to make the time-step quite large (0.01s) to produce noticeable differences between control schemes. So what happened when time-step is in the usual range of stable physics simulations, e.g. 250~1000Hz? (See also Issues, for additional ablations)\n\n- Is using final loss as the main performance metric fair? The losses of several tasks include action squared, which means totally different quantaties for muscle and torque control. I seem to believe the metric should always be action-space-agnostic. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "(Edit 08/28: most of my questions and doubts are decently addressed by the rebuttal and revision. Bumping my score to Weak Accept.)\n\nThis paper is a good benchmarking effort. While I believe it should be eventually published and shared somewhere, I’m not sure if CoRL is the best fit. Also given the weaknesses, the paper might also not reaching the bar of CoRL.\n\nMy main concern, besides doubting whether the CoRL community would care enough about this topic, is what these experiments really tell us. As aforementioned, an alternative conclusion that one could draw from these experiments can be - we need to make simulation more numerically stable, by for example adding springs and dampers to each torque-controlled joint, which is quite standard in most simulation environments, or by not making simulation frequency too low. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Comparison with setting spring & damping in each simulated joint for torque control - could the authors at least discuss why they think this will be insufficient to bring the performance close to muscle-like controls?\n\n- Comparison with using torque rate control: it does not seem very fair to compare muscle control to torque in the sense that muscle control do not change activation level directly. A more fair action space to compare against might be “delta torque”, or “torque rate“, where delta torque at each step is added to the previous executed torque.\n\n- Ablation of using smaller simulation time steps - see weaknesses.\n\n- Metrics are simply losses which include actions - see weaknesses.\n\n- Why is number of random seeds for each task sometimes 5, sometimes 8?"}}, {"review_id": "3MOTMytHek", "reviewer": "Reviewer_XEHZ", "summary": "This paper performs a variety of experiments to investigate the benefits of replacing standard torque actuation with bioinspired muscle dynamics for learning-based control of multi-link rigid body manipulator-style system (i.e., humanoid systems). Across a wide variety of tasks and experiments, it is found that muscle morphologies can increase the learning rate and data efficiency, while also providing higher robustness to disturbances not seen during controller training. The paper also clearly notes that muscle morphologies are limited in that they are much more difficult to implement in both software and hardware, and that existing/classical torque control approaches can likely be tuned to attain similar performance. Overall, however, the results of this paper are interesting and compelling.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\nFirst, this paper is scoped well, and the writing quality is very good. The main text provides all details needed to clearly understand the story and key ideas. The supplementary content and videos provide useful additional details and visualizations.\n\nSecond, the experiments appear well thought out and thorough, and cover many representative anthropomorphic tasks. Due to this variety of experimental design, the numerical results are quite convincing.\n\nThird, the paper is very up-front and clear about limitations and possible alternative interpretations of the numerical study. This makes the overall message more compelling, as it invites readers to further investigate the utility of muscle morphologies for themselves.\n\nWEAKNESSES\n\nThe abstract should highlight the exciting takeaways of the extensive numerical experiments. Currently, it only states that the paper \"investigates\" the differences between torque and muscle actuator morphologies.\n\nIt would be nice to see how the torque and muscle morphologies perform across a range of mass changes, instead of just a single 1.5 or 1.0 kg change. I am inclined to believe that the muscle morphology really does have an advantage here, but I think the experiment can be made stronger.\n\nThe paper has no hardware experiment. This is a very minor weakness, as creating a muscle actuator morphology on hardware would itself be a very good CoRL paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper is very well scoped, and provides compelling evidence in support of a useful hypothesis for task-dependent robot actuator and control design. The experiments are extensive and thorough, and show a general trend of utility for the muscle actuators. Critically, muscle actuators are not presented as a silver bullet; instead, the limitations are discussed very clearly and give the paper good tutorial value for the robotics community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Figure 1: The current illustration is odd because the control input branches off, then is recombined into the final torque. Perhaps it makes more sense to split this into two separate, parallel flowcharts, with the input $u$ and output $\\tau$ the same for both?\n\n14: The first sentence would really benefit from an example of a \"complex anthropomorphic motion.\"\n\n20: This paragraph could benefit from a reference to Fig. 1, to more fully explain the difference between the torque and muscle morphologies.\n\nFig 3: It is a bit difficult to tell the difference between the similarly-colored curves in the top row (but I think this will be hard to modify to make it clearer). It would also help to explain what $c$ is in the figure caption, since that parameter does not clearly appear in any equations, only in a single sentence of the text.\n\n162: The phrase \"vary much stronger\" is a bit weird here. I guess this should be \"vary much more,\" to avoid confusion with \"stronger\" referring to actuator effort.\n\n209: \"dependant\" is spelled incorrectly"}}]

Author rebuttal:

Decision: Accept (poster)