Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This is a benchmarking paper evaluating muscle-like control against vanilla torque control in simple, anthropomorphic tasks. There are two kinds of muscle models deployed - the demoa model and a much simpler model written in Mujoco. There are three types of control algorithms being tested, optimal control (trajectory optimization) with CMA-ES, MPC, and DRL with MPO. The paper shows that vanilla torque control often leads to a very large variance across random seeds, therefore hurting its mean performance, and robustness against perturbation; while muscle-like control often manifests a much smaller variance and better robustness.
strengths_and_weaknesses: Strengths:

- The authors seem very honest by reporting a lot of details in their experiments, and providing code (though I don’t have the chance to take a look at the code)

- Though number of tasks evaluated is not great, and the tasks are somewhat simplistic, the paper covers two models and three control modalities which I believe is a plus.

Weaknesses:

- Looking at the result figures, torque control is not always bad, just has a really large *variance* across random seeds. But the paper does not give much insight on why this is the case - for example, state-of-the-art DRL algorithms seem to be quite random-seed-robust on simple tasks such as hopping or reaching even with torque control. Why is such *robustness across seeds* absent in the experiments here?

- Related to the point above, I’m not sure if the difference in performance actually mainly comes from *simulation & numerical stability*. The authors mentioned that they have to make the time-step quite large (0.01s) to produce noticeable differences between control schemes. So what happened when time-step is in the usual range of stable physics simulations, e.g. 250~1000Hz? (See also Issues, for additional ablations)

- Is using final loss as the main performance metric fair? The losses of several tasks include action squared, which means totally different quantaties for muscle and torque control. I seem to believe the metric should always be action-space-agnostic. 

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: (Edit 08/28: most of my questions and doubts are decently addressed by the rebuttal and revision. Bumping my score to Weak Accept.)

This paper is a good benchmarking effort. While I believe it should be eventually published and shared somewhere, I’m not sure if CoRL is the best fit. Also given the weaknesses, the paper might also not reaching the bar of CoRL.

My main concern, besides doubting whether the CoRL community would care enough about this topic, is what these experiments really tell us. As aforementioned, an alternative conclusion that one could draw from these experiments can be - we need to make simulation more numerically stable, by for example adding springs and dampers to each torque-controlled joint, which is quite standard in most simulation environments, or by not making simulation frequency too low. 
issues: - Comparison with setting spring & damping in each simulated joint for torque control - could the authors at least discuss why they think this will be insufficient to bring the performance close to muscle-like controls?

- Comparison with using torque rate control: it does not seem very fair to compare muscle control to torque in the sense that muscle control do not change activation level directly. A more fair action space to compare against might be “delta torque”, or “torque rate“, where delta torque at each step is added to the previous executed torque.

- Ablation of using smaller simulation time steps - see weaknesses.

- Metrics are simply losses which include actions - see weaknesses.

- Why is number of random seeds for each task sometimes 5, sometimes 8?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper evaluates the impact of using a muscle actuator model as an action space for learning-based control. Data efficiency, hyperparameter sensitivity, and robustness of the learned behavior are evaluated on a set of simulated robotics tasks. The results indicate some empirical benefits of muscle actuation over direct torque control.
strengths_and_weaknesses: Strengths: The analysis is thorough and the premise of bio-inspired action space design is interesting. I learned about some empirical trends for optimization with MTUs vs torque control.

Weaknesses: The paper should address its relationship to some related prior work. Also, the takeaways for robot learning are ambiguous without comparing to the commonly used PD control action space.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is technically sound, and the question of action space design for learning-based control is interesting. However, uncited previous work seems to have provided a similar set of results. The authors should clarify how their work differs. Also, to make a valuable paper for robot learning practitioners, the authors should add the PD control action space as a baseline.
issues: - A related work is notably omitted: “Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?” (Peng et al. 2017 / https://arxiv.org/pdf/1611.01055.pdf). Here, the authors compared Torque, MTU, position, and velocity action spaces regarding data efficiency and robustness on several RL locomotion tasks. This paper may have substantial differences from the results presented here but the distinction is currently unclear to me. Please add information to elucidate the differences, so that I can better assess the impact and originality of this work.
- It's much more common to use proportional-derivative control than direct torque control in learning motor skills for robots. This is because position targets are a more efficient, robust, action space than torque control. Why not make this comparison in your experiments? It's likely to be more interesting to roboticists than a comparison to torque control.
- In future work, I would also be curious about how model-based RL performs in the learning with muscles setting, given that it potentially needs to learn a more complex forward dynamics model. This could make for some interesting analysis. But this is just a suggestion and by no means needs to be addressed here.
- Another good reference to consider for the impact of environment design on algorithm performance is Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning (Reda et al 2020 /  https://www.cs.ubc.ca/~van/papers/2020-MIG-envdesign/2020-MIG-envdesign.pdf)
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper performs a variety of experiments to investigate the benefits of replacing standard torque actuation with bioinspired muscle dynamics for learning-based control of multi-link rigid body manipulator-style system (i.e., humanoid systems). Across a wide variety of tasks and experiments, it is found that muscle morphologies can increase the learning rate and data efficiency, while also providing higher robustness to disturbances not seen during controller training. The paper also clearly notes that muscle morphologies are limited in that they are much more difficult to implement in both software and hardware, and that existing/classical torque control approaches can likely be tuned to attain similar performance. Overall, however, the results of this paper are interesting and compelling.
strengths_and_weaknesses: STRENGTHS

First, this paper is scoped well, and the writing quality is very good. The main text provides all details needed to clearly understand the story and key ideas. The supplementary content and videos provide useful additional details and visualizations.

Second, the experiments appear well thought out and thorough, and cover many representative anthropomorphic tasks. Due to this variety of experimental design, the numerical results are quite convincing.

Third, the paper is very up-front and clear about limitations and possible alternative interpretations of the numerical study. This makes the overall message more compelling, as it invites readers to further investigate the utility of muscle morphologies for themselves.

WEAKNESSES

The abstract should highlight the exciting takeaways of the extensive numerical experiments. Currently, it only states that the paper "investigates" the differences between torque and muscle actuator morphologies.

It would be nice to see how the torque and muscle morphologies perform across a range of mass changes, instead of just a single 1.5 or 1.0 kg change. I am inclined to believe that the muscle morphology really does have an advantage here, but I think the experiment can be made stronger.

The paper has no hardware experiment. This is a very minor weakness, as creating a muscle actuator morphology on hardware would itself be a very good CoRL paper.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think this paper is very well scoped, and provides compelling evidence in support of a useful hypothesis for task-dependent robot actuator and control design. The experiments are extensive and thorough, and show a general trend of utility for the muscle actuators. Critically, muscle actuators are not presented as a silver bullet; instead, the limitations are discussed very clearly and give the paper good tutorial value for the robotics community.
issues: Figure 1: The current illustration is odd because the control input branches off, then is recombined into the final torque. Perhaps it makes more sense to split this into two separate, parallel flowcharts, with the input $u$ and output $\tau$ the same for both?

14: The first sentence would really benefit from an example of a "complex anthropomorphic motion."

20: This paragraph could benefit from a reference to Fig. 1, to more fully explain the difference between the torque and muscle morphologies.

Fig 3: It is a bit difficult to tell the difference between the similarly-colored curves in the top row (but I think this will be hard to modify to make it clearer). It would also help to explain what $c$ is in the figure caption, since that parameter does not clearly appear in any equations, only in a single sentence of the text.

162: The phrase "vary much stronger" is a bit weird here. I guess this should be "vary much more," to avoid confusion with "stronger" referring to actuator effort.

209: "dependant" is spelled incorrectly
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper argues that humans (and other animals) are able to perform a diverse range of motions *thanks to their muscle’s unique dynamical properties.* In particular, the authors hypothesize that "highly non-linear muscle dynamics” provide inherent stability that makes learning tasks easier. 

To verify this, the authors test if calculating robot torques ($\tau$) from a control signal ($u \in [0,1]$) using a muscle activation model [1] (See *Eq 3*) results in more stable learning, when compared to a standard (linear) idealized model: $\tau = \tau_{max} u$. This experiment is repeated across different tasks, robot morphologies, learning/control algorithms, and simulated environments. The muscle morphology has better data efficiency and is more robust to hyper-parameters changes.


[1] Siebert, T., and C. Rode. "Computational modeling of muscle biomechanics." Computational modelling of biomechanics and biotribology in the musculoskeletal system. Woodhead Publishing, 2014. 173-204.
strengths_and_weaknesses: **Strengths**
* This paper demonstrates how muscle based actuation models can help learn better controllers in a variety of different scenarios. This is a non-trivial insight that could guide further hardware development.

**Weaknesses** 
* It’s unclear if the proposed benefits are actually unique to muscle morphology. The key issue is the baseline used in their study (ideal actuator) is very naive. Most robots already use impedance controllers [2], position controllers [3], and even learned controllers [4] that may provide similar benefits (e.g. “smooth control” constraints) without requiring custom hardware.
* The study doesn’t verify that key muscle properties (e.g. non-linearity) are actually important for better learning. For example, why couldn’t a well tuned low pass filter achieve the same level of smoothness? Thus, some of the claims in the paper (e.g. that non-linearity is a strength) are not properly backed up.
* Finally, this paper is purely demonstrated in simulation, and it’s unclear if its conclusions will generalize to the wild. Indeed, it’s likely that real muscles have properties that are difficult to simulate and/or deviate from the paper’s activation model in significant ways. This limits the study’s potential impact on practical robot learning applications. 

[2] Hogan, Neville. "Impedance control: An approach to manipulation: Part II—Implementation." (1985): 8-16.

[3] Rocco, Paolo. "Stability of PID control for industrial robot arms." IEEE transactions on robotics and automation 12.4 (1996): 606-614.

[4] Martín-Martín, Roberto, et al. "Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks." 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper investigates a fascinating topic and provides interesting insight on where muscle morphology may outperform idealized torque actuators. However, the practical takeaways are muddled because the baseline is very naive, and the simulated results may not be indicative of real world performance/behavior. As a result, I don’t think this paper has quite reached the acceptance threshold, but I will keep my mind open throughout the rebuttal period.
issues: In general, i would ask the authors to respond to my points above. Some specific issues are outlined below: 

* I would encourage the authors to implement stronger baselines (e.g. impedance controller [2]) as part of their study. In other words, investigate if muscles provide any benefit compared to the current state of the art in robotics. 
* The authors should discuss the real world applicability of their findings. What should roboticist take away from this study?
review_title: null

Meta-review: metareview: This paper compares the learning performance of anthropomorphic systems actuated by physiological muscles as well as typical robotic controllers such as PD position control and direct torque input. The simulation results demonstrate that the nonlinear muscle model is superior in terms of data efficiency, sensitivity to hyperparameters, and robustness of the resulting policy in many test cases.

The presented idea is inspiring and potentially has deep implication to machine learning and robot design, while practical application to robot hardware may be questionable at least in the short term. Nevertheless, the paper will likely stimulate active discussions on how the choice of robot actuators may affect robot learning performance.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)