Learning Symmetric Locomotion using Cumulative Fatigue for Reinforcement Learning

Abstract: Modern deep reinforcement learning (DRL) methods allow simulated characters to learn complex skills such as locomotion from scratch. However, without further exploitation of domain-specific knowledge, such as motion capture data, finite state machines or morphological specifications, physics-based locomotion generation with DRL often results in unrealistic motions. One explanation for this is that present RL models do not estimate biomechanical effort; instead, they minimize instantaneous squared joint actuation torques as a proxy for the actual subjective cost of actions. To mitigate this discrepancy in a computationally efficient manner, we propose a method for mapping actuation torques to subjective effort without simulating muscles and their energy expenditure. Our approach is based on the Three Compartment Controller model, in which the relationships of variables such as maximum voluntary joint torques, recovery, and cumulative fatigue are present. We extend this method for sustained symmetric locomotion tasks for deep reinforcement learning using a Normalized Cumulative Fatigue (NCF) model.
In summary, in this paper we present the first RL model to use biomechanical cumulative effort for full-body movement generation without the use of any finite state machines, morphological specification or motion capture data. Our results show that the learned policies are more symmetric, periodic and robust compared to methods found in previous literature.

Reviews and author responses:

The paper proposes to use a normalized cumulative fatigue (NCF)-based reward to learn symmetric locomotion with Deep RL. The motivation is that most prior work on locomotion synthesis does not estimate cumulative biomechanical effort and only minimizes instantaneous joint torques. The paper derives the NCF reward based on the Three Component Controlller (3CC) fatigue model uses the reward alongside a symmetric loss proposed in prior work. The main contributions claimed in the paper are (1) it is the first work to use biomechanical effort to improve full-body movement generation, and (2) experiments show that the method can generate more natural and symmetric locomotion compared to the baselines. **Strength:**

- The motivation for using biomechanical effort to improve the realism of synthesized motions seems reasonable, as it mimics how real humans generate their motions.
- The paper derives a simple NCF reward based on the 3CC fatigue model, which has very few hyperparameters and seems quite easy to use as a plug-in.
- The paper employs various metrics to demonstrate the effectiveness of the proposed approach compared against the baselines.
- The paper is generally well-written and easy to follow.

**Weakness:**

- The novelty of the proposed method is limited. The 3CC model has been used by prior work [Cheetma et al, 2020] for motion synthesis inside a Deep RL framework. Although [Cheetma et al, 2020] performs only synthesize upper-body motion, in principle, it can also be applied for full-body motion synthesis. The main difference of this paper is the different design of the fatigue function, but the ideas are quite similar.
- The experiment results are not also very convincing. The method needs to also use the symmetry loss proposed in [Yu et al. 2018] to achieve competitive performance in all the metrics and environments. Without the symmetry loss, the performance of the proposed approach is (sometimes significantly) worse than [Yu et al. 2018]. This makes the method seems a bit incremental. Additionally, there is no comparison with the closest related work [Cheetma et al, 2020]. The paper argued in Sec. 4.2 about the difference with [Cheetma et al, 2020] in reward design but there is no comparison with it. Without experiments, it is difficult to tell which design is better.
- The method has only been tested on simple walking or stepping environments. What about running and other motions? The paper would be much stronger and less incremental if it can show the generation of new motions (even with manually-designed reward) instead of just walking forward.
- Finally, I feel the paper may be more suitable for graphics, animation, or biomechanics conferences than ICLR as its main focus and contribution is on bio-inspired reward design rather than learning. I feel the paper may pique more interest of graphics and animation audiences.

**Additional Comments:**

- How to choose the hyperparameter R in the reward? An ablation study on different R values could help.
- Can the proposed reward also be used to improve imitation learning-based methods such as GAIL [1] or DeepMimic [2]?

[1] Learning human behaviors from motion capture by adversarial imitation. Merel et al. arXiv 2017.
[2] DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. Peng et al. SIGGRAPH 2018. In summary, this paper has limited novelty compared to prior work [Cheetma et al, 2020] and the proposed reward is incremental in its usefulness. The experimental results only show simple walking motions, which is similar to the motion generation ability of prior work [Yu et al. 2018].

------- post rebuttal ----
The authors' responses addressed some of my concerns so I increase my score. Still, I feel more qualitative results in other motion types are needed to showcase a clear advantage of the proposed approach.

 Thank you for the response and additional qualitative results. I have increased my score to 6.

Regarding point 4, I think you can still generate other motions for qualitative comparison even though you may not be able to evaluate them quantitatively.

 Thank you for your follow up questions. 

1. It is true that the episode length varies during training, however we set R according to the set maximum episode length, which is a fixed value. We simply want to make sure that the fatigue is able to accumulate and not saturate within a maximum given episode for training - which was set to 1000 here. We will clarify this further in paper. R in itself is not morphonology independent and should therefore be set for each joint and task separately. We already mentioned this in the limitations. However, the scheme of setting R such that at least the fatigue accumulates during the maximum episode length is morphology independent, and only depends on the training settings.

2. With our new results we have shown that our method is able to do the same tasks with less energy consumption (Figure 4), as well as improve on symmetry in the borderline cases, as was shown in the updated suppl. video. Additionally, unlike the 3CC model, the NCF can be used in predefined environments, where the 3CC-models almost fail completely (Tables 1-6).

 Thanks for the details response! I greatly appreciate the additional experiments that makes the experiment more concrete. It helped address some of my comments but I still have a few questions regarding the paper:

1) The rebuttal mentioned that the hyper-parameter R of the proposed algorithm is chosen based on the episode length. However for locomotion this can be somewhat arbitrary (from a few seconds to tens of seconds). Would that lead to a different R and how would that affect the behavior? Would the same scheme work for different morphologies?

2) As mentioned in the original review, I feel one issue with the work is that it doesn't show a significant advantage over the baseline methods. Here by significant advantage I mean either achieve notably better results on existing tasks, or succeed in tasks that no baseline method could work. As a result, despite that I think the idea seems interesting, the current results haven't pushed the work beyond the borderline region in my opinion. Please let me know if I'm missing anything here like new results that I missed.

 We updated the results for models with 3CC, you can find the detail in the overall response.

 We updated the results for models with 3CC, you can find the detail in the overall response.

 We updated the results for models with 3CC, you can find the detail in the overall response.

 We updated the results for models with 3CC, you can find the detail in the overall response.

 The 20 runs have finished, and these are the updated results for the models with 3CC:



Table 1: Number of models that successfully learned locomotion (could walk 20 or more strides)per method and environment.

| Method | Walker2D | Walker3D | Stepper |
|---|---|---|---|
| PPO+3CC | 20/20 | 0/20 | 0/20 |
| SYMM+3CC | 20/20 | 3/20 | 9/20 |


Table 2: Normalized Squared Torques (median[std]): Lower numbers are better.


| Method | Walker2D | Walker3D | Stepper
|---|---|---|---|
| PPO+3CC | 0.220 [0.010] | NaN | NaN
|SYMM+3CC| 0.194 [0.030] | 0.106 [0.007] | 0.082 [0.011]


Table 3: Cumulative Fatigue (median[std]): Lower numbers are better.

| Method | Walker2D | Walker3D | Stepper
|---|---|---|---|
| PPO+3CC | 0.623 [0.025] | NaN | NaN
|SYMM+3CC| 0.587 [0.043] | 0.924 [0.023] | 0.816 [0.063]

Table 4: Vectorized Symmetry Index (median[std]): Lower numbers are better.

| Method | Walker2D | Walker3D | Stepper
|---|---|---|---|
| PPO+3CC | 25.116 [10.056] | NaN | NaN
| SYMM+3CC | 1.626[1.076] | 10.106[0.628] | 12.401[4.011]


Table 5: Phase-Portrait Index (median[std]): Lower numbers are better.

| Method | Walker2D | Walker3D | Stepper
|---|---|---|---|
| PPO+3CC | 0.719 [0.372] | NaN | NaN
| SYMM+3CC | 0.167 [0.050] | 0.2223 [0.053] | 0.227[0.045]


Table 6: Spectral Entropy (median[std]) of Hip Angle: Lower numbers are better.

| Method | Walker2D | Walker3D | Stepper
|---|---|---|---|
| PPO+3CC | 1.760 [0.253] | NaN | NaN
| SYMM+3CC | 1.427 [0.334] | 1.630 [0.358] | 0.810 [0.532]

 We thank the reviewer for their valuable feedback.

Regarding point 1 and 3: The torque is not modified by the fatigue model, as our main assumption for the NCF model is that the target load can be held forever, meaning that the difference between target load (TL) and active motor units M_A is always zero. From a biomechanical perspective the maximum action bound or torque would need to be adjusted if the active motor units do not equal the target load anymore (see Figure 1b). Doing that would require additional knowledge or hyper-parameter tuning on setting a proper MVT for each joint. If it is set very high, this model becomes similar to the NCF as the target load could probably be held forever. If it is set too low, the model may not even be able to walk because it would be too constraining. 
A strength of our NCF compared to the original 3CC model is that such tuning for the MVT is not necessary, meaning that even fantasy characters can be more easily simulated without having to check what a biomechanically accurate MVT would look like. Another strength of our method is that baseline implementations and environments do not need to be changed. Our model can be simply plugged into existing hyperparameter-tuned implementations and still achieve better results without having to perform a separate hyperparameter search for the already existing rewards and torque bounds. This is what we also did for the experiments in the paper: We assume that the baseline implementations for PPO and SYMM already use the best hyperparameters for the respective models and environments, as we are using the official implementation of Abdulhosseini et al. (2019) (https://github.com/UBCMOCCA/SymmetricRL), who have published an analysis of various existing symmetry enforcement methods (including the one from Yu et al.) in their paper. We do not change their hyperparameter settings, except that we replace their squared torque reward with our proposed cumulative fatigue reward. Same applies for the PPO benchmark. With that we already achieve better results than the benchmark models, without having to search for fitting hyperparameters for the already existing rewards, environments and characters, but only for our own reward.

We have added additional explanations in the text to make our points clearer to the reader. If there are still things unclear, we appreciate any kind of feedback.

Point 2: We tried to reduce this by training 10 additional models. We have furthermore included additional quantitative results (fatigue and instantaneous torque reward measurements) which show that the proposed method produces more relaxed movements, as well as qualitative results in the form of a video to further stress the improvements of the proposed method, which were also requested by reviewer *rhvx*; and the requested Cheema et al. comparison by reviewer *fKKh* (though some models are still training but we are confident that they will not change the overall story). Furthermore, symmetric motions are lower fatiguing, which implies that a more biomechanically accurate fatigue constraint should aid to the symmetry.

Point 4: In the NCF the F is canceled out and we are only interested in the R. The R was chosen such that the M_F does not saturate by 1000 steps, which is the episode length for training, so we can take advantage of the fatigue accumulation. To check for that we ran the NCF model on a trained SYMM model.

 We thank the reviewer for their valuable feedback. Here are our updates regarding the issues mentioned by the reviewer:

1. We have uploaded a video showing our model compared to benchmark models in the supplementary material.
2. That was due to randomness. We tried to mitigate this by training and validating 10 additional models. Results and plots have been updated - some plots and tables needed to be moved to the appendix, since we included the requested additional qualitative metrics. 
3. The Spectral Entropy measures the uncertainty of the periodicity of a function. For locomotion the angle should be a periodic function over time. More periodicity of the angle also indicates more periodicity of the action, which makes the motion more stable and able to run longer. Furthermore, for PPI the pre-condition for this metric is that the motion is (almost) periodic.
4. The requested qualitative measures and a supplementary video have been added.

 We thank the reviewer for their valuable feedback.

Regarding point 1 in the weaknesses: The changes in the design of the fatigue function allow the cumulative fatigue reward to be applied on existing models without having to hyper-parameter-tune the environments, as well as canceling out several hyper-parameters that would need to be tuned in the original 3CC model (F, L_D, L_R, MVT for each joint).

Point 2: We have included two more metrics as suggested by reviewer *rhvx*. The cumulative fatigue and the torque metrics show that our model needs much less torque to achieve similar or better results than the baseline models (Figure 4 and 5).

Point 3: We have included comparisons against the model by Cheema et al. and clarified the differences between our and their method in more detail in Section 4.1 last paragraph. In our case, we cannot use the difference of TL and M_A because our assumption for the NCF is that the target load can be held forever, i.e. TL = M_A, as the difference would always be zero. We show that our normalization method enables us to plug-in the method into existing environments without hyper-parameter tuning of the environment itself, whereas with Cheema et al. the models become too fatigued using the existing maximum torques. This is shown in Table 2 and 3 where the 3CC models produce the least amount of torque but the number of successful models and the quality of the movement is reduced.
Note that some models are still running but we are certain that they will produce similar results.

Point 4: We have added a supplementary video showing that our characters do in fact run. Furthermore, we have chosen a periodic movement because its gait quality can be easily assessed quantitatively, which is not the case for arbitrarily generated movements. Hence, locomotion is used as a proof-of-concept study. We have further added additional quantitative metrics in terms of fatigue and overall “relaxed” movements, as well as a supplementary video showing qualitative results.

Point 5: As ICLR attracts a more general crowd, we were hoping we would perhaps attract both of these crowds (graphics and biology), in addition to the general Reinforcement Learning crowd.

Regarding point 1 in the additional comments: The R was chosen such that the M_F does not saturate by 1000 steps, which is the episode length for training. With this we can take advantage of the fatigue accumulation. To assess this, we ran the NCF model on a trained SYMM model. We added this in the paper.

Point 2: Theoretically, the proposed method should be able to improve those when replacing the squared torque term with the cumulative fatigue. However, it is unclear by how much, and if such changes are visible. It is nevertheless an interesting experiment to consider. In our paper we wanted to emphasize on the fact that we do not require any form of reference data - including data on biomechanical MVTs, such that fantasy characters, as well as already existing implementations - as it is the case for us - could be improved that may not use correct biologically correct maximum voluntary torques or other biological features, such as height, weight, bone density etc., (for more on that please refer to the answer to reviewer *cgvV*).

 We thank the reviewer for their valuable feedback, and their encouraging comments. However, we have a question regarding “Correctness 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.”, as the reviewer is not pointing out any flaws regarding correctness, we have the assumption this was a mistake by the reviewer. If this is not the case, we would appreciate more clarification on that.

 We thank the reviewers for their time, valuable feedback, and reviews. Please find the detailed responses to your comments and questions below. The updates to the paper can be summarized to:

- Computed 10 additional runs for each model to reduce randomness for reviewer *cgvV*
- Added an additional model as benchmark (Cheema et al.) as requested by reviewer *fKKh* (some models are still computing but we are very confident about the results)
- Added two additional metrics and supplementary video with qualitative results as request by reviewer *rhvx*

Which can then be further detailed to:

1. Highlighted in the contributions paragraph in the Introduction what the advantages of our NCF compared to 3CC are.
2. Highlighted the main differences between 3CC and NCF in Section 4.1.
3. Explained why we can’t use the difference between M_A and TL as a reward [Cheema et al.] for our model in Section 4.2.
4. Added how R was chosen at the end of Section 4.2.
5. Highlighted in the beginning of Section 5 that we assume the baseline SYMM models already use the best hyper-parameters, as we are taking a published implementation by Abdolhosseini et al. (2019).
6. Added the 3CC model and reward design by [Cheema et al.] as an additional benchmark in the Experiments Section 5.
7. Increased the number of trained models per method from 10 to 20. 
8. Added additional quantitative measures such as the squared torques (ST) and the cumulative fatigue (CF) in the metrics section.
9. Separated the ablation studies from the quantitative metrics and moved tables and Figures to Section 5.3 Ablation Studies.
10. Added an explanation as to why 3CC compared to NCF fails in Section 5.3 and the Conclusion.
11. Replaced the previous tables for VSI, PPI and SE with plots (Figure 4) and moved the respective tables to the appendix.
12. Moved the violin plots to the appendix.
13. Added tables and plots regarding CF and ST (Figure 5 and Tables 2 and 3) in Section 5.3.
14. Added a supplementary video showing qualitative results.

Below you will further find our responses to the respective reviewers, addressing their specific questions, feedback and concerns.

This paper presents a Reinforcement Learning (RL) model that uses cumulative effort to improve full-body movement generation. The use of a cumulative fatigue reward is a proper way of including biomechanical motion constrains when learning full-body movements.

The proposed model uses a so-called Normalized Cumulative Fatigue (NFC), which is based on the Three Compartment Controller (3CC) model proposed by Xia & Frey Law (2008).

The main contribution of this work is the proposed fatigue reward.
 1. The paper is well written and very clear. The analysis of the state of the art is correct and no relevant references are missing.

2. The main contribution of the paper is the so-called Normalized Cumulative Fatigue (NFC), which is based on the Three Compartment Controller (3CC) model proposed by Xia & Frey Law (2008).

3. Results

In the reported experiments (Walker2D, Walker3D and Stepper) authors build upon the open-source implementation of Abdolhosseini et al. (2019), and include the proposed NFC and the symmetry loss proposed by (Yu et al., 2018) as a symmetry enforcement method. In these experiments: PPO denotes the Abdolhosseini original method, PPO+Fatigue denotes PPO plus NFC, SYMM denotes PPO plus the symmetry loss (Yu et al., 2018), and SYMM+Fatigue is SYMM plus NFC. 

The use of the fatigue reward improves the learning of the different movements, when compared to the use of instantaneous torque reward. The amount of improvement depends on what index is measured. For instance, when the number of successfully learned locomotion models is measured, the improvement is marginal, in fact best results are obtained by SYMM. When the Vectorised symmetry index is used, best method is SYMM+Fatigue, and the amount of improvement in some cases is marginal (Walker2D and Walker3D), and in other more substantial (Stepper).


 The proposed method is based on a previous work (Abdolhosseini et al. (2019)), which is modified by including the fatigue reward and the symmetry loss proposed by (Yu et al., 2018). The fatigue reward is based on the Three Compartment Controller (3CC) model (proposed by Xia & Frey Law (2008)). So, I am not very clear about how to assess the novelty of this work.

Regarding the obtained results, the use of both mechanisms (fatigue+symmetry) improve the learning results in the reported experiments. This improvement is some cases is marginal and in other substantial.

The paper proposes an improvement to the existing locomotion algorithm by incorporating a bio-inspired fatigue model, the Three Compartment Controller (3CC) model. The fatigue model computes for each motor and each actuation direction a fatigue value that penalizes the motor from exerting high force for a long period of time. The resulting model is used to construct a reward function to replace the traditional energy penalty term in locomotion learning. Results show that the proposed model can achieve some improvements on average compared to methods without the model in terms of motion symmetry.
 The paper introduces a method for training symmetric locomotion controllers for bipedal characters by leveraging a fatigue model. The idea of using bio-inspired model to model fatigues in the motors make a lot of sense and the proposed method seems reasonable. However, I do have a few questions and concerns regarding the proposed work:

- Is torque modified by the fatigue model, or it’s only used to compute the fatigue reward? If the former, what is the impact of the reward vs modifying the actions? If the latter, would modifying the action (maximum torque or action bound) based on the fatigue model also achieve good performance?

- The results indeed show improved performance on average. However, the results are not statistically significant with the large standard deviation in the metrics. Also, it’s hard to draw the conclusion that the fatigue reward would lead to more symmetric results (the walker results actually show worse symmetry with PPO only).

- For the SYMM baseline, it might be worth doing some hyper-parameter search for the coefficient of the energy term. If it can be shown that such a procedure couldn’t produce good result it would further stress the advantage of the proposed method.

- Some ablation on how the parameters of the proposed fatigue model (R and F) would impact the results is interesting to see.
 In general I think the idea of using a generic fatigue model to encourage more natural locomotion animation is a neat one. The authors proposed a concrete algorithm. However, the current result doesn’t seem to generate a significantly different result (judging from the metric statistics). If more analysis/results could be shown to demonstrate the advantage of the proposed method I think it would be a good and interesting contribution to the field.

This paper proposes to replace the commonly used torque square penalty for RL locomotion task with a fatigue-based reward that is more biologically plausible. Result demonstrates that the new reward is able to produce more symmetry motion compared to baseline on various benchmarks. Pro:
1. A novel biologically plausible energy reward for learning locomotion tasks.
2. Proposed reward is able to achieve more symmetry motion compared to baseline based on many metrics.

Issues:
1. There is no qualitative result, i.e., a video showing motions from different methods.
2. The percentage of successful models is smaller compared to baseline, it will be nice if the authors can comment on the reason behind it.
3. I am not sure why Spectral Entropy indicates the robustness of the model.
4. A lot of metrics are introduced to measure the symmetry of the model. It will be nice to provide other qualitative results. For example, what are their performance in terms of torque square, the proposed cumulative fatigue, and other task rewards? This paper introduces a biologically plausible reward based on cumulative fatigue. Many efforts are put into quantifying the advantage of the proposed method in terms of generating symmetry gait. It will be a nice contribution if the paper can also provide other qualitative results mentioned in the main review as well as clarify additional concerns raised. 

Meta-review: This paper investigated using bio-inspired cumulative fatigue model to improve bipedal locomotion via deep RL. The proposed method marginally improved bipedal locomotion behavior. The size of the experiments should be improved, together with generalization to other symmetric walkers. AC agrees with the reviewers that the empirical performance is not significant enough. The paper may fit the scope of a bipedal locomotion journal/conference better than ICLR.

Decision: Reject