Year: 2021

Abstract:
Modern deep reinforcement learning (DRL) methods allow simulated characters to learn complex skills such as locomotion from scratch. However, without further exploitation of domain-specific knowledge, such as motion capture data, finite state machines or morphological specifications, physics-based locomotion generation with DRL often results in unrealistic motions. One explanation for this is that present RL models do not estimate biomechanical effort; instead, they minimize instantaneous squared joint actuation torques as a proxy for the actual subjective cost of actions. To mitigate this discrepancy in a computationally efficient manner, we propose a method for mapping actuation torques to subjective effort without simulating muscles and their energy expenditure. Our approach is based on the Three Compartment Controller model, in which the relationships of variables such as maximum voluntary joint torques, recovery, and cumulative fatigue are present. We extend this method for sustained symmetric locomotion tasks for deep reinforcement learning using a Normalized Cumulative Fatigue (NCF) model. In summary, in this paper we present the first RL model to use biomechanical cumulative effort for full-body movement generation without the use of any finite state machines, morphological specification or motion capture data. Our results show that the learned policies are more symmetric, periodic and robust compared to methods found in previous literature.

text: Modern deep reinforcement learning (DRL) methods allow simulated characters to learn complex skills such as locomotion from scratch. However, without further exploitation of domain-specific knowledge, such as motion capture data, finite state machines or morphological specifications, physics-based locomotion generation with DRL often results in unrealistic motions. One explanation for this is that present RL models do not estimate biomechanical effort; instead, they minimize instantaneous squared joint actuation torques as a proxy for the actual subjective cost of actions. To mitigate this discrepancy in a computationally efficient manner, we propose a method for mapping actuation torques to subjective effort without simulating muscles and their energy expenditure. Our approach is based on the Three Compartment Controller model, in which the relationships of variables such as maximum voluntary joint torques, recovery, and cumulative fatigue are present. We extend this method for sustained symmetric locomotion tasks for deep reinforcement learning using a Normalized Cumulative Fatigue (NCF) model. In summary, in this paper we present the first RL model to use biomechanical cumulative effort for full-body movement generation without the use of any finite state machines, morphological specification or motion capture data. Our results show that the learned policies are more symmetric, periodic and robust compared to methods found in previous literature.

heading: 1 INTRODUCTION
text: It is a long standing task in computer animation to make characters walk on their own. In this context, Deep Reinfocement Learning (DRL) has become a promising method for automatic generation of movement controls for interactive, physics-based characters. However, in many cases the resulting motions are still not perceived as natural (Schulman et al., 2017). A common approach to mitigate this is to use motion capture or animation data (Peng et al., 2018; Bergamin et al., 2019; Won et al., 2020; Peng et al., 2021). Nevertheless, such approaches are limited to characters and movements to which data is readily available. Furthermore, obtaining qualitatively good data is oftentimes expensive, and many biomechanical constrains that are implicit in captured motions are not preserved during editing and retargeting – which is often required when data is limited. Another method for improving motion quality is to optimize for movement characteristics that shape the motion such as symmetric gait properties (Yu et al., 2018; Abdolhosseini et al., 2019) or minimal energy consumption and task goals. While such methods overcome the need of motion capture data, the absence of biomechanical constraints still may lead to unwanted behaviour and unnatural torque patterns. Another group of methods that have emerged come from bio-mechanical literature, which include musculoskeletal models and other forms of biological constraints. Previous works (Wang et al., 2012; Geijtenbeek et al., 2013; Lee et al., 2014) in this direction have explored biomimetic muscles and tendons to simulate a variety of human and animal motions. However, such musclebased methods are usually computationally expensive, especially under a reinforcement learning framework (Kidziński et al., 2018). In this research we work towards developing a cumulative fatigue reward based on biomechanical literature to account for a computationally efficient way to include motion constraints that are implicit in articulated figures driven by musculotendon units, in the context of locomotion. To improve on quality we further incorporate movement characteristics, such as gait symmetry enforcement methods by Abdolhosseini et al. (2019) and Yu et al. (2018).
Contributions. In this paper we present the first RL model to use biomechanical cumulative effort for full-body movement generation. We derive a Normalized Cumulative Fatigue (NFC) model suitable for reinforcement learning based on the Three Compartment Controller (3CC) model by Xia & Frey Law (2008) and show that both models are equivalent under the assumption of sustained dynamic load conditions but that the 3CC model fails when applied to pre-existing benchmark environments without further hyper-parameter-tuning of the environment itself. Furthermore, the fatigue reward derived from our model more accurately reflects the embodied biomechanical nature of a simulated character when compared to a reward based on instantaneous torque (Yu et al., 2018; Abdolhosseini et al., 2019; Schulman et al., 2017). We apply the cumulative fatigue model to a simulated humanoid for learning sustained symmetric locomotion and show that our method is robust and can generate more relaxed, natural and symmetric locomotion especially in complex environments – without the need of motion capture data, finite state machines or morphological specifications, as well as no further hyper-parameter-tuning of possible pre-existing environments. Additionally, the simplification from 3CC to NCF allows the method to be more easily adaptable to arbitrary characters that may not exhibit biologically accurate properties.

heading: 2 RELATED WORK
text: Recent developments in DRL have seen significant progress in solving high-dimensional continuous control problems. For example, Schulman et al. (2015a) have proposed Trust Region Policy Optimization (TRPO) and show that this method can be used to generate biped locomotion in a 2D planar space. Later, by combining TRPO with Generalized Advantage Estimation, Schulman et al. (2015b) have extended their work for their humanoid locomotion task to three dimensions. Afterwards, they have proposed Proximal Policy Optimization (PPO), which further improves the data efficiency of the algorithm (Schulman et al., 2017). However, the resulting movements oftentimes still look jerky and unnatural. A common way to overcome these issues, is to exploit domain specific knowledge in various forms (Ramamurthy et al., 2019):
Imitation Learning. Oftentimes, reference motion is used in this regard. Peng et al. (2017) introduce a two-level hierarchical controller to generate locomotion: the low-level controller is learned by mimicking the reference locomotion data; the high-level controller is acting as a planner in order to respond to environment changes. However, this method is not capable of highly dynamic motions. Peng et al. (2018) address this issue and achieve significantly more natural-looking motions using imitation learning. More recently, they have extended their method with generative adversarial imitation learning (Peng et al., 2021). Other works in this direction include mimicking various features over a large dataset of movements with RL (Bergamin et al., 2019) or learning a mixture of experts models for various movements (Won et al., 2020). However, all these methods require readily available motion capture data as a prerequisite for training.
Optimizing Movement Characteristics. Instead of using reference data as prior knowledge, another option is to exploit the characteristics of specific types of motions that shall be generated using hand-crafted features. In this regard, Yu et al. (2018) exploit symmetry property of locomotion and propose the mirror symmetry loss. They combine it with energy optimization and add an external force that acts as a virtual assistant to learn symmetric locomotion from scratch. Abdolhosseini et al. (2019) emphasize the core idea behind the mirror symmetry loss, called symmetric policy, and analyze its performance in terms of locomotion symmetry by combining it with DRL in a variety of ways to produce symmetric gaits. They also show that symmetry enforcement methods improve gait symmetry in general, but cannot guarantee a symmetric gait. Furthermore, while such methods overcome the need for motion capture data, the absence of bio-mechanical constraints still leads to unwanted behaviour and less natural torque patterns.
Musculoskeletal Models. A more biologically-accurate approach for movement synthesis involves musculoskeletal models. Previous works (Taga, 1995; Anderson & Pandy, 2001; Geyer & Herr, 2010; Ackermann & van den Bogert, 2012; Ijspeert et al., 2007; Maufroy et al., 2008; Thelen et al., 2003) in biomechanics have developed musculoskeletal models that use biomimetic muscles and tendons to simulate a variety of human and animal motions. Controlling a muscle-based virtual characters has also been explored in computer animation – from upper body movements (Lee & Terzopoulos, 2006; Lee et al., 2009; 2018), to hand manipulation (Tsang et al., 2005; Sueda et al.,
2008), and full-body locomotion (Wang et al., 2012; Geijtenbeek et al., 2013; Lee et al., 2014). However, such muscle-based methods are usually computationally expensive, especially under a reinforcement learning framework (Kidziński et al., 2018). To address this issue, Jiang et al. (2019) have proposed a technique to transform an optimal control problem formulated in the muscle actuation space to an equivalent problem in the joint-actuation space by training the model with control signals obtained from the muscle actuation space. The result shows that as long as the model can reflect the underlying biomechanical properties, it is not necessary to model muscle and tendon details explicitly in order to generate more realistic motions. However, a disadvantage of this work is that they need to learn the mapping from the muscle-based actuation space to the torque-based space using reference data.
Biomechanical Cumulative Fatigue. Muscle fatigue is the failure to maintain the required or expected force (Edwards, 1981). In contrast to instantaneous fatigue, which does not take the endurance time into account, biomechanical fatigue assumes the fatigue to accumulate over time – i.e. the longer a task is done, the more fatiguing it becomes. Muscle fatigue is task-related and can vary across muscles and joints (Xia & Frey Law, 2008; Imbeau et al., 2006; Enoka & Duchateau, 2008; Jang et al., 2017; Frey Law & Avin, 2010; Frey-Law et al., 2012), which partially explains the challenging nature of representing muscle fatigue analytically. In this regard, Liu et al. (2002b) have proposed a motor unit (MU)-based fatigue model which uses three muscle activation states to estimate perceived biomechanical fatigeu: resting, activated and fatigued. The model is able to predict fatigue at static load conditions but fails at submaximal or dynamic conditions. Xia & Frey Law (2008) have proposed a Three-Compartment Controller (3CC) model which improves upon the model of Liu et al. (2002b) for dynamic load conditions by introducing a feed-back controller term between the active and rest MU-states based on torque without the need of explicit modeling of muscle actuators. The 3CC model, as a torque-based model for modeling muscle fatigue and recovery, has already shown effectiveness in motion analysis (Jang et al., 2017) and synthesis (Cheema et al., 2020). The follow-up work (Looft et al., 2018) has been successfully used in upper body motion synthesis under a DLR framework by Cheema et al. (2020) without any motion capture data for mid-air interaction analysis and synthesis. Inspired by their work, we extend it to full-body locomotion generation on arbitrary pre-existing characters and environments.

heading: 3 PRELIMINARIES: FATIGUE MODELING
text: Previous works in computer animation, robotics and standard RL (Yu et al., 2018; Abdolhosseini et al., 2019; Rajamäki & Hämäläinen, 2017; Peng et al., 2017) use instantaneous squared joint torques as a simple measurement to minimize the effort of a given task. However, such a measure is not very biologically accurate as it does not take the endurance time into account and thus the increasing amount of perceived fatigue the longer the task is sustained.
The Three-Compartment Controller (3CC) model (Xia & Frey Law, 2008) is a cumulative fatigue model that assumes motor units (MUs) to be in one of three possible states: 1) active (MA) – MUs
contributing to the task, 2) fatigued (MF ) – MUs without activation, and 3) resting (MR) – inactive MUs not required for the task. These are usually expressed as a percentage of maximum voluntary contraction (%MVC), which can practically be expressed as percentage of maximum voluntary force (%MVF) or torque (%MVT). Additionally, control theory is used to obtain behaviour matching muscle physiology, i.e. active MUs’ force production should decay (fatigue) over time when enough constant force is used. Such a cumulative fatigue model thus gives us a more accurate representation of perceived fatigue. This is expressed by the following system of equations:
∂MA ∂t = C(t)− F ·MA (1a)
∂MR ∂t = −C(t) +R ·MF (1b)
∂MF ∂t = F ·MA −R ·MF (1c)
Where F and R denote the fatigue and recovery coefficients. C(t) is a bounded proportional controller in order to produce the required force, i.e. target load (TL) by controlling the size of MA and MR:
C(t) =  LD · (TL−MA) if MA < TL and MR > TL−MA LD ·MR if MA < TL and MR ≤ TL−MA LR · (TL−MA) if MA ≥ TL
(2)
LD and LR are muscle force development, and relaxation factors, which describe the sensitivity towards the target load (Xia & Frey Law, 2008). The behavior of the 3CC model at different load conditions (TL) can be seen in Fig. 1. If the conditions cannot be matched any longer due to high fatigue and not enough rested MUs available, MA starts to decline (Fig. 1b). From Eq. 2 we can conclude that the target load can be held iff MA+MR ≥ TL. Liu et al. (2002a) have shown that the lower bound of MA +MR is RF+R , i.e. the target load TL can be held indefinitely iff TL ≤ R F+R (Fig. 1c) which results in TL = MA.

heading: 4 METHOD
text: In this section, we present our approach for biomechanical fatigue-based locomotion synthesis. An overview of our system can be seen in Fig. 2. A reward based on cumulative fatigue RF is added in addition to the existing environment rewards RE in benchmark models. If a reward RI based on instantaneous joint squared torques exists, it is replaced with the fatigue reward.

heading: 4.1 CUMULATIVE FATIGUE FOR SUSTAINED LOCOMOTION
text: The 3CC model describes the whole dynamic of motor activation, fatigue and rest. However, we are mostly interested in the fatigue compartment, which is described by Eq. 1c, for constructing the fatigue reward function of the locomotion tasks. For that we first discretize and rearrange Eq. 1c:
M (i+1) F
∆tF = (1−∆tR) ·
M (i) F ∆tF +M (i) A , (i ∈ N ) (3)
With ∆t = ti+1 − ti and M (0)F = 0. (i) and (i+1) are the abbreviations of consecutive timestamps ti and ti+1, respectively. The fatigue coefficient F can be eliminated by substituting f (i) = RF M (i) F , and MA can be set to MA = TL due to formulating locomotion as a sustained task, which results in: f (i+1) = (1−∆tR) · f (i) +∆tR · TL(i) (4) We call f (i) Normalized Cumulative Fatigue (NCF). Since the difference between original cumulative fatigue M (i)F and NCF f
(i) is just a scaling factor RF . The advantage of this is that F does not need to be defined explicitly reducing the hyper-parameter space. Furthermore, since locomotion can be executed for a considerably long time without fatiguing, we assume the extreme case that there are always sufficient non-fatigued motor units (MA +MR) in the 3CC model to produce the desired target load (TL). In this case TL can be set to TL = MA.
To make sure that TL ≤ RF+R , we assume τmax · F+R R ≤ MV T , where τmax is the maximum torque τ allowed at a joint in the simulation environment with TL = ττmax ≥ τ
MV T . This is in contrast to the original 3CC model and Cheema et al. (2020), where TL = ττmax = τ MV T . This change allows us to plug-in our method to pre-existing environments without explicitly having to fine-tune specific MV T s for each joint or check for specific LD, LR and F values for the system to function and hold true, as these are canceled out. In the following we describe how the NCF is used to compute the fatigue reward for RL.

heading: 4.2 FATIGUE REWARD FOR REINFORCEMENT LEARNING
text: In contrast to Cheema et al. (2020) who use the difference between MA and TL for the reward signal, we directly take the reformulation of the fatigue function (NCF) as a reward signal, since we assume TL = MA in our reformulation. Akin to them, we model two fatigue functions for each degree-of-freedom (DoF) roughly corresponding to antagonistic muscle pairs. We adopt this approach for all joints of the simulated character. Given a simulated character with n DoF, the magnitude of torque at axis j in “positive” and “negative” directions are denoted as τj,+ and τj,− (j ∈ [1, n]), respectively. When τj,+ ≥ 0, τj,− = 0, and vice versa. The target load at axis j can be expressed by TLj,+ =
τj,+ τj,max and TLj,− = τj,− τj,max , where τj,max is maximum torque magnitude at
axis j. Note that τj,max does not necessarily equal to MV T at axis j. It is just the maximum torque magnitude that actuator can apply on axis j. Then the NCF at axis j at time ti+1 in “positive” and “negative” directions are
f (i+1) j,+ = (1−∆tRj,+) · f (i) j,+ +∆tRj,+ · TL (i) j,+ (5a)
f (i+1) j,− = (1−∆tRj,−) · f (i) j,− +∆tRj,− · TL (i) j,− (5b)
respectively. The NCF of the simulated character with n DoF at time ti can be expressed by a vector:
f (i) = [f (i) 1,+, f (i) 1,−, f (i) 2,+, f (i) 2,−, · · · , f (i) n,+, f (i) n,−] T (6)
To penalize the use of excessive strength we use the L2 norm to formulate our fatigue reward:
RF = −wF ∥f∥2, (7)
where wF ≥ 0 is the fatigue reward weight. This reward is then used to replace the instantaneous squared joint torque effort of previous work. We set wF = 1 and the recovery coefficient to R = 0.2 for all joints. R was chosen such that MF does not saturate by the end of the training episode. With this we can take advantage of the fatigue accumulation.

heading: 4.3 SYMMETRY ENFORCEMENT USING MIRROR SYMMETRY LOSS
text: We adopt the mirror symmetry loss Lsym first proposed by Yu et al. (2018) to enforce a more symmetric gait:
Lsym(θ) = T∑
t=1
∥πθ(st)−Ma(πθ(Ms(st)))∥2, (8)
T denotes the episode length. Given a state space S and an action space A, a policy πθ : S → A is considered symmetric iff ∀s ∈ S, πθ(Ms(s)) = Ma(πθ(s)), with s ∈ S and a ∈ A. Ms : S → S
and Ma : A → A are here state and action mirroring functions with Ma(a) being the mirror action of action a, and Ms the mirror state of state s, respectively. Yu et al. (2018) optimize this as an auxiliary loss in addition to the default PPO by Schulman et al. (2017):
πθ = argmin θ
LPPO(θ) + wsymLsym(θ), (9)
where wsym is a scalar hyper-parameter used to balance the gait symmetry loss with the standard policy optimization loss which aims to maximize the original objective.

heading: 5 EXPERIMENTS
text: We build our experiments upon the open-source implementation of Abdolhosseini et al. (2019) who use PPO (Schulman et al., 2017) as their base algorithm in addition to their environment rewards (RE in Fig. 2), as well as the symmetry loss proposed by Yu et al. (2018) as an additional symmetry enforcement method (Abdolhosseini et al., 2019). All hyper-parameters, existing rewards and losses are kept the same except that the instantaneous squared torque “low-energy reward” (Yu et al., 2018; Abdolhosseini et al., 2019) (RI in Fig. 2) is replaced with our cumulative fatigue reward RF . Note, that we assume that Abdolhosseini et al. (2019) already did a hyper-parameter search for the symmetry enforcement methods and have published their best results in their comparative symmetry enforcement study (Abdolhosseini et al., 2019), which we build upon. For the comparison against Cheema et al. (2020), we assume F = 10 · R and LD = LR = 10, based on average values from biomechanical literature (Looft et al., 2018; Xia & Frey Law, 2008).

heading: 5.1 ENVIRONMENTS
text: In our experiments we evaluate our method in three different locomotion environments, adopted from Abdolhosseini et al. (2019) (Fig. 3). We use PyBullet as the physics engine and PyTorch as the learning framework.
Walker2D The character contains 6 joints (hips, knees, ankles) with one DoF. Hence, the action space consists of a 6D vector, where each element represents the normalized torque (between -1 and 1 per DoF) applied to a specific joint. The observation space is 22D and consists of root information (root z-coordinate, x and y direction vector, root velocity, roll, and pitch), joint angles, joint angular velocities, and binary foot contact information.
Walker3D The character has 21-DoF corresponding to abdomen (3 DoF), hips (3 DoF), knees (1 DoF), ankles (1 DoF), shoulders (3 DoF), and elbows (1 DoF). The observation space is 52D, consisting of root information (roll, pitch and root velocity), joint angles (21D), joint angular velocities (21D), binary foot contact information (2D), and target position (3D).
Stepper The stepper environment uses the same character as Walker3D, having the same action space as Walker3D, i.e. 21D. The task for the character is to navigate terrain consisting of a sequence of stepping blocks, which are generated consecutively. The character receives the positions of two upcoming blocks as an offset (x,y,z) of the character root frame. The observation space is 55D, similar to Walker3D with additional 3D for the second stepping block target position.

heading: 5.2 EVALUATION METRICS
text: In order to evaluate the effect of our method on locomotion symmetry, we utilize four gait symmetry metrics:
Number of successful models. To determine whether a model was successful or not, we compute if the trained model can walk a certain number of strides, akin to Abdolhosseini et al. (2019). For that we use 20 strides. We train 20 models from 20 different random seeds for each environment and benchmark method. The results can be seen in Table 1. The following metrics are then computed over these successful models:
Normalized Squared Torques (ST). A simple measure for effort is the squared instantaneous torque applied at each time step. The Normalized Squared Torque for one episode is defined as:
ST = 1
T T∑ i=1 1 n n∑ j=1 ∥ τ ij τ ij,max ∥2 (10)
In our evaluation, we evaluate at most 21 episodes. For each episode, the agent needs to at least produce 20 strides and we choose the first 1500 steps. Then we average squared torque cost for each episode as the squared torque cost for the model.
Cumulative Fatigue (CF). The cumulative fatigue is calculated using the fatigue term derived from NCF, which is described in Section 4 Equation 4 and 7 (as a qualitative measure, we omit the weight term −wf ). The Cumulative Fatigue for one episode is defined as:
CF = 1
T T∑ i=1 ∥f (i)∥ (11)
Vectorized Symmetry Index (VSI). A widely-used metric in the biomechanics literature is the Robinson et al. (1987) Symmetry Index (SI), which uses scalar features of the left and right side of the body to determine symmetry. Yu et al. (2018) choose to use the average torques as a feature. However, their metric does not consider torque differences between two sides of the same joint, and neglects the direction of torque. To overcome these drawbacks, we propose VSI, which accepts vectorized features:
V SI = 2∥Xr −Xl∥2 ∥Xl∥2 + ∥Xr∥2 × 100% (12)
With Xr = 1T ∑T t=1(τ r+ t , τ r− t ) and Xl = 1 T ∑T t=1(τ l+ t , τ l− t ), respectively, where τ + t represent the positive torque directions and τ−t the negative at time t. r and l denote the right and the left side, respectively.
Phase-Portrait Index (PPI). VSI only measures the torque symmetry. However, phase-portraits can be used to investigate gait symmetry. A phase-portrait is a 2D scatter plot of which the x and the y axis represent rotation angle and angular velocity of a given joint, usually over a single gait cycle. If the gait is asymmetric the phase-portraits of the two sides will not fully overlap (see Fig. ??). To quantify this information, we adopt the phase-portrait index (PPI) proposed by Abdolhosseini et al. (2019), which is defined as:
PPI = 1
c min s c−1∑ t=0 ∥qrt − qlt+s∥1 + ∥q̇rt − q̇lt+s∥1 (13)
Where c is the length of a gait cycle, qrt and q̇ r t are the normalized right joint position and velocity at time t. Similarly, qlt+s is the normalized left joint position at time t + s mod c, as the elements that are shifted beyond the last position are reintroduced at the beginning.
Spectral Entropy (SE). Spectral entropy is commonly used tool to measure the uncertainty of the frequency of a given signal. In this work, we use spectral entropy to measure the periodicity of locomotion:
SE = − N−1∑ f=0 f ln ( ∥x̂(f)∥2∑N−1 f ′=0 ∥x̂(f ′)∥2 ) (14)
x̂(f) denotes the Fourier transform of a signal. The SE can be used to measure the periodicity of a given signal since, in general, the more periodicity the signal has, the less uncertainty the signal frequency has. We compute the SE of the left and right hip joint rotation angle and average between the two signals.

heading: 5.3 ABLATION STUDIES
text: The following results are computed over these successful models. Note, how in contrast to ours, the 3CC models fail to produce locomotion movements for the majority of models in the more difficult environments, due to them fatiguing too much. This is shown in Tables 3 and 2, where the 3CC models produce the least amount of torque but the number of successful models (Table 1). This is because the set MV T s in the pre-defined environments are set too low for the model to work properly without changing the environments themselves. Due to this we were unable to add the results in the given plots but included them in the tables in the appendix and in the supplementary video for qualitative results. Our overall results show that our proposed model produces more relaxed movements (Fig. 4) in terms of torque actuation than the baseline models, while still finishing the tasks competitively with the baselines or even better in terms of symmetry and periodicity (Fig. 5).

heading: 6 LIMITATIONS AND FUTURE WORK
text: While our cumulative fatigue reward improves on existing benchmark methods in terms of relaxedness, symmetry and periodicity, the 3CC model, which it is based on, is not task- and joint-agnostic (Imbeau et al., 2006; Frey Law & Avin, 2010; Frey-Law et al., 2012) and thus our model can be
improved by optimizing R for each joint based on a sensitivity analysis for various tasks, including locomotion. Additionally, the 3CC model has initially been designed for isometric tasks and is not verified for isokinetic tasks such as locomotion from the viewpoint of biomechanics (Xia & Frey Law, 2008; Looft et al., 2018). To conclude if NCF is valid for locomotion from a biomechanical perspective, a separate study with real participants needs to be conducted. Furthermore, in this paper we mostly focus on symmetry, however extensions to other methods and features have yet to be done.

heading: 7 CONCLUSION
text: In this paper we presented, to the best of our knowledge, the first work to use biomechanical cumulative fatigue for full-body continuous control in complex environments. We derived a Normalized Cumulative Fatigue (NCF) model from the 3CC model and applied it to a humanoid locomotion in pre-existing simulated environments. The agents are trained on a locomotion task. Our results show that our method generally performs better than existing methods in terms of relaxedness, locomotion symmetry and periodicity. Compared to the 3CC model, our NCF model is easier to use, because it does not need additional hyper-parameter tuning of the environment and cancels out hyper-parameters needed in the original 3CC implementation, such as F , LD, LR and MVC for every joint.

heading: A APPENDIX
text: Table 6: Spectral Entropy (median[std]) of Hip Angle: Lower numbers are better.
Method EnvironmentWalker2D Walker3D Stepper PPO (Schulman et al., 2017) 1.911 [0.272] 2.518 [0.695] 1.825 [0.942] PPO+3CC 1.760 [0.253] NaN NaN PPO+NCF 1.697 [0.275] 1.923 [0.652] 2.286 [0.936] SYMM (Yu et al., 2018) 1.537 [0.398] 1.252 [0.361] 0.994 [0.425] SYMM+3CC 1.427 [0.334] NaN 0.561 [0.001] SYMM+NCF (Ours) 1.394 [0.347] 1.262 [0.390] 0.788 [0.277]
Figure 10: Violin plots using the Spectral Entropy (SE) for the hip angle.

Review summaries:
The paper proposes to use a normalized cumulative fatigue (NCF)-based reward to learn symmetric locomotion with Deep RL. The motivation is that most prior work on locomotion synthesis does not estimate cumulative biomechanical effort and only minimizes instantaneous joint torques. The paper derives the NCF reward based on the Three Component Controlller (3CC) fatigue model uses the reward alongside a symmetric loss proposed in prior work. The main contributions claimed in the paper are (1) it is the first work to use biomechanical effort to improve full-body movement generation, and (2) experiments show that the method can generate more natural and symmetric locomotion compared to the baselines.

This paper presents a Reinforcement Learning (RL) model that uses cumulative effort to improve full-body movement generation. The use of a cumulative fatigue reward is a proper way of including biomechanical motion constrains when learning full-body movements.  The proposed model uses a so-called Normalized Cumulative Fatigue (NFC), which is based on the Three Compartment Controller (3CC) model proposed by Xia & Frey Law (2008).  The main contribution of this work is the proposed fatigue reward. 

The paper proposes an improvement to the existing locomotion algorithm by incorporating a bio-inspired fatigue model, the Three Compartment Controller (3CC) model. The fatigue model computes for each motor and each actuation direction a fatigue value that penalizes the motor from exerting high force for a long period of time. The resulting model is used to construct a reward function to replace the traditional energy penalty term in locomotion learning. Results show that the proposed model can achieve some improvements on average compared to methods without the model in terms of motion symmetry.