Learning Symmetric Locomotion using Cumulative Fatigue for Reinforcement Learning

Modern deep reinforcement learning ( DRL ) methods allow simulated characters to learn complex skills such as locomotion from scratch . However , without further exploitation of domain-specific knowledge , such as motion capture data , finite state machines or morphological specifications , physics-based locomotion generation with DRL often results in unrealistic motions . One explanation for this is that present RL models do not estimate biomechanical effort ; instead , they minimize instantaneous squared joint actuation torques as a proxy for the actual subjective cost of actions . To mitigate this discrepancy in a computationally efficient manner , we propose a method for mapping actuation torques to subjective effort without simulating muscles and their energy expenditure . Our approach is based on the Three Compartment Controller model , in which the relationships of variables such as maximum voluntary joint torques , recovery , and cumulative fatigue are present . We extend this method for sustained symmetric locomotion tasks for deep reinforcement learning using a Normalized Cumulative Fatigue ( NCF ) model . In summary , in this paper we present the first RL model to use biomechanical cumulative effort for full-body movement generation without the use of any finite state machines , morphological specification or motion capture data . Our results show that the learned policies are more symmetric , periodic and robust compared to methods found in previous literature . 1 INTRODUCTION . It is a long standing task in computer animation to make characters walk on their own . In this context , Deep Reinfocement Learning ( DRL ) has become a promising method for automatic generation of movement controls for interactive , physics-based characters . However , in many cases the resulting motions are still not perceived as natural ( Schulman et al. , 2017 ) . A common approach to mitigate this is to use motion capture or animation data ( Peng et al. , 2018 ; Bergamin et al. , 2019 ; Won et al. , 2020 ; Peng et al. , 2021 ) . Nevertheless , such approaches are limited to characters and movements to which data is readily available . Furthermore , obtaining qualitatively good data is oftentimes expensive , and many biomechanical constrains that are implicit in captured motions are not preserved during editing and retargeting – which is often required when data is limited . Another method for improving motion quality is to optimize for movement characteristics that shape the motion such as symmetric gait properties ( Yu et al. , 2018 ; Abdolhosseini et al. , 2019 ) or minimal energy consumption and task goals . While such methods overcome the need of motion capture data , the absence of biomechanical constraints still may lead to unwanted behaviour and unnatural torque patterns . Another group of methods that have emerged come from bio-mechanical literature , which include musculoskeletal models and other forms of biological constraints . Previous works ( Wang et al. , 2012 ; Geijtenbeek et al. , 2013 ; Lee et al. , 2014 ) in this direction have explored biomimetic muscles and tendons to simulate a variety of human and animal motions . However , such musclebased methods are usually computationally expensive , especially under a reinforcement learning framework ( Kidziński et al. , 2018 ) . In this research we work towards developing a cumulative fatigue reward based on biomechanical literature to account for a computationally efficient way to include motion constraints that are implicit in articulated figures driven by musculotendon units , in the context of locomotion . To improve on quality we further incorporate movement characteristics , such as gait symmetry enforcement methods by Abdolhosseini et al . ( 2019 ) and Yu et al . ( 2018 ) . Contributions . In this paper we present the first RL model to use biomechanical cumulative effort for full-body movement generation . We derive a Normalized Cumulative Fatigue ( NFC ) model suitable for reinforcement learning based on the Three Compartment Controller ( 3CC ) model by Xia & Frey Law ( 2008 ) and show that both models are equivalent under the assumption of sustained dynamic load conditions but that the 3CC model fails when applied to pre-existing benchmark environments without further hyper-parameter-tuning of the environment itself . Furthermore , the fatigue reward derived from our model more accurately reflects the embodied biomechanical nature of a simulated character when compared to a reward based on instantaneous torque ( Yu et al. , 2018 ; Abdolhosseini et al. , 2019 ; Schulman et al. , 2017 ) . We apply the cumulative fatigue model to a simulated humanoid for learning sustained symmetric locomotion and show that our method is robust and can generate more relaxed , natural and symmetric locomotion especially in complex environments – without the need of motion capture data , finite state machines or morphological specifications , as well as no further hyper-parameter-tuning of possible pre-existing environments . Additionally , the simplification from 3CC to NCF allows the method to be more easily adaptable to arbitrary characters that may not exhibit biologically accurate properties . 2 RELATED WORK . Recent developments in DRL have seen significant progress in solving high-dimensional continuous control problems . For example , Schulman et al . ( 2015a ) have proposed Trust Region Policy Optimization ( TRPO ) and show that this method can be used to generate biped locomotion in a 2D planar space . Later , by combining TRPO with Generalized Advantage Estimation , Schulman et al . ( 2015b ) have extended their work for their humanoid locomotion task to three dimensions . Afterwards , they have proposed Proximal Policy Optimization ( PPO ) , which further improves the data efficiency of the algorithm ( Schulman et al. , 2017 ) . However , the resulting movements oftentimes still look jerky and unnatural . A common way to overcome these issues , is to exploit domain specific knowledge in various forms ( Ramamurthy et al. , 2019 ) : Imitation Learning . Oftentimes , reference motion is used in this regard . Peng et al . ( 2017 ) introduce a two-level hierarchical controller to generate locomotion : the low-level controller is learned by mimicking the reference locomotion data ; the high-level controller is acting as a planner in order to respond to environment changes . However , this method is not capable of highly dynamic motions . Peng et al . ( 2018 ) address this issue and achieve significantly more natural-looking motions using imitation learning . More recently , they have extended their method with generative adversarial imitation learning ( Peng et al. , 2021 ) . Other works in this direction include mimicking various features over a large dataset of movements with RL ( Bergamin et al. , 2019 ) or learning a mixture of experts models for various movements ( Won et al. , 2020 ) . However , all these methods require readily available motion capture data as a prerequisite for training . Optimizing Movement Characteristics . Instead of using reference data as prior knowledge , another option is to exploit the characteristics of specific types of motions that shall be generated using hand-crafted features . In this regard , Yu et al . ( 2018 ) exploit symmetry property of locomotion and propose the mirror symmetry loss . They combine it with energy optimization and add an external force that acts as a virtual assistant to learn symmetric locomotion from scratch . Abdolhosseini et al . ( 2019 ) emphasize the core idea behind the mirror symmetry loss , called symmetric policy , and analyze its performance in terms of locomotion symmetry by combining it with DRL in a variety of ways to produce symmetric gaits . They also show that symmetry enforcement methods improve gait symmetry in general , but can not guarantee a symmetric gait . Furthermore , while such methods overcome the need for motion capture data , the absence of bio-mechanical constraints still leads to unwanted behaviour and less natural torque patterns . Musculoskeletal Models . A more biologically-accurate approach for movement synthesis involves musculoskeletal models . Previous works ( Taga , 1995 ; Anderson & Pandy , 2001 ; Geyer & Herr , 2010 ; Ackermann & van den Bogert , 2012 ; Ijspeert et al. , 2007 ; Maufroy et al. , 2008 ; Thelen et al. , 2003 ) in biomechanics have developed musculoskeletal models that use biomimetic muscles and tendons to simulate a variety of human and animal motions . Controlling a muscle-based virtual characters has also been explored in computer animation – from upper body movements ( Lee & Terzopoulos , 2006 ; Lee et al. , 2009 ; 2018 ) , to hand manipulation ( Tsang et al. , 2005 ; Sueda et al. , 2008 ) , and full-body locomotion ( Wang et al. , 2012 ; Geijtenbeek et al. , 2013 ; Lee et al. , 2014 ) . However , such muscle-based methods are usually computationally expensive , especially under a reinforcement learning framework ( Kidziński et al. , 2018 ) . To address this issue , Jiang et al . ( 2019 ) have proposed a technique to transform an optimal control problem formulated in the muscle actuation space to an equivalent problem in the joint-actuation space by training the model with control signals obtained from the muscle actuation space . The result shows that as long as the model can reflect the underlying biomechanical properties , it is not necessary to model muscle and tendon details explicitly in order to generate more realistic motions . However , a disadvantage of this work is that they need to learn the mapping from the muscle-based actuation space to the torque-based space using reference data . Biomechanical Cumulative Fatigue . Muscle fatigue is the failure to maintain the required or expected force ( Edwards , 1981 ) . In contrast to instantaneous fatigue , which does not take the endurance time into account , biomechanical fatigue assumes the fatigue to accumulate over time – i.e . the longer a task is done , the more fatiguing it becomes . Muscle fatigue is task-related and can vary across muscles and joints ( Xia & Frey Law , 2008 ; Imbeau et al. , 2006 ; Enoka & Duchateau , 2008 ; Jang et al. , 2017 ; Frey Law & Avin , 2010 ; Frey-Law et al. , 2012 ) , which partially explains the challenging nature of representing muscle fatigue analytically . In this regard , Liu et al . ( 2002b ) have proposed a motor unit ( MU ) -based fatigue model which uses three muscle activation states to estimate perceived biomechanical fatigeu : resting , activated and fatigued . The model is able to predict fatigue at static load conditions but fails at submaximal or dynamic conditions . Xia & Frey Law ( 2008 ) have proposed a Three-Compartment Controller ( 3CC ) model which improves upon the model of Liu et al . ( 2002b ) for dynamic load conditions by introducing a feed-back controller term between the active and rest MU-states based on torque without the need of explicit modeling of muscle actuators . The 3CC model , as a torque-based model for modeling muscle fatigue and recovery , has already shown effectiveness in motion analysis ( Jang et al. , 2017 ) and synthesis ( Cheema et al. , 2020 ) . The follow-up work ( Looft et al. , 2018 ) has been successfully used in upper body motion synthesis under a DLR framework by Cheema et al . ( 2020 ) without any motion capture data for mid-air interaction analysis and synthesis . Inspired by their work , we extend it to full-body locomotion generation on arbitrary pre-existing characters and environments . 3 PRELIMINARIES : FATIGUE MODELING . Previous works in computer animation , robotics and standard RL ( Yu et al. , 2018 ; Abdolhosseini et al. , 2019 ; Rajamäki & Hämäläinen , 2017 ; Peng et al. , 2017 ) use instantaneous squared joint torques as a simple measurement to minimize the effort of a given task . However , such a measure is not very biologically accurate as it does not take the endurance time into account and thus the increasing amount of perceived fatigue the longer the task is sustained . The Three-Compartment Controller ( 3CC ) model ( Xia & Frey Law , 2008 ) is a cumulative fatigue model that assumes motor units ( MUs ) to be in one of three possible states : 1 ) active ( MA ) – MUs contributing to the task , 2 ) fatigued ( MF ) – MUs without activation , and 3 ) resting ( MR ) – inactive MUs not required for the task . These are usually expressed as a percentage of maximum voluntary contraction ( % MVC ) , which can practically be expressed as percentage of maximum voluntary force ( % MVF ) or torque ( % MVT ) . Additionally , control theory is used to obtain behaviour matching muscle physiology , i.e . active MUs ’ force production should decay ( fatigue ) over time when enough constant force is used . Such a cumulative fatigue model thus gives us a more accurate representation of perceived fatigue . This is expressed by the following system of equations : ∂MA ∂t = C ( t ) − F ·MA ( 1a ) ∂MR ∂t = −C ( t ) +R ·MF ( 1b ) ∂MF ∂t = F ·MA −R ·MF ( 1c ) Where F and R denote the fatigue and recovery coefficients . C ( t ) is a bounded proportional controller in order to produce the required force , i.e . target load ( TL ) by controlling the size of MA and MR : C ( t ) = LD · ( TL−MA ) if MA < TL and MR > TL−MA LD ·MR if MA < TL and MR ≤ TL−MA LR · ( TL−MA ) if MA ≥ TL ( 2 ) LD and LR are muscle force development , and relaxation factors , which describe the sensitivity towards the target load ( Xia & Frey Law , 2008 ) . The behavior of the 3CC model at different load conditions ( TL ) can be seen in Fig . 1 . If the conditions can not be matched any longer due to high fatigue and not enough rested MUs available , MA starts to decline ( Fig . 1b ) . From Eq . 2 we can conclude that the target load can be held iff MA+MR ≥ TL . Liu et al . ( 2002a ) have shown that the lower bound of MA +MR is RF+R , i.e . the target load TL can be held indefinitely iff TL ≤ R F+R ( Fig . 1c ) which results in TL = MA .

Summary: This paper presents a Reinforcement Learning (RL) model that uses cumulative effort to improve full-body movement generation. The use of a cumulative fatigue reward is a proper way of including biomechanical motion constrains when learning full-body movements.  The proposed model uses a so-called Normalized Cumulative Fatigue (NFC), which is based on the Three Compartment Controller (3CC) model proposed by Xia & Frey Law (2008).  The main contribution of this work is the proposed fatigue reward.