Learning Symmetric Locomotion using Cumulative Fatigue for Reinforcement Learning

Modern deep reinforcement learning ( DRL ) methods allow simulated characters to learn complex skills such as locomotion from scratch . However , without further exploitation of domain-specific knowledge , such as motion capture data , finite state machines or morphological specifications , physics-based locomotion generation with DRL often results in unrealistic motions . One explanation for this is that present RL models do not estimate biomechanical effort ; instead , they minimize instantaneous squared joint actuation torques as a proxy for the actual subjective cost of actions . To mitigate this discrepancy in a computationally efficient manner , we propose a method for mapping actuation torques to subjective effort without simulating muscles and their energy expenditure . Our approach is based on the Three Compartment Controller model , in which the relationships of variables such as maximum voluntary joint torques , recovery , and cumulative fatigue are present . We extend this method for sustained symmetric locomotion tasks for deep reinforcement learning using a Normalized Cumulative Fatigue ( NCF ) model . In summary , in this paper we present the first RL model to use biomechanical cumulative effort for full-body movement generation without the use of any finite state machines , morphological specification or motion capture data . Our results show that the learned policies are more symmetric , periodic and robust compared to methods found in previous literature . 1 INTRODUCTION . It is a long standing task in computer animation to make characters walk on their own . In this context , Deep Reinfocement Learning ( DRL ) has become a promising method for automatic generation of movement controls for interactive , physics-based characters . However , in many cases the resulting motions are still not perceived as natural ( Schulman et al. , 2017 ) . A common approach to mitigate this is to use motion capture or animation data ( Peng et al. , 2018 ; Bergamin et al. , 2019 ; Won et al. , 2020 ; Peng et al. , 2021 ) . Nevertheless , such approaches are limited to characters and movements to which data is readily available . Furthermore , obtaining qualitatively good data is oftentimes expensive , and many biomechanical constrains that are implicit in captured motions are not preserved during editing and retargeting – which is often required when data is limited . Another method for improving motion quality is to optimize for movement characteristics that shape the motion such as symmetric gait properties ( Yu et al. , 2018 ; Abdolhosseini et al. , 2019 ) or minimal energy consumption and task goals . While such methods overcome the need of motion capture data , the absence of biomechanical constraints still may lead to unwanted behaviour and unnatural torque patterns . Another group of methods that have emerged come from bio-mechanical literature , which include musculoskeletal models and other forms of biological constraints . Previous works ( Wang et al. , 2012 ; Geijtenbeek et al. , 2013 ; Lee et al. , 2014 ) in this direction have explored biomimetic muscles and tendons to simulate a variety of human and animal motions . However , such musclebased methods are usually computationally expensive , especially under a reinforcement learning framework ( Kidziński et al. , 2018 ) . In this research we work towards developing a cumulative fatigue reward based on biomechanical literature to account for a computationally efficient way to include motion constraints that are implicit in articulated figures driven by musculotendon units , in the context of locomotion . To improve on quality we further incorporate movement characteristics , such as gait symmetry enforcement methods by Abdolhosseini et al . ( 2019 ) and Yu et al . ( 2018 ) . Contributions . In this paper we present the first RL model to use biomechanical cumulative effort for full-body movement generation . We derive a Normalized Cumulative Fatigue ( NFC ) model suitable for reinforcement learning based on the Three Compartment Controller ( 3CC ) model by Xia & Frey Law ( 2008 ) and show that both models are equivalent under the assumption of sustained dynamic load conditions but that the 3CC model fails when applied to pre-existing benchmark environments without further hyper-parameter-tuning of the environment itself . Furthermore , the fatigue reward derived from our model more accurately reflects the embodied biomechanical nature of a simulated character when compared to a reward based on instantaneous torque ( Yu et al. , 2018 ; Abdolhosseini et al. , 2019 ; Schulman et al. , 2017 ) . We apply the cumulative fatigue model to a simulated humanoid for learning sustained symmetric locomotion and show that our method is robust and can generate more relaxed , natural and symmetric locomotion especially in complex environments – without the need of motion capture data , finite state machines or morphological specifications , as well as no further hyper-parameter-tuning of possible pre-existing environments . Additionally , the simplification from 3CC to NCF allows the method to be more easily adaptable to arbitrary characters that may not exhibit biologically accurate properties . 2 RELATED WORK . Recent developments in DRL have seen significant progress in solving high-dimensional continuous control problems . For example , Schulman et al . ( 2015a ) have proposed Trust Region Policy Optimization ( TRPO ) and show that this method can be used to generate biped locomotion in a 2D planar space . Later , by combining TRPO with Generalized Advantage Estimation , Schulman et al . ( 2015b ) have extended their work for their humanoid locomotion task to three dimensions . Afterwards , they have proposed Proximal Policy Optimization ( PPO ) , which further improves the data efficiency of the algorithm ( Schulman et al. , 2017 ) . However , the resulting movements oftentimes still look jerky and unnatural . A common way to overcome these issues , is to exploit domain specific knowledge in various forms ( Ramamurthy et al. , 2019 ) : Imitation Learning . Oftentimes , reference motion is used in this regard . Peng et al . ( 2017 ) introduce a two-level hierarchical controller to generate locomotion : the low-level controller is learned by mimicking the reference locomotion data ; the high-level controller is acting as a planner in order to respond to environment changes . However , this method is not capable of highly dynamic motions . Peng et al . ( 2018 ) address this issue and achieve significantly more natural-looking motions using imitation learning . More recently , they have extended their method with generative adversarial imitation learning ( Peng et al. , 2021 ) . Other works in this direction include mimicking various features over a large dataset of movements with RL ( Bergamin et al. , 2019 ) or learning a mixture of experts models for various movements ( Won et al. , 2020 ) . However , all these methods require readily available motion capture data as a prerequisite for training . Optimizing Movement Characteristics . Instead of using reference data as prior knowledge , another option is to exploit the characteristics of specific types of motions that shall be generated using hand-crafted features . In this regard , Yu et al . ( 2018 ) exploit symmetry property of locomotion and propose the mirror symmetry loss . They combine it with energy optimization and add an external force that acts as a virtual assistant to learn symmetric locomotion from scratch . Abdolhosseini et al . ( 2019 ) emphasize the core idea behind the mirror symmetry loss , called symmetric policy , and analyze its performance in terms of locomotion symmetry by combining it with DRL in a variety of ways to produce symmetric gaits . They also show that symmetry enforcement methods improve gait symmetry in general , but can not guarantee a symmetric gait . Furthermore , while such methods overcome the need for motion capture data , the absence of bio-mechanical constraints still leads to unwanted behaviour and less natural torque patterns . Musculoskeletal Models . A more biologically-accurate approach for movement synthesis involves musculoskeletal models . Previous works ( Taga , 1995 ; Anderson & Pandy , 2001 ; Geyer & Herr , 2010 ; Ackermann & van den Bogert , 2012 ; Ijspeert et al. , 2007 ; Maufroy et al. , 2008 ; Thelen et al. , 2003 ) in biomechanics have developed musculoskeletal models that use biomimetic muscles and tendons to simulate a variety of human and animal motions . Controlling a muscle-based virtual characters has also been explored in computer animation – from upper body movements ( Lee & Terzopoulos , 2006 ; Lee et al. , 2009 ; 2018 ) , to hand manipulation ( Tsang et al. , 2005 ; Sueda et al. , 2008 ) , and full-body locomotion ( Wang et al. , 2012 ; Geijtenbeek et al. , 2013 ; Lee et al. , 2014 ) . However , such muscle-based methods are usually computationally expensive , especially under a reinforcement learning framework ( Kidziński et al. , 2018 ) . To address this issue , Jiang et al . ( 2019 ) have proposed a technique to transform an optimal control problem formulated in the muscle actuation space to an equivalent problem in the joint-actuation space by training the model with control signals obtained from the muscle actuation space . The result shows that as long as the model can reflect the underlying biomechanical properties , it is not necessary to model muscle and tendon details explicitly in order to generate more realistic motions . However , a disadvantage of this work is that they need to learn the mapping from the muscle-based actuation space to the torque-based space using reference data . Biomechanical Cumulative Fatigue . Muscle fatigue is the failure to maintain the required or expected force ( Edwards , 1981 ) . In contrast to instantaneous fatigue , which does not take the endurance time into account , biomechanical fatigue assumes the fatigue to accumulate over time – i.e . the longer a task is done , the more fatiguing it becomes . Muscle fatigue is task-related and can vary across muscles and joints ( Xia & Frey Law , 2008 ; Imbeau et al. , 2006 ; Enoka & Duchateau , 2008 ; Jang et al. , 2017 ; Frey Law & Avin , 2010 ; Frey-Law et al. , 2012 ) , which partially explains the challenging nature of representing muscle fatigue analytically . In this regard , Liu et al . ( 2002b ) have proposed a motor unit ( MU ) -based fatigue model which uses three muscle activation states to estimate perceived biomechanical fatigeu : resting , activated and fatigued . The model is able to predict fatigue at static load conditions but fails at submaximal or dynamic conditions . Xia & Frey Law ( 2008 ) have proposed a Three-Compartment Controller ( 3CC ) model which improves upon the model of Liu et al . ( 2002b ) for dynamic load conditions by introducing a feed-back controller term between the active and rest MU-states based on torque without the need of explicit modeling of muscle actuators . The 3CC model , as a torque-based model for modeling muscle fatigue and recovery , has already shown effectiveness in motion analysis ( Jang et al. , 2017 ) and synthesis ( Cheema et al. , 2020 ) . The follow-up work ( Looft et al. , 2018 ) has been successfully used in upper body motion synthesis under a DLR framework by Cheema et al . ( 2020 ) without any motion capture data for mid-air interaction analysis and synthesis . Inspired by their work , we extend it to full-body locomotion generation on arbitrary pre-existing characters and environments . 3 PRELIMINARIES : FATIGUE MODELING . Previous works in computer animation , robotics and standard RL ( Yu et al. , 2018 ; Abdolhosseini et al. , 2019 ; Rajamäki & Hämäläinen , 2017 ; Peng et al. , 2017 ) use instantaneous squared joint torques as a simple measurement to minimize the effort of a given task . However , such a measure is not very biologically accurate as it does not take the endurance time into account and thus the increasing amount of perceived fatigue the longer the task is sustained . The Three-Compartment Controller ( 3CC ) model ( Xia & Frey Law , 2008 ) is a cumulative fatigue model that assumes motor units ( MUs ) to be in one of three possible states : 1 ) active ( MA ) – MUs contributing to the task , 2 ) fatigued ( MF ) – MUs without activation , and 3 ) resting ( MR ) – inactive MUs not required for the task . These are usually expressed as a percentage of maximum voluntary contraction ( % MVC ) , which can practically be expressed as percentage of maximum voluntary force ( % MVF ) or torque ( % MVT ) . Additionally , control theory is used to obtain behaviour matching muscle physiology , i.e . active MUs ’ force production should decay ( fatigue ) over time when enough constant force is used . Such a cumulative fatigue model thus gives us a more accurate representation of perceived fatigue . This is expressed by the following system of equations : ∂MA ∂t = C ( t ) − F ·MA ( 1a ) ∂MR ∂t = −C ( t ) +R ·MF ( 1b ) ∂MF ∂t = F ·MA −R ·MF ( 1c ) Where F and R denote the fatigue and recovery coefficients . C ( t ) is a bounded proportional controller in order to produce the required force , i.e . target load ( TL ) by controlling the size of MA and MR : C ( t ) = LD · ( TL−MA ) if MA < TL and MR > TL−MA LD ·MR if MA < TL and MR ≤ TL−MA LR · ( TL−MA ) if MA ≥ TL ( 2 ) LD and LR are muscle force development , and relaxation factors , which describe the sensitivity towards the target load ( Xia & Frey Law , 2008 ) . The behavior of the 3CC model at different load conditions ( TL ) can be seen in Fig . 1 . If the conditions can not be matched any longer due to high fatigue and not enough rested MUs available , MA starts to decline ( Fig . 1b ) . From Eq . 2 we can conclude that the target load can be held iff MA+MR ≥ TL . Liu et al . ( 2002a ) have shown that the lower bound of MA +MR is RF+R , i.e . the target load TL can be held indefinitely iff TL ≤ R F+R ( Fig . 1c ) which results in TL = MA .

Summary: The paper proposes an improvement to the existing locomotion algorithm by incorporating a bio-inspired fatigue model, the Three Compartment Controller (3CC) model. The fatigue model computes for each motor and each actuation direction a fatigue value that penalizes the motor from exerting high force for a long period of time. The resulting model is used to construct a reward function to replace the traditional energy penalty term in locomotion learning. Results show that the proposed model can achieve some improvements on average compared to methods without the model in terms of motion symmetry.