Title: LEARNING LOCOMOTION SKILLS USING DEEPRL: DOES THE CHOICE OF ACTION SPACE MATTER?

Abstract: The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gaitcycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.

1. INTRODUCTION:

The introduction of deep learning models to reinforcement learning (RL) has enabled policies to operate directly on high-dimensional, low-level state features. As a result, deep reinforcement learning (DeepRL) has demonstrated impressive capabilities, such as developing control policies that can map from input image pixels to output joint torques (Lillicrap et al., 2015) . However, the quality and robustness often falls short of what has been achieved with hand-crafted action abstractions, e.g., Coros et al. (2011) ; Geijtenbeek et al. (2013) . While much is known about the learning of state representations, the choice of action parameterization is a design decision whose impact is not yet well understood.

Joint torques can be thought of as the most basic and generic representation for driving the movement of articulated figures, given that muscles and other actuation models eventually result in joint torques. However this ignores the intrinsic embodied nature of biological systems, particularly the synergy between control and biomechanics. Passive-dynamics, such as elasticity and damping from muscles and tendons, play an integral role in shaping motions: they provide mechanisms for energy storage, and mechanical impedance which generates instantaneous feedback without requiring any explicit computation. Loeb coins the term preflexes (Loeb, 1995) to describe these effects, and their impact on motion control has been described as providing intelligence by mechanics (Blickhan et al., 2007) . This can also be thought of as a kind of partitioning of the computations between the control and physical system.

In this paper we explore the impact of four different actuation models on learning to control dynamic articulated figure locomotion: (1) torques (Tor); (2) activations for musculotendon units (MTU); (3) target joint angles for proportional-derivative controllers (PD); and (4) target joint velocities (Vel). Because Deep RL methods are capable of learning control policies for all these models, it now becomes possible to directly assess how the choice of actuation model affects the learning difficulty. We also assess the learned policies with respect to robustness, motion quality, and policy query rates. We show that action spaces which incorporate local feedback can significantly improve learning speed and performance, while still preserving the generality afforded by torque-level control. Such parameterizations also allow for more complex body structures and subjective improvements in motion quality.

Our specific contributions are: (1) We introduce a DeepRL framework for motion imitation tasks;

(2) We evaluate the impact of four different actuation models on learned control policies according to four criteria; and (3) We propose an optimization approach that combines policy learning and actuator optimization, allowing neural networks to effective control complex muscle models.

2. BACKGROUND:

Our task will be structured as a standard reinforcement problem where an agent interacts with its environment according to a policy in order to maximize a reward signal. The policy π(s, a) = p(a|s) represents the conditional probability density function of selecting action a ∈ A in state s ∈ S. At each control step t, the agent observes a state s t and samples an action a t from π. The environment in turn responds with a scalar reward r t , and a new state s t = s t+1 sampled from its dynamics p(s |s, a). For a parameterized policy π θ (s, a), the goal of the agent is learn the parameters θ which maximizes the expected cumulative reward

J(π θ ) = E T t=0 γ t r t π θ

with γ ∈ [0, 1] as the discount factor, and T as the horizon. The gradient of the expected reward θ J(π θ ) can be determined according to the policy gradient theorem (Sutton et al., 2001) , which provides a direction of improvement to adjust the policy parameters θ.

θ J(π θ ) = S d θ (s) A θ log(π θ (s, a))A(s, a)da ds

where d θ (s) = S T t=0 γ t p 0 (s 0 )p(s 0 → s|t, π θ )ds 0 is the discounted state distribution, where p 0 (s) represents the initial state distribution, and p(s 0 → s|t, π θ ) models the likelihood of reaching state s by starting at s 0 and following the policy π θ (s, a) for t steps (Silver et al., 2014) . A(s, a) represents a generalized advantage function. The choice of advantage function gives rise to a family of policy gradient algorithms, but in this work, we will focus on the one-step temporal difference advantage function (Schulman et al., 2015) A(s t , a t ) = r t + γV (s t ) -V (s t ) where V (s) = E T t=0 γ t r t s 0 = s, π θ is the state-value function, and can be defined recursively via the Bellman equation

V (s t ) = E rt,s t [r t + γV (s t )|s t , π θ ]

A parameterized value function V φ (s), with parameters φ, can be learned iteratively in a manner similar to Q-Learning by minimizing the Bellman loss,

L(φ) = E st,rt,s t 1 2 (y t -V φ (s t )) 2 , y t = r t + γV φ (s t )

π θ and V φ can be trained in tandem using an actor-critic framework (Konda & Tsitsiklis, 2000) .

In this work, each policy will be represented as a gaussian distribution with a parameterized mean µ θ (s) and fixed covariance matrix Σ = diag{σ 2 i }, where σ i is manually specified for each action parameter. Actions can be sampled from the distribution by applying gaussian noise to the mean action a t = µ θ (s t ) + N (0, Σ) The corresponding policy gradient will assume the form

θ J(π θ ) = S d θ (s) A θ µ θ (s)Σ -1 (a -µ θ (s)) A(s, a)da ds

which can be interpreted as shifting the mean of the action distribution towards actions that lead to higher than expected rewards, while moving away from actions that lead to lower than expected rewards.

3. TASK REPRESENTATION:



3.1. REFERENCE MOTION:

In our task, the goal of a policy is to imitate a given reference motion {q * t } which consists of a sequence of kinematic poses q * t in reduced coordinates. The reference velocity q * t at a given time t is approximated by finite-difference q * t ≈ q * t+ t -q * t t

. Reference motions are generated via either using a recorded simulation result from a preexisting controller ("Sim"), or via manually-authored keyframes. Since hand-crafted reference motions may not be physically realizable, the goal is to closely reproduce a motion while satisfying physical constraints.

3.2. STATES:

To define the state of the agent, a feature transformation Φ(q, q) is used to extract a set of features from the reduced-coordinate pose q and velocity q. The features consist of the height of the root (pelvis) from the ground, the position of each link with respect to the root, and the center of mass velocity of each link. When training a policy to imitate a cyclic reference motion {q * t }, knowledge of the motion phase can help simplify learning. Therefore, we augment the state features with a set of target features Φ(q * t , qt * ), resulting in a combined state represented by s t = (Φ(q t , qt ), Φ(q * t , qt * )). Similar results can also be achieved by providing a single motion phase variable as a state feature, as we show in Figure 15 (supplemental material).

3.3. ACTIONS:

We train separate policies for each of the four actuation models, as described below. Each actuation model also has related actuation parameters, such as feedback gains for PD-controllers and musculotendon properties for MTUs. These parameters can be manually specified, as we do for the PD and Vel models, or they can be optimized for the task at hand, as for the MTU models. Table 1 provides a list of actuator parameters for each actuation model.

Target Joint Angles (PD): Each action represents a set of target angles q, where qi specifies the target angles for joint i. q is applied to PD-controllers which compute torques according to τ i = k i p (q i -q i ) + k i d ( qi -qi ), where qi = 0, and k i p and k i d are manually-specified gains.

Target Joint Velocities (Vel): Each action specifies a set of target velocities q which are used to compute torques according to τ i = k i d ( qi -qi ), where the gains k i d are specified to be the same as those used for target angles.

Torques (Tor): Each action directly specifies torques for every joint, and constant torques are applied for the duration of a control step. Due to torque limits, actions are bounded by manually specified limits for each joint. Unlike the other actuation models, the torque model does not require additional actuator parameters, and can thus be regarded as requiring the least amount of domain knowledge. Torque limits are excluded from the actuator parameter set as they are common for all parameterizations.

Muscle Activations (MTU): Each action specifies activations for a set of musculotendon units (MTU). Detailed modeling and implementation information are available in Wang et al. (2012) . Each MTU is modeled as a contractile element (CE) attached to a serial elastic element (SE) and parallel elastic element (PE). The force exerted by the MTU can be calculated according to F M T U = F SE = F CE + F P E . Both F SE and F P E are modeled as passive springs, while F CE is actively controlled according to

F CE = a M T U F 0 f l (l CE )f v (v CE )

, with a M T U being the muscle activation, F 0 the maximum isometric force, l CE and v CE being the length and velocity of the contractile element. The functions f l (l CE ) and f v (v CE ) represent the force-length and force-velocity relationships, modeling the variations in the maximum force that can be exerted by a muscle as a function of its length and contraction velocity. Analytic forms are available in Geyer et al. (2003) . Activations are bounded between [0, 1]. The length of each contractile element l CE are included as state features. To simplify control and reduce the number of internal state parameters per MTU, the policies directly control muscle activations instead of indirectly through excitations (Wang et al., 2012) .

Actuation Model:

Actuator Parameters Target Joint Angles (PD) proportional gains k p , derivative gains k d Target Joint Velocities (Vel) derivative gains k d Torques (Tor) none Muscle Activations (MTU) optimal contractile element length, serial elastic element rest length, maximum isometric force, pennation, moment arm, maximum moment arm joint orientation, rest joint orientation.

Table 1 : Actuation models and their respective actuator parameters.

3.4. REWARD:

The reward function consists of a weighted sum of terms that encourage the policy to track a reference motion. r = w pose r pose + w vel r vel + w end r end + w root r root + w com r com w pose = 0.5, w vel = 0.05, w end = 0.15, w root = 0.1, w com = 0.2 Details of each term are available in the supplemental material. r pose penalizes deviation of the character pose from the reference pose, and r vel penalizes deviation of the joint velocities. r end and r root accounts for the position error of the end-effectors and root. r com penalizes deviations in the center of mass velocity from that of the reference motion.

3.5. INITIAL STATE DISTRIBUTION:

We design the initial state distribution, p 0 (s), to sample states uniformly along the reference trajectory. At the start of each episode, q * and q * are sampled from the reference trajectory, and used to initialize the pose and velocity of the agent. This helps guide the agent to explore states near the target trajectory.

4. ACTOR-CRITIC LEARNING ALGORITHM:

Instead of directly using the temporal difference advantage function, we adapt a positive temporal difference (PTD) update as proposed by Van Hasselt (2012) .

A(s, a) = I [δ > 0] = 1, δ > 0 0, otherwise δ = r + γV (s ) -V (s)

Unlike more conventional policy gradient methods, PTD is less sensitive to the scale of the advantage function and avoids instabilities that can result from negative TD updates. For a Gaussian policy, a negative TD update moves the mean of the distribution away from an observed action, effectively shifting the mean towards an unknown action that may be no better than the current mean action (Van Hasselt, 2012) . In expectation, these updates converges to the true policy gradient, but for stochastic estimates of the policy gradient, these updates can cause the agent to adopt undesirable behaviours which affect subsequent experiences collected by the agent. Furthermore, we incorporate experience replay, which has been demonstrated to improve stability when training neural network policies with Q-learning in discrete action spaces. Experience replay often requires off-policy methods, such as importance weighting, to account for differences between the policy being trained and the behavior policy used to generate experiences (WawrzyńSki & Tanwani, 2013) . However, we have not found importance weighting to be beneficial for PTD.

Stochastic policies are used during training for exploration, while deterministic policy are deployed for evaluation at runtime. The choice between a stochastic and deterministic policy can be specified by the addition of a binary indicator variable λ ∈ [0, 1]

a t = µ θ (s t ) + λN (0, Σ)

where λ = 1 corresponds to a stochastic policy with exploration noise, and λ = 0 corresponds to a deterministic policy that always selects the mean of the distribution. Noise from a stochastic policy will result in a state distribution that differs from that of the deterministic policy at runtime. To imitate this discrepancy, we incorporate -greedy exploration in addition to the original Gaussian exploration. During training, λ is determined by a Bernoulli random variable λ ∼ Ber( ), where λ = 1 with probability ∈ [0, 1]. The exploration rate is annealed linearly from 1 to 0.2 over 500k iterations, which slowly adjusts the state distribution encountered during training to better resemble the distribution at runtime. Since the policy gradient is defined for stochastic policies, only tuples recorded with exploration noise (i.e. λ = 1) can be used to update the actor, while the critic can be updated using all tuples.

Training proceeds episodically, where the initial state of each episode is sampled from p 0 (s), and the episode duration is drawn from an exponential distribution with a mean of 2s. To discourage falling, an episode will also terminate if any part of the character's trunk makes contact with the ground for an extended period of time, leaving the agent with zero reward for all subsequent steps. Algorithm 1 in the supplemental material summarizes the complete learning process.

MTU Actuator Optimization: Actuation models such as MTUs are defined by further parameters whose values impact performance (Geijtenbeek et al., 2013) . Geyer et al. (2003) uses existing anatomical estimates for humans to determine MTU parameters, but such data is not be available for more arbitrary creatures. Alternatively, Geijtenbeek et al. ( 2013) uses covariance matrix adaptation (CMA), a derivative-free evolutionary search strategy, to simultaneously optimize MTU and policy parameters. This approach is limited to policies with reasonably low dimensional parameter spaces, and is thus ill-suited for neural network models with hundreds of thousands of parameters. To avoid manual-tuning of actuator parameters, we propose a heuristic approach that alternates between policy learning and actuator optimization, as detailed in the supplemental material. 

5. RESULTS:

The motions are best seen in the supplemental video https://youtu.be/L3vDo3nLI98. We evaluate the action parameterizations by training policies for a simulated 2D biped, dog, and raptor as shown in Figure 1 . Depending on the agent and the actuation model, our systems have 58-214 state dimensions, 6-44 action dimensions, and 0-282 actuator parameters, as summarized in Table 3 (supplemental materials). The MTU models have at least double the number of action parameters because they come in antagonistic pairs. As well, additional MTUs are used for the legs to more accurately reflect bipedal biomechanics. This includes MTUs that span multiple joints.

Each policy is represented by a three layer neural network, as illustrated in Figure 8 (supplemental material) with 512 and 256 fully-connected units, followed by a linear output layer where the number of output units vary according to the number of action parameters for each character and actuation model. ReLU activation functions are used for both hidden layers. Each network has approximately 200k parameters. The value function is represented by a similar network, except having a single linear output unit. The policies are queried at 60Hz for a control step of about 0.0167s. Each network is randomly initialized and trained for about 1 million iterations, requiring 32 million tuples, the equivalent of approximately 6 days of simulated time. Each policy requires about 10 hours for the biped, and 20 hours for the raptor and dog on an 8-core Intel Xeon E5-2687W. Figure 5 illustrates the performance improvement per pass. Figure 6 compares the performance of MTUs before and after optimization. For most examples, the optimized actuator parameters significantly improve learning speed and final performance. For the sake of comparison, after a set of actuator parameters has been optimized, a new policy is retrained with the new actuator parameters and its performance compared to the other actuation models.

Policy Performance and Learning Speed: Figure 2 shows learning curves for the policies and the performance of the final policies are summarized in Table 4 . Performance is evaluated using the normalized cumulative reward (NCR), calculated from the average cumulative reward over 32 episodes with lengths of 10s, and normalized by the maximum and minimum cumulative reward possible for each episode. No discounting is applied when calculating the NCR. The initial state of each episode is sampled from the reference motion according to p(s 0 ). To compare learning speeds, we use the normalized area under each learning curve (AUC) as a proxy for the learning speed of a particular actuation model, where 0 represents the worst possible performance and no progress during training, and 1 represents the best possible performance without requiring training.

PD performs well across all examples, achieving comparable-to-the-best performance for all motions. PD also learns faster than the other parameterizations for 5 of the 7 motions. The final performance of Tor is among the poorest for all the motions. Differences in performance appear more pronounced as characters become more complex. For the simple 7-link biped, most parameterizations achieve similar performance. However, for the more complex dog and raptor, the performance of Tor policies deteriorate with respect to other policies such as PD and Vel. MTU policies often exhibited the slowest learning speed, which may be a consequence of the higher dimensional action spaces, i.e., requiring antagonistic muscle pairs, and complex muscle dynamics. Nonetheless, once optimized, the MTU policies produce more natural motions and responsive behaviors as compared to other parameterizations. We note that the naturalness of motions is not well captured by the reward, since it primarily gauges similarity to the reference motion, which may not be representative of natural responses when perturbed from the nominal trajectory. A sensitivity analysis of the policies' performance to variations in network architecture and hyperparameters are available in the supplemental material.

Policy Robustness: To evaluate robustness, we recorded the NCR achieved by each policy when subjected to external perturbations. The perturbations assume the form of random forces applied to the trunk of the characters. Figure 3 illustrates the performance of the policies when subjected to perturbations of different magnitudes. The magnitude of the forces are constant, but direction varies randomly. Each force is applied for 0.1 to 0.4s, with 1 to 4s between each perturbation. Performance is estimated using the average over 128 episodes of length 20s each. For the biped walk, the Tor policy is significantly less robust than those for the other types of actions, while the MTU policy is the least robust for the raptor run. Overall, the PD policies are among the most robust for all the motions. In addition to external forces, we also evaluate robustness over randomly generated terrain consisting of bumps with varying heights and slopes with varying steepness. We evaluate the performance on irregular terrain (Figure 12 , supplemental material). There are few discernible patterns for this test. The Vel and MTU policies are significantly worse than the Tor and PD policies for the dog bound on the bumpy terrain. The unnatural jittery behavior of the dog Tor policy proves to be surprisingly robust for this scenario. We suspect that the behavior prevents the trunk from contacting the ground for extended periods for time, and thereby escaping our system's fall detection.

Figure 3 : Performance when subjected to random perturbation forces of different magnitudes.

Query Rate: Figure 4 compares the performance of different parameterizations for different policy query rates. Separate policies are trained with queries of 15Hz, 30Hz, 60Hz, and 120Hz. Actuation models that incorporate low-level feedback such as PD and Vel, appear to cope more effectively to lower query rates, while the Tor degrades more rapidly at lower query rates. It is not yet obvious to us why MTU policies appear to perform better at lower query rates and worse at higher rates. Lastly, Figure 14 shows the policy outputs as a function of time for the four actuation models, for a particular joint, as well as showing the resulting joint torque. Interestingly, the MTU action is visibly smoother than the other actions and results in joint torques profiles that are smoother than those seen for PD and Vel. Recent work on deterministic policy gradients (Lillicrap et al., 2015) and on RL benchmarks, e.g., OpenAI Gym, generally use joint torques as the action space, as do the test suites in recent work (Schulman et al., 2015) on using generalized advantage estimation. Other recent work uses: the PR2 effort control interface as a proxy for torque control (Levine et al., 2015) ; joint velocities (Gu et al., 2016) ; velocities under an implicit control policy (Mordatch et al., 2015) ; or provide abstract actions (Hausknecht & Stone, 2015) . Our learning procedures are based on prior work using actorcritic approaches with positive temporal difference updates (Van Hasselt, 2012) .

Work in biomechanics has long recognized the embodied nature of the control problem and the view that musculotendon systems provide "preflexes" (Loeb, 1995) that effectively provide a form intelligence by mechanics (Blickhan et al., 2007) , as well as allowing for energy storage. The control strategies for physics-based character simulations in computer animation also use all the forms of actuation that we evaluate in this paper. Representative examples include quadratic programs that solve for joint torques (de Lasa et al., 2010) , joint velocities for skilled bicycle stunts (Tan et al., 2014) , muscle models for locomotion (Wang et al., 2012; Geijtenbeek et al., 2013) , mixed use of feed-forward torques and joint target angles (Coros et al., 2011) , and joint target angles computed by learned linear (time-indexed) feedback strategies (Liu et al., 2016) . Lastly, control methods in robotics use a mix of actuation types, including direct-drive torques (or their virtualized equivalents), series elastic actuators, PD control, and velocity control. These methods often rely heavily on modelbased solutions and thus we do not describe these in further detail here.

7. CONCLUSIONS:

Our experiments suggest that action parameterizations that include basic local feedback, such as PD target angles, MTU activations, or target velocities, can improve policy performance and learning speed across different motions and character morphologies. Such models more accurately reflect the embodied nature of control in biomechanical systems, and the role of mechanical components in shaping the overall dynamics of motions and their control. The difference between low-level and high-level action parameterizations grow with the complexity of the characters, with high-level parameterizations scaling more gracefully to complex characters. As a caveat, there may well be tasks, such as impedance control, where lower-level action parameterizations such as Tor may prove advantageous. We believe that no single action parameterization will be the best for all problems. However, since objectives for motion control problems are often naturally expressed in terms of kinematic properties, higher-level actions such as target joint angles and velocities may be effective for a wide variety of motion control problems. We hope that our work will help open discussions around the choice of action parameterizations.

Our results have only been demonstrated on planar articulated figure simulations; the extension to 3D currently remains as future work. Furthermore, our current torque limits are still large as compared to what might be physically realizable. Tuning actuator parameters for complex actuation models such as MTUs remains challenging. Though our actuator optimization technique is able to improve performance as compared to manual tuning, the resulting parameters may still not be optimal for the desired task. Therefore, our comparisons of MTUs to other action parameterizations may not be reflective of the full potential of MTUs with more optimal actuator parameters. Furthermore, our actuator optimization currently tunes parameters for a specific motion, rather than a larger suite of motions, as might be expected in nature.

Since the reward terms are mainly calculated according to joint positions and velocities, it may seem that it is inherently biased in favour of PD and Vel. However, the real challenges for the control policies lie elsewhere, such as learning to compensate for gravity and ground-reaction forces, and learning foot-placement strategies that are needed to maintain balance for the locomotion gaits. The reference pose terms provide little information on how to achieve these hidden aspects of motion control that will ultimately determine the success of the locomotion policy. While we have yet to provide a concrete answer for the generalization of our results to different reward functions, we believe that the choice of action parameterization is a design decision that deserves greater attention regardless of the choice of reward function.

Finally, it is reasonable to expect that evolutionary processes would result in the effective co-design of actuation mechanics and control capabilities. Developing optimization and learning algorithms to allow for this kind of co-design is a fascinating possibility for future work. for each τ i do 21:

SUPPLEMENTARY MATERIAL:

δ i ← r i + γV φ (s i ) -V φ (s i ) 22: φ ← φ + α V 1 n δ i φ V φ (s i ) 23:

end for 24:

Update actor:

25:

Sample minibatch of n tuples {τ j = (s j , a j , r j , λ j , s j )} from replay memory where λ j = 1 26:

for each τ j do 27:

δ j ← r j + γV φ (s j ) -V φ (s j )

28:

if δ j > 0 then 29:

a j ← a j -µ θ (s j ) 30:

ãj ← BoundActionGradient( a j , µ θ (s j ))

31:

θ ← θ + α π 1 n θ µ θ (s j )Σ -1 ãj 32:

end if

33::

end for 34: end while Algorithm 2 Alternating Actuator Optimization 1: θ ← θ 0 2: ψ ← ψ 0 3: while not done do 4:

θ ← argmax θ J(π θ , ψ) with Algorithm 1 5:

ψ ← argmax ψ J(π θ , ψ ) with CMA 6: end while

MTU Actuator Optimization:

The actuator parameters ψ can be interpreted as a parameterization of the dynamics of the system p(s |s, a, ψ). The expected cumulative reward can then be re-parameterized according to

J(π θ , ψ) = S d θ (s|ψ)

A π θ (s, a)A(s, a)da ds where d θ (s|ψ) = S T t=0 γ t p 0 (s 0 )p(s 0 → s|t, π θ , ψ)ds 0 . θ and ψ are then learned in tandem following Algorithm 2. This alternating method optimizes both the control and dynamics in order to maximize the expected value of the agent, as analogous to the role of evolution in biomechanics. During each pass, the policy parameters θ are trained to improve the agent's expected value for a fixed set of actuator parameters ψ. Next, ψ is optimized using CMA to improve performance while keeping θ fixed. The expected value of each CMA sample of ψ is estimated using the average cumulative reward over multiple rollouts.

Figure 5 illustrates the improvement in performance during the optimization process, as applied to motions for three different agents. Figure 6 compares the learning curves for the initial and final MTU parameters, for the same three motions. 

Bounded Action Space:

Properties such as torque and neural activation limits result in bounds on the range of values that can be assumed by actions for a particular parameterization. Improper enforcement of these bounds can lead to unstable learning as the gradient information outside the bounds may not be reliable (Hausknecht & Stone, 2015) . To ensure that all actions respect their bounds, we adopt a method similar to the inverting gradients approach proposed by Hausknecht & Stone (2015) . Let a = (a -µ(s))A(s, a) be the empirical action gradient from the policy gradient estimate of a Gaussian policy. Given the lower and upper bounds [l i , u i ] of the ith action parameter, the bounded gradient of the ith action parameter ãi is determined according to

ãi =    l i -µ i (s), µ i (s) < l i and a i < 0 u i -µ i (s), µ i (s) > u i and a i > 0 a i , otherwise

Unlike the inverting gradients approach, which scales all gradients depending on proximity to the bounds, this method preserves the empirical gradients when bounds are respected, and alters the gradients only when bounds are violated.

Reward:

The terms of the reward function are defined as follows:

r pose = exp -||q * -q|| 2 W , r vel = exp -|| q * -q|| 2 W r end = exp -40 e ||x * e -x e || 2 r root = exp -10(h * root -h root ) 2 , r com = exp -10|| ẋ * com -ẋcom || 2

q and q * denotes the character pose and reference pose represented in reduced-coordinates, while q and q * are the respective joints velocities. W is a manually-specified per joint diagonal weighting matrix. h root is the height of the root from the ground, and ẋcom is the center of mass velocity. 

Sensitivity Analysis:

We further analyze the sensitivity of the results to different initializations and design decisions. Figure 9 compares the learning curves from multiple policies trained using different random initializations of the networks. Four policies are trained for each actuation model. The results for a particular actuation model are similar across different runs, and the trends between the various actuation models also appear to be consistent. To evaluate the sensitivity to the amount of exploration noise applied during training, we trained policies where the standard deviation of the action distribution is twice and half of the default values. Figure 10 illustrates the learning curves for each policy.

Overall, the performance of the policies do not appear to change significantly for the particular range of values. Finally, Figure 11 compares the results using different network architectures. The network variations include doubling the number of units in both hidden layers, halving the number of hidden units, and inserting an additional layer with 512 units between the two existing hidden layers.

The choice of network structure does not appear to have a noticeable impact on the results, and the differences between the actuation models appear to be consistent across the different networks. 

:



Figures and Tables:

Figure 1: Simulated articulated figures and their state representation. Revolute joints connect all links. From left to right: 7-link biped; 19-link raptor; 21-link dog; State features: root height, relative position (red) of each link with respect to the root and their respective linear velocity (green).

Figure 2: Learning curves for each policy during 1 million iterations.

Figure 4: Performance of policies with different query rates for the biped (left) and dog (right). Separate policies are trained for each query rate.

Figure 5: Performance of intermediate MTU policies and actuator parameters per pass of actuator optimization following Algorithm 2.

Figure 6: Learning curves comparing initial and optimized MTU parameters.

Figure 7: Left: fixed initial state biases agent to regions of the state space near the initial state, particular during early iterations of training. Right: initial states sampled from reference trajectory allows agent to explore state space more uniformly around reference trajectory.

Figure 8: Neural Network Architecture. Each policy is represented by a three layered network, with 512 and 256 fully-connected hidden units, followed by a linear output layer.

Figure 9: Learning curves from different random network initializations. Four policies are trained for each actuation model.

Figure 10: Learning curves comparing the effects of scaling the standard deviation of the action distribution by 1x, 2x, and 1/2x.

Figure 11: Learning curves for different network architectures. The network structures include, doubling the number of units in each hidden layer, halving the number of units, and inserting an additional hidden layer with 512 units between the two existing hidden layers.

Figure 12: Performance of different action parameterizations when traveling across randomly generated irregular terrain. (left) Dog running across bumpy terrain, where the height of each bump varies uniformly between 0 and a specified maximum height. (middle) and (right) biped and dog traveling across randomly generated slopes with bounded maximum steepness.

Figure 13: Simulated Motions Using the PD Action Representation. The top row uses an MTU action space while the remainder are driven by a PD action space.

Figure 14: Policy actions over time and the resulting torques for the four action types. Data is from one biped walk cycle (1s). Left: Actions (60 Hz), for the right hip for PD, Vel, and Tor, and the right gluteal muscle for MTU. Right: Torques applied to the right hip joint, sampled at 600 Hz.

Figure 15: Learning curves for different state representations including state + target state, state + phase, and only state.

Sample minibatch of n tuples {τ i = (s i , a i , r i , λ i , s i )} from replay memory

Table content: <table><tr><td colspan="2">Algorithm 1 Actor-critic Learning Using Positive Temporal Differences</td></tr><tr><td colspan="2">1: θ ← random weights</td></tr><tr><td colspan="2">2: φ ← random weights</td></tr><tr><td colspan="2">3: while not done do</td></tr><tr><td>4:</td><td>for step = 1, ..., m do</td></tr><tr><td>5:</td><td>s ← start state</td></tr><tr><td>6:</td><td>λ ← Ber( t )</td></tr><tr><td>7:</td><td>a ← µ θ (s) + λN (0, Σ)</td></tr><tr><td>8:</td><td>Apply a and simulate forward 1 step</td></tr><tr><td>9:</td><td>s ← end state</td></tr><tr><td>10:</td><td>r ← reward</td></tr><tr><td>11:</td><td>τ ← (s, a, r, s , λ)</td></tr><tr><td>12:</td><td>store τ in replay memory</td></tr><tr><td>13:</td><td>if episode terminated then</td></tr><tr><td>14:</td><td>Sample s 0 from p 0 (s)</td></tr><tr><td>15:</td><td>Reinitialize state s to s 0</td></tr><tr><td>16:</td><td>end if</td></tr><tr><td>17:</td><td>end for</td></tr><tr><td>18:</td><td>Update critic:</td></tr><tr><td>19:</td><td/></tr></table>

Training hyperparameters.

Table content: <table><tr><td colspan="4">Character + Actuation Model State Parameters Action Parameters Actuator Parameters</td></tr><tr><td>Biped + Tor</td><td>58</td><td>6</td><td>0</td></tr><tr><td>Biped + Vel</td><td>58</td><td>6</td><td>6</td></tr><tr><td>Biped + PD</td><td>58</td><td>6</td><td>12</td></tr><tr><td>Biped + MTU</td><td>74</td><td>16</td><td>114</td></tr><tr><td>Raptor + Tor</td><td>154</td><td>18</td><td>0</td></tr><tr><td>Raptor + Vel</td><td>154</td><td>18</td><td>18</td></tr><tr><td>Raptor + PD</td><td>154</td><td>18</td><td>36</td></tr><tr><td>Raptor + MTU</td><td>194</td><td>40</td><td>258</td></tr><tr><td>Dog + Tor</td><td>170</td><td>20</td><td>0</td></tr><tr><td>Dog + Vel</td><td>170</td><td>20</td><td>20</td></tr><tr><td>Dog + PD</td><td>170</td><td>20</td><td>40</td></tr><tr><td>Dog + MTU</td><td>214</td><td>44</td><td>282</td></tr></table>

The number of state, action, and actuation model parameters for different characters and actuation models.

Table content: <table><tr><td>Character + Actuation</td><td>Motion</td><td colspan="2">Performance (NCR) Learning Speed (AUC)</td></tr><tr><td>Biped + Tor</td><td>Walk</td><td>0.7662 ± 0.3117</td><td>0.4788</td></tr><tr><td>Biped + Vel</td><td>Walk</td><td>0.9520 ± 0.0034</td><td>0.6308</td></tr><tr><td>Biped + PD</td><td>Walk</td><td>0.9524 ± 0.0034</td><td>0.6997</td></tr><tr><td>Biped + MTU</td><td>Walk</td><td>0.9584 ± 0.0065</td><td>0.7165</td></tr><tr><td>Biped + Tor</td><td>March</td><td>0.9353 ± 0.0072</td><td>0.7478</td></tr><tr><td>Biped + Vel</td><td>March</td><td>0.9784 ± 0.0018</td><td>0.9035</td></tr><tr><td>Biped + PD</td><td>March</td><td>0.9767 ± 0.0068</td><td>0.9136</td></tr><tr><td>Biped + MTU</td><td>March</td><td>0.9484 ± 0.0021</td><td>0.5587</td></tr><tr><td>Biped + Tor</td><td>Run</td><td>0.9032 ± 0.0102</td><td>0.6938</td></tr><tr><td>Biped + Vel</td><td>Run</td><td>0.9070 ± 0.0106</td><td>0.7301</td></tr><tr><td>Biped + PD</td><td>Run</td><td>0.9057 ± 0.0056</td><td>0.7880</td></tr><tr><td>Biped + MTU</td><td>Run</td><td>0.8988 ± 0.0094</td><td>0.5360</td></tr><tr><td>Raptor + Tor</td><td>Run (Sim)</td><td>0.7265 ± 0.0037</td><td>0.5061</td></tr><tr><td>Raptor + Vel</td><td>Run (Sim)</td><td>0.9612 ± 0.0055</td><td>0.8118</td></tr><tr><td>Raptor + PD</td><td>Run (Sim)</td><td>0.9863 ± 0.0017</td><td>0.9282</td></tr><tr><td>Raptor + MTU</td><td>Run (Sim)</td><td>0.9708 ± 0.0023</td><td>0.6330</td></tr><tr><td>Raptor + Tor</td><td>Run</td><td>0.6141 ± 0.0091</td><td>0.3814</td></tr><tr><td>Raptor + Vel</td><td>Run</td><td>0.8732 ± 0.0037</td><td>0.7008</td></tr><tr><td>Raptor + PD</td><td>Run</td><td>0.9548 ± 0.0010</td><td>0.8372</td></tr><tr><td>Raptor + MTU</td><td>Run</td><td>0.9533 ± 0.0015</td><td>0.7258</td></tr><tr><td>Dog + Tor</td><td>Bound (Sim)</td><td>0.7888 ± 0.0046</td><td>0.4895</td></tr><tr><td>Dog + Vel</td><td>Bound (Sim)</td><td>0.9788 ± 0.0044</td><td>0.7862</td></tr><tr><td>Dog + PD</td><td>Bound (Sim)</td><td>0.9797 ± 0.0012</td><td>0.9280</td></tr><tr><td>Dog + MTU</td><td>Bound (Sim)</td><td>0.9033 ± 0.0029</td><td>0.6825</td></tr><tr><td>Dog + Tor</td><td>Rear-Up</td><td>0.8151 ± 0.0113</td><td>0.5550</td></tr><tr><td>Dog + Vel</td><td>Rear-Up</td><td>0.7364 ± 0.2707</td><td>0.7454</td></tr><tr><td>Dog + PD</td><td>Rear-Up</td><td>0.9565 ± 0.0058</td><td>0.8701</td></tr><tr><td>Dog + MTU</td><td>Rear-Up</td><td>0.8744 ± 0.2566</td><td>0.7932</td></tr></table>

Performance of policies trained for the various characters and actuation models. Performance is measured using the normalized cumulative reward (NCR) and learning speed is represented by the normalized area under each learning curve (AUC). The best performing parameterizations for each character and motion are in bold.

Table content: <table/>

Peer review (iclr2017):
This paper addresses a question that is often overlooked in reinforcement learning or locomotion experiment. My biggest point of critique is that it*s difficult to draw conclusions or reason beyond the results of the experiments. The authors only consider a single neural network architecture and a single reward function. For example, is the torque controller limited by the policy network? My suggestion is to vary the number of neurons or show that the same results hold for a different state representation (e.g. trained on pixel data). In the paper*s current form, the term *DeepRL* seems arbitrary. On the positive side, the paper is well-structured and easy to read. The experiments are sound, clear and easy to interpret. It*s definitely an interesting line of work and beyond the extension to 3D, I would argue that considering more realistic physical constraints (e.g. actuator constraints, communication delays etc. on real robots) could greatly improve the impact of this work.