Gen Endo
Tokyo Institute of Technology
2-12-1 Ookayama, Meguro-ku
Tokyo, 152-8550, Japan
gendo@sms.titech.ac.jp

Jun Morimoto
ATR Computational Neuroscience Laboratories
Computational Brain Project, ICORP
Japan Science and Technology Agency
2-2-2 Hikaridai, Seika-cho, Soraku-gun
Kyoto, 619-0288, Japan
xmorimo@atr.jp

Learning CPG-based
Biped Locomotion with a
Policy Gradient Method:
Application to a
Humanoid Robot

Takamitsu Matsubara
ATR Computational Neuroscience Laboratories
2-2-2 Hikaridai, Seika-cho, Soraku-gun
Kyoto, 619-0288, Japan
andNara Institute of Science and Technology
8916-5 Takayama-cho, Ikoma-shi
Nara, 630-0192, Japan
takam-m@atr.jp

Jun Nakanishi
ATR Computational Neuroscience Laboratories
Computational Brain Project, ICORP
Japan Science and Technology Agency
2-2-2 Hikaridai, Seika-cho, Soraku-gun
Kyoto, 619-0288, Japan
jun@atr.jp

Gordon Cheng
ATR Computational Neuroscience Laboratories
ICORP, Japan Science and Technology Agency
2-2-2 Hikaridai, Seika-cho, Soraku-gun
Kyoto, 619-0288, Japan
gordon@atr.jp

Abstract
In this paper we describe a learning framework for a central patterngenerator (CPG)-based biped locomotion controller using a policy
gradient method. Our goals in this study are to achieve CPG-based
biped walking with a 3D hardware humanoid and to develop an
efficient learning algorithm with CPG by reducing the dimensionality of the state space used for learning. We demonstrate that an appropriate feedback controller can be acquired within a few thousand

The International Journal of Robotics ResearchVol. 27, No. 2, February 2008, pp. 213–228
DOI: 10.1177/0278364907084980
c SAGE Publications 2008 Los Angeles, London, New Delhi and Singapore
1trials by numerical simulations and the controller obtained in numerical simulation achieves stable walking with a physical robot in thereal world. Numerical simulations and hardware experiments evaluate the walking velocity and stability. The results suggest that thelearning algorithm is capable of adapting to environmental changes.
Furthermore, we present an online learning scheme with an initial
policy for a hardware robot to improve the controller within 200 iterations.

KEY WORDS—humanoid robots, reinforcement learning,
bipedal locomotion, central pattern generator
213

214

THE INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH / February 2008

1. Introduction
Humanoid research and development has made remarkableprogress over the past ten years (Hirai et al. 19981 Nishiwaki etal. 20001 Kuroki et al. 20011 Hirukawa et al. 20041 Park et al.
2005). Many of the successful humanoids utilize a pre-planned
nominal trajectory designed in a typically known environment.
Despite our best effort, it seems difficult to consider every possible situation in advance when designing such complex controllers. For a broad range of applications working within unknown environments, equipping humanoid robots with a learning capability provides a promising avenue of research. In thispaper, we present a learning framework for bipedal locomotion
for a humanoid robot.
Learning a biped walking pattern for a humanoid robot isa challenging task, owing to the large-scale problem involved
in dealing with the real world. Unlike resolving simple tasks
with robots with fewer degrees of freedom, we cannot directly
apply conventional learning methods to humanoid robots, owing to the dimensionality explosion that is bound to occur. Our
goals in this paper are to acquire a successful walking pattern
through learning and to achieve robust walking with a hardware 3D full-body humanoid robot (Kuroki et al. 2001). See
Figure 1.
While many attempts have been made to investigate learning algorithms for simulated biped walking, there are only afew successful implementations on real hardware, for example
Benbrahim and Franklin (1997), Tedrake et al. (2004) and Morimoto et al. (2005). To the best of our knowledge, Tedrake etal. (2004) have given the only example of an implementationof a learning algorithm on a 3D hardware robot. They developed a simple physical 3D biped robot with specially designed
round soles, possessing the basic properties of a passive dynamic walker (McGeer 1990). They implemented a learningalgorithm on the hardware robot and successfully obtained an
appropriate feedback controller for ankle roll joints via online

Fig. 1. The hardware platform of the full-body humanoid robot: joint configuration and its specification.

learning. With the help of their specific mechanical design thatenabled them to embed an intrinsic walking pattern possessing
passive dynamics, the state space for learning was drasticallyreduced from 18 to 2 in spite of the complexity of the 3D bipedmodel, which usually suffers from a dimensionality explosion.
The question is whether we can easily extend their approachto a general humanoid robot. They used the desired state on the
return map taken from the gait of the robot walking down ona slope without actuation in order to define the reward function for reinforcement learning. Their learning algorithm owes
much to the intrinsic passive dynamical characteristics of therobot that can walk down a slope without actuation. Although,humans also have passive dynamic properties in their joints
and muscles, it is extremely difficult with the current hardware
technology to design general humanoid robots that have bothpassive dynamic properties and high-power joint actuation forvarious tasks. Therefore, their approach cannot be directly applied to general humanoid robots that are not mechanicallydesigned with specific dynamical characteristics for walking.
Moreover, developing a specific humanoid hardware with unifunctionality, for example walking, may lose other importantfeatures of the humanoid robot such as versatility and the capability to achieve various tasks.
Therefore, instead of gait implementation by mechanicaldesign, we introduce the idea of using a central pattern generator (CPG), which has been hypothesized to exist in the central nervous system of animals (McMahon 19841 Orlovsky etal. 19991 Cohen 2003). It is known that during locomotion afeed-forward excitation to the muscles exists that can be independent of sensory feedback and brain input (Grillner et al.
1995). The feed-forward muscle activation is generated by a
CPG within the spinal cord. The most interesting property of
the CPG is that the basic pattern produced by intrinsic oscillation can interact with feedback signals. The intrinsic oscillation of CPG synchronizes the oscillation of feedback signals.
This phenomenon is known as “entrainment”.
It can be demonstrated with numerical simulations that
CPG can generate a robust biped walking pattern with appropriate feedback signals even in an unpredictable environment,
owing to the entrainment property of the CPG (Taga 19951
Miyakoshi et al. 1998). However, designing appropriate feedback pathways in neural oscillators often requires much effort
to manually tune the parameters of the oscillator. Thus, a genetic algorithm (Hase and Yamazaki 1998) and reinforcement
learning (Mori et al. 2004) have been proposed to optimize theopen parameters of the CPG for biped locomotion. However,these methods often require a large number of iterations to obtain a solution owing to the large dimensionality of the state
space used for optimization.
Our primary goals are to achieve biped walking with learning for a 3D full-body humanoid robot, which is not designed
for a specific walking purpose, and to develop an efficientlearning algorithm that can be implemented on a hardware robot with additional online learning capability to improve theEndo, Morimoto, Nakanishi, Cheng, and Matsubara / Learning CPG-based Biped Locomotion with a Policy Gradient Method
controller. In a physical robot, we cannot accurately observe allstates of the system owing to the limited number of equipped
sensors and measurement noise in practice. Thus, we find it
natural to postulate the learning problem as a partially observable Markov decision problem (POMDP).
In this paper, we use a policy gradient method which can beapplied to POMDP (Kimura and Kobayashi 1998). In POMDP,
it is generally known that a large number of iterations would be
required for learning compared with learning in Markov decision problem (MDP), owing to the lack of information, whichyields large variances in the estimated gradient of expected reward with respect to the policy parameters (Sutton et al. 20001
Konda and Tsitsiklis 2003). However, in the proposed framework, when the CPG and the mechanical system of the robot
converge to a periodic trajectory owing to entrainment, the internal states of the CPG and the states of the robot will be
synchronized. Thus, by using the state space that is only composed of the observables, thus reducing the number of states,
efficient learning can achieve steady periodic biped locomotion even in the POMDP.
In our previous work, we demonstrated a learning framework for a CPG-based biped locomotion with a policy gradientmethod on a two-dimensional planar biped robot (Matsubara
et al. 2006). The robot had four leg joints and the control architecture of the robot consisted of a CPG-based controller for
two hip joints and a state-machine controller for the two knee
joints. Appropriate sensory feedback for the CPG to perform
steady walking could be learned within a few hundred trials in
simulations and we applied the controllers acquired from numerical simulations to a physical five-link biped robot. We empirically verified that the robot was able to successfully walkin a real environment.
In this paper, we extend our previous approach to a 3D
full-body humanoid robot and present that the policy gradientmethod can acquire a steady walking pattern for a general 3D
full-body humanoid. As a 3D full-body humanoid robot has
many degrees of freedom, the dynamics of the 3D humanoidis much more complicated than a 2D system. Thus, we propose the idea of allocating CPGs in a task space coordinate
system, while exploiting symmetry to simplify the control architecture. In this paper, we demonstrate that an appropriate
feedback controller for a 3D full-body humanoid can be acquired by using a policy gradient method and the obtained controller in numerical simulation can achieve stable walking witha physical robot in the real world. Moreover, we discuss a turning walk with a desired turning radius and an online learning
scheme with initial policy for a hardware robot to improve thecontroller.

2. CPG Control Architecture
In this section we describe the basic framework of our CPG
control architecture. Our goals are to generate a steady straightwalking pattern and a steady circular walking pattern with

215

Fig. 2. Basic framework of CPG control architecture. It consists of three parts: CPG controller, robot and CPG feedback
controller. The CPG controller generates the leg trajectory using a neural oscillator with a feedback signal. The leg trajectory is converted into joint torques via inverse kinematic calculation and PD servo. Then the robot interacts with the environment. The CPG feedback controller drives a feedback signal to
CPG using incoming sensory information. The following notation is used: a j is a feedback signal to CPG and q j is the output
of a neural oscillator1 Rdesired is a parameter that specifies theturning radius1 pl1r indicates left/right leg position with respect
to the body-fixed Cartesian coordinates1 22 roll 1 22 pitch are the angular velocity of the body in the roll and pitch direction, respectively1 2 roll is inclination angle in the direction of roll withrespect to world coordinates1 and f zl1r is the vertical reaction
force of left/right leg.

variable turning radius. Figure 2 shows the basic framework of
the CPG control architecture. We introduce a neural oscillator
to model a CPG and the oscillator output q j is transformed intoleg position with respect to body-fixed Cartesian coordinate
system. We also introduce Rdesired , a parameter that specifies aturning radius defined on the ground to generate circular walking. Rdesired modulates leg position, pl1r , based on geometricalconstraints to walk along a specified circular walking trajectory. Then, the desired joint position for the joint PD servo isobtained through inverse kinematics and is used to control an
actuator.
The CPG feedback controller generates the feedback signal to the CPG, a j , using sensory information from the robot.
The CPG feedback controller consists of reinforcement learning for the oscillator allocated for the X (forward) direction
and biologically inspired feedback arranged for the Z (vertical) direction. Finally, a j is fed back to the neural oscillator
which automatically adjusts its output due to the entrainment
property.
This framework provides us with an inherent rhythmic
walking pattern modulated by the CPG feedback controller
using sensory information from the environment. We discuss
the above-mentioned framework in detail in the following sections.

216

THE INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH / February 2008

2.1. Neural Oscillator Model
There are several ways to model a CPG. It can be mathematically modeled with a non-linear oscillator, such as a Van der
Pol oscillator, phase oscillator or neural oscillator. One recent
example is the work of Aoi and Tsuchiya (2005), who applied non-linear oscillators to the control of a small walking
humanoid robot, utilizing the foot-contact to reset the phaseof the oscillators to increase the stability of the system. Theirresults demonstrated one successful application of phase oscillators to humanoid locomotion.
In the learning framework we present in this paper, we propose to change the modulation of the phase as well as the
trajectories of the walking patterns of the robot to attain successful and robust locomotion. Our focus, in particular, is on acoupled neural oscillator model proposed by Matsuoka (1985).
The Matsuoka oscillator has a number of beneficial properties, most notably, it allows modulations of sensory feedback
for adaptation to the environment and control of the amplitudewith a single scale factor. These are some of the well-studied
aspects of the Matsuoka oscillator, making it suitable for ourinvestigation.
The oscillator dynamics of jth neural unit are
3 CPG z2 j

3 4z j 4n1

4 jk qk 4 5 z 5j 6 c 6 a j 1

Fig. 3. Schematic of a coupled neural oscillator: two neuralunits have mutual inhibition1 a112 and q112 are input/output signals, respectively. Lines ending with black or white circles
indicate inhibitory or excitory neural connections, respectively.

(1)

k31

3 5CPG z25 j

3 4z j 6 q j 1

(2)

qj

3 max601 z j 71

(3)

where n is the number of neurons. The model represents thefiring rate of a neuron by a continuous variable q j with time.
Here Z j represents mean membrane potential and Z 5j is a variable which represents the self-inhibition effect. Adaptation ismodeled by the dynamics of Z 5j in (2) and the influence of thedegree of adaptation on Z j is represented by a constant parameter 5 . We use 3 CPG and 3 5CPG to denote the time constantsof the mean membrane potential Z j and adaptation effect of
the jth neuron, respectively. We also use 4 jk to denote an inhibitory synaptic weight from the kth neuron to the jth neuron,c to denote a tonic input with a constant rate and a j to denotea feedback signal.
Figure 3 shows a schematic of a coupled oscillator where
two neural units are connected by mutual inhibitions. The circles with numbers represent neural units whose dynamics aredefined by (1)–(3). Lines ending with black or white circles indicate inhibitory or excitory neural connections, respectively.
The properties of the Matsuoka neural oscillator model
have been explored numerically, signifying the relationshipbetween the parameters and the oscillator output (Williamson
1998). For example, the two time constants 3 CPG and 3 5CPG determine the frequency and shape of the output, and if the ratio
3 CPG 83 5CPG is kept constant, the natural frequency of the oscillator is proportional to 183 CPG . The tonic input c controls theFig. 4. Entrainment property of a neural oscillator: a sinusoid
input is fed into the oscillator at 4.1 s for 10 s. The dashed,
a1 , and solid, 6q1 4 q2 7, lines are the input and output signal of the neural oscillator, respectively. The parameters are
3 CPG 3 09224, 3 5CPG 3 09280, 12 3 21 3 290, 5 3 295,
c 3 2936, where the natural frequency of the oscillator andthe amplitude are 09775 and 190, respectively. The frequencyof the input sinusoid is 0.4 and the amplitude is also 0.4.

amplitude of the output of the oscillator. It is demonstrated thatthe phase difference between the periodic input signal a j andthe output q j is tightly locked through entrainment when theamplitude of a j is large enough and its frequency is close to
the oscillator’s natural frequency.
Figure 4 shows a time course of the oscillator output wherea sinusoid input a1 63 4a2 7 is fed into the oscillator at 4.1 s
for 10 s. The frequency of the oscillator output is immediately
entrained to the frequency of the input sinusoid and phase difference between the input and the output becomes constant.
Figure 4 demonstrates that a neural oscillator has inherent dynamics which can be modulated by a input signal. The key
issues to perform robust biped walking are how to allocate theEndo, Morimoto, Nakanishi, Cheng, and Matsubara / Learning CPG-based Biped Locomotion with a Policy Gradient Method

217

neural oscillator to control biped walking and how to derive input signals a j to exploit the entrainment property of the neuraloscillator. We discuss them in the following section.
2.2. CPG Arrangement and Leg Trajectory
In many of the previous applications of neural-oscillator-based
locomotion studies, an oscillator is allocated at each joint andits output is used as a joint torque command to the robot (Taga19951 Hase and Yamazaki 19971 Ishiguro et al. 2003). However, it is difficult to obtain appropriate feedback pathways
for all of the oscillators to achieve the desired behavior with
the increase in the number of degrees of freedom of the robot
because neural oscillators are intrinsically non-linear. Moreover, the realization of precise torque control of each joint isalso difficult for a hardware robot in practice. Thus, to simplify the problem, we have proposed a new oscillator arrangement with respect to the position of the tip of the leg in the
Cartesian coordinate system, which can be reasonably considered as the task coordinates for walking (Endo et al. 2005b).
We allocate only six neural units exploiting symmetry of thewalking pattern between the legs. We decompose overall walking motion into stepping motion in place produced in thefrontal plane and propulsive motion generated in the sagittal
plane. The effectiveness of this decomposition has been empirically demonstrated in our previous studies (Endo et al. 2004,
2005b).
Figure 5 illustrates the proposed neural arrangement for the
stepping motion in place in the frontal plane. We employ acoupled oscillator with mutual inhibitions (4 12 3 4 21 3 290)
and allocate it to control the position of both legs pzl , pzr along
the Z (vertical) direction in a symmetrical manner with a radphase difference:pzl

3

Z 0 4 Az 6q1 4 q2 71

(4)

pzr

3

Z 0 6 Az 6q1 4 q2 71(5)

where Z 0 is a position offset and Az is the amplitude scaling
factor.
For propulsive motion in the sagittal plane, we introduce
a quad-element neural oscillator to produce coordinated leg
movements with a stepping motion based on the following observation: as illustrated in Figure 6, when the robot is walking forward, the leg trajectory with respect to the body coordinates in the sagittal plane can be roughly approximated withthe shape of an ellipsoid. Suppose that the output trajectories ofthe oscillators can be approximated as pxl 3 A x cos6 t 6 x 7
and pzl 3 Az cos6 t 6 z 7, respectively. Then, to form theellipsoidal trajectory on the X–Z plane, pxl and pzl need to satisfy the relationship pxl 3 A x cos and pzl 3 Az sin , whereis the angle defined in Figure 6. Thus, the desired phase difference between vertical and horizontal oscillation should be
x 4 z 3 82. To embed this phase difference as an intrinsic

Fig. 5. Neural oscillator allocation and biologically inspired
feedback pathways for a stepping motion in place. The neuraloscillator output, 6q1 4 q2 7, symmetrically controls the left andright leg position in the vertical direction Z with respect tothe body-fixed coordinates Rbody where Z 0 1 A x are an initialoffset and a gain, respectively. The reaction force informationin the Z direction, f Zl1r , is used as the extensor response andthe posture inclination in the roll direction, 2 Roll , is used as thevestibulospinal reflex. Here a112 are feedback signals derived
from (14).

property, we install a quad-element neural oscillator with unidirectional circular inhibitions (4 34 3 4 43 3 4 56 3 4 65 3
290, 4 35 3 4 63 3 4 46 3 4 54 3 095). It generates an inherent phase difference of 82 between two coupled oscillators,(q3 4 q4 ) and (q5 4 q6 ) (see Matsuoka (1985)). Therefore, if(q3 4q4 ) is entrained to the vertical leg movements, then an appropriate horizontal oscillation with desired phase difference isachieved by (q5 4 q6 )1.
Similar to the Z direction, the neural output (q5 4 q6 ) is
allocated to control the position of both legs pxl , pxr along the
X (forward) direction in the sagittal plane:pxl

3

X 0 4 A x 6q5 4 q6 71

(6)

pxr

3

X 0 6 A x 6q5 4 q6 71(7)

where X 0 is an offset and A x is the amplitude scaling factor.

1. At the beginning of the investigation, we directly used (q1 1 q2 ) for a quadelement oscillator. However, the oscillator dynamics of the stepping motion
interfered with the dynamics of propulsive motion via uni-directional circularinhibitions. As a result, the biologically inspired feedback signal, which wasderived for a coupled oscillator, was not suf2ciently effective to produce robust
stepping motion in place. Thus, in order to clearly divide stepping motion andpropulsive motion, we introduce duplicated neural units (q3 1 q4 ) for a quadelement oscillator to produce behavior similar to (q1 1 q2 ).

218

THE INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH / February 2008

Fig. 6. A quad-element neural oscillator for propulsive motion in the sagittal plane. A schematic of a quad-element
neural oscillator is shown on the left-hand side and an ellipsoidal leg trajectory in the X–Z plane is shown on the righthand side. The propulsive leg trajectory can be estimated withthe shape of an ellipsoid. The ellipsoid can be expressed as
pxr 3 X 0 6 A x cos , pzr 3 Z 0 6 Az sin , where is a parameter shown in this figure. Thus, oscillatory movements in the
X direction and Z direction needs a phase difference of 82.
A quad-element neural oscillator consists of two coupled oscillators with a uni-directional circular inhibitory neural connection. The oscillator output 6q5 4 q6 7 has an inherent phasedifference of 82 with respect to the output 6q3 4q4 7. As duplicated feedback from the stepping motion is fed into the neuralunits 631 47, the oscillator output 6q5 4 q6 7 tends to keep thephase difference of 82. The output 6q5 4 q6 7 is used to movethe left/right leg symmetrically in the X direction.

Fig. 7. Foot prints captured in double support phase during aturning walk. Step length and leg yaw rotation are modulatedwith respect to a desired circular radius, Rdesired . We assume
that the origin of body-fixed coordinates moves along the desired circular arc (dashed arc). We use 2 A x to denote the step
length for a straight walk and 2Alx , 2Arx are modulated step
lengths in order to satisfy kinematic constraints without slippage due to lateral leg position offset Y0l , Y0r , respectively. Herel1rindicates the angle of necessary yaw rotation.
4 pyaw

slippage of the stance leg, the inner step length should be decreased while the outer step length should be increased due toleg position offset Y0l1r 67Y0l1r 7  7Rdesired 77:
Al1r
x 3

This framework provides us with a basic walking patternwhich can be modulated by feedback signals a j .

(8)where A x is the nominal step length in the case of straightwalking and Al1r
x are modulated step lengths of the left andright legs. Thus, (6) and (7) are rewritten as follows:

2.3. Turning Controller
We introduce an additional mechanism to control the walking
direction by modulating the right and left step length as wellas the yaw rotation of both legs. In this controller, we focus on
kinematic constraints to walk along a specified desired circularradius, Rdesired , defined in the horizontal walking surface. The
desired circular radius Rdesired modulates the mapping from a
CPG output q j to leg position pl1r .
As illustrated in Figure 7, we assume that the origin ofbody-fixed coordinates moves with constant velocity along aparticular circular arc defined by Rdesired (Rdesired  0, when
the robot makes a right turn), and left and right stance legs also
move along the concentric circular arcs defined by Rdesired 6Y0l
and Rdesired 4 Y0r , where Y0l1r is leg position offset in the lateral direction. In order to satisfy kinematic constraints without

Rdesired 6 Y0l1r
8 Ax 1
Rdesiredpxl

3

X 0 4 Alx 8 6q5 4 q6 71

(9)

pxr

3

X 0 6 Arx 8 6q5 4 q6 79

(10)

l1r
, pl1r
The yaw rotation and Y position of the legs, pyaw
y , are also
controlled to satisfy kinematic constraints to keep constant angular velocity around the center of turning as follows:

pxl1r 4 X 0

l1r
pyaw

3 4

pl1r
y

l1r
3 Y0l1r 4 61 4 cos6 pyaw
77 8 6Rdesired 6 Y0l1r 79 (12)Rdesired 6 Y0l1r

1

(11)

Considering only kinematic constraints is not sufficient toachieve an executable walking motion because it neglects theeffect of dynamics. As demonstrated, the proposed framework
possesses an entrainment property that to some extent could

Endo, Morimoto, Nakanishi, Cheng, and Matsubara / Learning CPG-based Biped Locomotion with a Policy Gradient Method
better cope with the effect of dynamics. In addition, the learning algorithm is capable of adjusting the basic walking pattern
via CPG feedback signals. Therefore, in our case, we can conceive that kinematic constraints can be adequate to generate
a turning motion. The advantage of this turning controller isthat we can continuously vary the walking direction by only
adjusting a single parameter, Rdesired .
2.4. Sensory Feedback
In biological systems, several reflexes have been found thatgenerate recovery momentum according to the inclination of
the body by adjusting the leg length. For example, a decerebrate cat stomps stronger when vertical perturbation force isapplied to its planter during extensor muscle activation. This
reflex is called an extensor response (Cohen and Boothe 1999).
It is generally known that the vestibular system measures the
body’s inclination and activates contralateral muscles to keep
upper body stabilized. This is one of the basic posture controls in humans and is called the vestibulospinal reflex. The
effectiveness of these feedback pathways was experimentallydemonstrated with a hardware quadruped robot (Kimura et al.
2007) to maintain the balance of the body when walking overunknown terrain.
As a first step in this study, we focus on acquiring the feedback controller (a5 ) for the propulsive leg movement in the X
direction (as illustrated in Figure 5). The explicitly designed
sensory feedback pathways for stepping motion in place (a1 )are strongly motivated by biological observations (Figure 5).
Based on the assumption of symmetric leg movements generated by the coupled oscillator, we introduce symmetrical
feedback signals to the oscillator as follows:
a2m 3 4a2m41 1

6m 3 11 21 379(13)

We then introduce the extensor response and vestibulospinal reflex by adjusting the leg length according to the vertical reaction force or the inclination of the body as follows:
a1 3 h ER 6 f zr 4 f zl 78mg 6 h VSR 2 roll 1(14)where 6 f zr 4 f zl 7 are the right/left vertical reaction force differences normalized by total body weight mg and h ER , h VSR
are scaling factors. The extensor response, the first term of theright-hand side of (14), extends the leg length when the vertical reaction force is applied. This motion moves the centerof pressure of the robot to the opposite side. Thus, repetitive
stepping constructs a negative feedback loop to keep the center of pressure between two feet. The vestibulospinal reflex,
the second term of the right-hand side of (14), also extends theleg length according to the inclination of the body. This reflex
also constructs a negative feedback loop to maintain the upright posture. The extensor response and vestibulospinal reflex
construct redundant feedback pathways which are tolerant of

219

failure of sensing in a hardware system and they can be combined nicely by adjusting scaling factors. A dominant feedback pathway to increase robustness against perturbation canchange on a case-by-case basis, depending on the perturbations. We have demonstrated the robustness of stepping motion against perturbation experimentally (Endo et al. 2005b).
We investigated scaling factors empirically using the hardware robot by applying various perturbations and determinedh ER 3 094, h VSR 3 198.
The same feedback signal is fed back to the quad-element
neural oscillator for the propulsive motion, a3 3 a1 , to induce cooperative leg movements with 82 phase difference
between the Z and X direction (Figure 6).
For the propulsive leg movements in the X direction, thefeedback signal a j is represented with a policy gradient:a j 6t7 3 a max
j g6 j 6t771(15)

where the function g is a saturation function defined by
g6 j 6t77 3

2

2
arctan23
 j 6t7 1

and a maxis the maximum value of the feedback signal. Thejoutput of the feedback controller  j is sampled from a stochastic policy which locally maximizes expected total return(the accumulated reward, which is defined in the following
section). The stochastic policy is estimated with a probability
distribution wa 6x1  j 7 3 P6 j 7 x9 wa 7:4546 j 4  j 6x9 w 772
1
1 (16)
wa 6x1  j 7 3 2  6w 7 expj
2 2j 6w 7
where x denotes partial states of the robot, and wa 3 [6w 7T ,6w 7T ] is the parameter vector of the policy. We can equivalently represent  j by
 j 6t7 3  j 6x6t79 w 7 6  j 6w 7n j 6t71

(17)

N 601 17. Here N 601 17 is a normal distribuwhere, n j 6t7tion which has a mean  3 0 and a variance  2 3 1. In thenext section, we discuss the learning algorithm that we use toacquire a feedback controller a j35 for the propulsive motion.

3. Learning the Sensory Feedback Controller
Promising results have been demonstrated in applications of
the policy gradient technique to POMDP biped walking locomotion (Tedrake et al. 20041 Endo et al. 2005a). In our previous work (Matsubara et al. 2006), we compared the policy
gradient method with a conventional value-function-based reinforcement learning scheme on a 2D biped walking task. The
policy gradient method could acquire steady walking with asmaller number of trials, suggesting that the policy gradient

220

THE INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH / February 2008

method is suitable for POMDPs. In this paper, we use the samelearning algorithm for the acquisition of a policy for the sensory feedback controller of the neural oscillator model in the
X direction.
Humanoid robots have many degrees of freedom with alarge number of equipped sensors within a single hardware
system1 therefore, it is not feasible to use all states of the robot for learning. Thus, we have to select a reasonable numberof state variables for learning. To describe the representative
motion of the robot, we focus on the states of the pelvis of therobot. The position of the pelvis can roughly approximates thelocation of the center of mass (COM) of the system. We chosethe pelvis angular velocity x 3 622 roll 1 22 pitch 7T as the input state
to the learning algorithm, and 22 roll , 22 pitch are measured withgyro sensors located on the pelvis of the hardware. Therefore,
the other states of the robot, such as inclinations of the pelvis,
the linear velocity with respect to world coordinates, and theposition and velocity of each joint are considered hidden variables for the learning algorithm.
The learning framework of the policy gradient method for
our CPG control architecture is illustrated Figure 8. First, theCPG controller generates leg trajectory in the X direction, allowing the robot to interact with the physical environment. The
partial states of the robot and reward information are sent to
the learning agent. A critic tries to estimate the value using the
temporal difference (TD) error, represented in continuous timeand space. Then, an actor generates a CPG feedback signal forthe leg trajectory in the X direction based on a stochastic policy defined by a probability distribution with parameter vectors
w. Both the value function and the policy are updated using a
TD error and eligibility trace according to Kimura’s update
rule (see Section 3.2 for further details).
In the following sections, we introduce the definition of thevalue function in continuous time and space to derive the TD
error (Doya 2000). Then, we discuss the learning method of apolicy for the sensory feedback controller. We use a normalized Gaussian network (NGnet) to approximate both the valuefunction and the policy (see the Appendix). Finally, we designa reward function to generate steady walking.

3.1. Learning the Value Function
Consider the dynamics of the robot including the CPG definedin continuous time and continuous states,
dxall 6t7
3 f 6xall 6t71 a6t771
dt

(18)

X
1l is all of the states of the robot and thewhere xall
CPG, and a A 1m is the output of the feedback controller
to the CPG. We denote the immediate reward for the state andaction as(19)
r6t7 3 r 6xall 6t71 a6t779

Fig. 8. Schematic diagram of CPG feedback learning. Herea j 6t7 is the feedback signal derived with a policy gradientmethod1 x is the input states used for learning, where 22 roll ,22 pitch are the pelvis angular velocity. A critic estimates thevalue Vwc 6t7, where wc is the parameter of the function approximator. We use 6t7 to denote a TD error in continuous
time and space,  j 6t7 to denote a stochastic policy defined bya probability distribution, where w , w are parameter vectorsof the policy, and n j 6t7 to denote a normal distribution.

The value function of state xall 6t7 based on a policy 6xall 1 a7
is defined as
8 9
67 
8
e46s4t783 r 6xall 6s71 a6s77 ds 88
V 6xall 6t77 3 E
1 (20)twhere 3 is a time constant for discounting future rewards. The
consistency condition for the value function is given by thetime derivative of (20) as1
dV 6xall 6t77
3 V 6xall 6t77 4 r 6t79
dt
3

(21)

We denote the current estimate of the value function as
V 6xall 6t77 3 V 6xall 6t79 wc 7, where wc is the parameter ofthe function approximator. If the current estimate of the valuefunction V is perfect, it should satisfy the consistency condition of (21). If this condition is not satisfied, the prediction
should be adjusted to decrease the inconsistency,
6t7 3 r6t7 41
V 6t7 6 V2 6t79
3(22)This is the continuous-time counterpart of TD error (Doya2000).
As we consider a learning framework in POMDPs, i.e. weonly observe partial states x from all states xall , the TD error usually does not converge to zero. However, Kimura andEndo, Morimoto, Nakanishi, Cheng, and Matsubara / Learning CPG-based Biped Locomotion with a Policy Gradient Method
Kobayashi (1998) suggested that the approximated value function can be useful to reduce the variance of the gradient estimation in (25), even if the consistency condition in (21) is notsatisfied.
The parameter vector of the value function wc is updatedwith TD error and an eligibility trace. The eligibility trace isused to assign credit for the TD error backwards in time topreviously visited states. The update laws for wc and the eligibility trace vector ec for wc are defined, respectively, as1 Vwc
e2 c 6t7 3 4 c ec 6t7 6
1

wc

(23)

2 c 6t7 3w(24)

6t7ec 6t71

where is the learning rate and  c is the time constant of theeligibility trace.
We manually tune 3 1 1  c based on the following intuitions. For the learning rate, , we try to maximize the learning rate that is small enough to estimate the expectation in (20)by averaging sample data. The time constant 3 corresponds tothe discount rate in a discrete time learning system. Smaller
3 means that future reward is considered to yield a smaller
value than the actual value. We set a sufficiently large value
for this parameter to take into account the negative reward thatis caused when the robot falls over. The time constant  c controls credit assignment of the reward to the previously visitedstate. As we consider a partially observable environment, it isimportant to assign the credit from an actual acquired reward,
which does not depend on the estimated value function. However, too large a  c leads to large variance in the value function
estimation. In this study, we select the learning parameters as
3 3 190, 3 78,  c 3 095.

3.2. Learning a Policy of the Sensory Feedback Controller
Kimura and Kobayashi (1998) presented that by using TD error 6t7 and an eligibility trace vector ea 6t7, it is possible toobtain an estimate of the gradient of the expected actual return Vt with respect to the parameter vector wa in the limit of a 3 3 as
E Vt 7
wa
where
Vt 3

7 

wa

 3 E6t7e 6t71
ae46s4t783 r 6s7 ds1

(25)

(26)

t

wa is the parameter vector of the policy wa 3 6x1 a9 wa 7
and ea 6t7 is the eligibility trace vector for the parameter vector
wa . The parameter vector of the policy wa is updated with TD
error and the eligibility trace. The eligibility trace is used toassign credit for the TD error backwards in time to previously

221

generated actions. The update laws for wa and the eligibility
trace vector ea 6t7 can be derived, respectively, as
1
 ln wa1e2 a 6t7 3 4 a ea 6t7 6

wa

(27)

2 a 6t7 3 6t7ea 6t71
w

(28)

where  is the learning rate and  a is the time constant of theeligibility trace.
In the actor–critic algorithm of Sutton et al. (2000) andKonda and Tsitsiklis (2003), a Q-function was used to update
the parameters of the actor. In contrast, in Kimura’s approach,
the TD error is used to update the policy parameters. This isbecause if the target dynamics depend on a policy parameter,
information for the proper gradient direction of the policy parameter can be acquired, because the TD error depends on the
dynamics of the environment.
The basic intuition for updating the policy with TD error
and the eligibility trace is as follows: larger TD error indicatesthat the generated action gave a better result, i.e. acquired alarger reward and/or achieved a state that has a larger estimatedvalue than the expected value. To increase the chances of acquiring a larger reward, we can increase the policy parametersthat contribute to increasing the TD error. The eligibility traces
represent the contribution of each policy parameter.
We manually tune 1  a based on the following intuitions.
For a learning rate , we try to maximize the learning ratethat is small enough to estimate the expectation in (25) by averaging the sample data (using a small learning rate has thesame effect as averaging the sample data). The time constant a controls the credit assignment of the reward to the previously generated action. As we consider a partially observable
environment, it is important to assign the credit from the actual
acquired reward, which does not depend on the estimated valuefunction. However, too large a  a leads to a large variance inthe gradient estimation. In this study, we set the learning parameters to   3   3 195,   3 190,   3 091.
3.3. Rewards
We design a reward function:
r6x7 3 k H 6h 1 4 h 5 7 6 k S  x 1(29)

where h 1 is the pelvis height of the robot, h 5 is a threshold parameter for h 1 and  x is forward velocity with respect to theground. The reward function is designed to keep the height ofthe pelvis as the first term, while at the same time to achieve
forward progress with the second term. In this study, the parameters were chosen as k S in the range 0.0–10.0, k H 3 1090,
h 5 3 09272, where the unit for h 1 and h 5 is meters and for  x ismeters per second. The threshold parameter h 5 is determinedby a position offset Z 0 . The robot receives a punishment (negative reward) r 3 41, if it falls over.

222

THE INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH / February 2008

4. Experiments
4.1. Dynamics Simulator
We developed a dynamics simulator for our biped robot using SD/FAST (Symbolic Dynamics Inc.). We used detailed
mass property data calculated from 3D CAD data and massmeasurement of actual parts of the robot. We introduced an
actuator model for all joints incorporating PD servo and gearhead friction. The coefficients of PD gains, Coulomb friction
and viscous friction were experimentally determined using the
hardware robot. We also introduced a sensor model with a digital filter whose coefficients were carefully tuned to match theproperty of the actual sensor. To calculate reaction forces fromthe environment, we assumed that each sole has four contact
points and reaction forces are obtained by a simple springdumper model. The integration time step of the numerical simulator was set to 0.1 ms and the learning was performed at
16 ms interval. The total computation including learning algorithm required 3.8 times longer than real time.

4.2. Simulation Results and Hardware VerificationsFirst, we carried out numerical experiments using the dynamics simulator to acquire a feedback controller (policy) for the
CPG for biped walking. Then, we implemented the acquired
policies on the hardware robot.
We conducted a number of trials in a numerical experimentin the following sequence. At the beginning of each trial, we
utilized a hand-designed feedback controller to initiate walking gait for several steps in order to start the learning processwith the appropriate state of the robot. Then, we switched thefeedback controller for the learning algorithm at random in order to generate various initial state inputs. Each trial is terminated if the immediate reward is 41 and below, or the robot walks for 20 s. We repeated the numerical experiment trials and the policy was saved at every 50 trials. The learning
process is considered a success when the robot does not fall
over after 20 successive trials. We terminated the experimentin cases where an additional 3,000 trials were done after successful acquisition or 5,000 trials even without successfullyacquiring a policy.
As expected, at the beginning of the learning process therobot immediately fell over within a few steps straight afterswitching to the learned feedback controller. The policy gradually increases the output amplitude of the feedback controller
to improve walking motion as the learning proceeded. Basedon 27 numerical experiments with various velocity rewards,
k S , walking motion was successfully acquired for 20 experiments (Table 1)1 we could not observe policy convergence forseven failed experiments. The required trials are the averaged
value of each successful experiment with the same k S . Typically, it takes 20 h to run one simulation for 1,000 trials andFig. 9. Typical learning curve. Steady walking without fallingover is acquired after 180 trials in this example.

Table 1. Achievements in the acquisition of straight walking.
Number of achievements
kS

Number ofexperiments

Simulation(trials)

Hardware(trials)

0.0

41(2,385)

0
(—)

1.0

3

3
(528)

3
(1,600)

2.0

3

3
(195)

1
(800)

3.0

4

4
(464)

2
(1,500)

4.0

3

2
(192)

2
(350)

5.0

5

2
(1,011)

1
(600)

10.0

5

5
(95)

5
(460)

Sum
(average)

27

20
(696)

14
(885)

the policy was acquired on average after 696 trials. Figures 9
and 10 show a typical example of accumulated reward at eachtrial and an acquired policy, respectively. Figure 10 shows thatwhile 22 roll dominates the policy output, 22 pitch does not assert
much influence. The reason is that 22 roll is always being generated by the stepping motion regardless of the propulsive motion. Therefore, the policy tries to utilize 22 roll to generate synchronized leg movements. We also observed that 22 pitch is suppressed by the reward function because 22 pitch lowers the pelvis
height by a pitching oscillation, which can cause falling.
We transferred the acquired 20 successful learning processes onto the robot by using a series of policies with different trials but the same learning process. We then used theseEndo, Morimoto, Nakanishi, Cheng, and Matsubara / Learning CPG-based Biped Locomotion with a Policy Gradient Method223

turbation in the real environment. We also carried out walking experiments on a slope and the acquired policy achieved
steady walking in the range of 63 to 44 inclination, suggesting sufficient walking stability.
4.3. Velocity Control

Fig. 10. Typical learned policy. The gray scale shows the acquired stochastic policy where the horizontal and vertical axes
are input states used for learning 22 roll and 22 pitch , respectively.

policies to determine the required additional iterations for the
robot to walk. The walking experiment on the hardware wasconsidered a success when the robot achieved steady walking
on the carpet floor for 3 m without falling over. We verified
improvements of the policy in accordance with the numbersof trials in the numerical simulation. With the policy on theearly stage of a learning process, the robot exhibited back andforth stepping then immediately fell over. With the policy on
the intermediate stage, the robot performed unsteady forward
walking and occasional stepping on the spot. With the policy
after substantial trials, the robot finally achieved steady walking.
We confirmed that 14 of the successful learning processes
in the numerical experiments performed successful walking on
the hardware (Table 1). Figure 11 shows snapshots of an acquired walking pattern. The six policies, which could not succeed on the hardware experiment, had similar profiles to a typical policy shown in Figure 10. However, the output amplitude
was slightly smaller or significantly larger than the policy thatis applicable to the hardware. In particular, the large feedback
signal to CPG led to instantaneous leg movement when 22 roll
was across zero. Consequently, the leg movement exceeded thecurrent limit of the actuator of the robot, which was not modeled in the dynamics simulator, and the robot fell over. The
policy with slightly smaller output can be improved via online
learning, which is discussed in the following section.
In our experiments, an additional 189 trials (on average) innumerical simulations were required for the policy to achieve
walking in the physical environment. We confirmed that theoutput amplitude of the feedback signal progressively increased in accordance with the number of trials. This result
suggests that the learning process gradually exploits the entrainment property of the CPG through iterations, and consequently the policy acquires adequate robustness against per-

To control walking velocity, the relationship between the reward function and the acquired velocity was investigated.
We set the parameters in (1) and (2) to 3 CPG 3 09105,
3 5CPG 3 09132, c 3 2908, 5 3 295 to generate an inherent oscillation, where its amplitude is 1.0 and period is 0.8.
As we set A x 3 09015 m, the expected walking velocity
with intrinsic oscillation, (step length)/(step cycle), becomes
609015  190786098827 3 09075 m s41 .
We measured the average walking velocity both in numerical simulations and hardware experiments with various k S in
the range 0.0–5.0 (Figure 12). The resultant walking velocity
in the simulation increased as we increased k S and hardware
experiments also demonstrated a similar tendency.
This result shows that the reward function works appropriately to obtain a desirable feedback policy, which is difficult
for a hand-designed controller to accomplish. Thus, we believe that it would be possible to acquire different feedback
controllers with some other criteria, such as energy efficiency,
by using the same scheme.
4.4. Stability Analysis
To quantify the stability of an acquired walking controller, weconsider the periodic walking motion as discrete dynamics andanalyze the local stability around a fixed point using a returnmap. We perturbed the target trajectory on (q5 4 q6 ) to change
the step length at random timings during steady walking, andcaptured the states of the robot when the left leg touched down.
We measured two steps, just after the perturbation (dn ) andthe next step (dn61 ). If acquired walking motion is locally stable, the absolute eigenvalue of the return map should be lessthan 1.
Figure 13 shows the return map with 50 data points witha white dot indicating a fixed point derived by averaging
100 steps without perturbation. The estimated eigenvalue is4090101 calculated with a least-squares fit. The results suggests that even if step length was reduced to half of the nominal
step length by perturbation, for example pushed forward, thefeedback controller quickly converges to the steady walking
pattern within one step.
4.5. Turning Walk
We carried out circular arc walking experiments using the turning controller. We set Rdesired 3 0931 0951 190 m and A x 3
l1r
is small, we
09015 m, Y0l 3 0904 m, Y0r 3 40904 m. As pyaw

224

THE INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH / February 2008

Fig. 11. Snapshots of straight steady walking with the acquired feedback controller (A x 3 09015 m, Az 3 09005 m,  x 309077 m s41 . Photos were captured every 0.1 s.)

Fig. 12. The relationship between acquired average velocity
and velocity reward k S . Each data point used different policy
and average velocity was derived by averaging steady walking
velocity for 10 s. The solid line shows linear approximation for
all data points.

can abbreviate the second term in the right-hand side of (12).
To calculate the forward walking velocity  x for the rewardin (29), we used the relative velocity of the stance leg withrespect to the pelvis in the X direction. We performed 15 experiments on each Rdesired in numerical simulations, a total of
45 experiments were carried out. The numerical simulations
were performed on a PC cluster using 45 nodes with AMD
Opteron 248 CPU over three days. Note that the learning algorithm itself is exactly the same for the case of straight walking,because leg trajectory modulation by introducing the Rdesired
parameter can be regarded as an environmental change for thelearning algorithm.
The learning algorithm successfully acquired a turning
walk with 78% acquisition rate on average in numerical simulations (Table 2). Figure 14 shows trajectory of the COM during the turning walk with respect to a ground-fixed world coordinate system. The simulated robot started walking at position 601 07 in the direction of the X axis with Rdesired 30931 0951 190 m. We can see that the robot followed circularFig. 13. Return map of the step length. We captured the step
length when the left leg touched down. Here dn and dn61 are
the step length just after the perturbation and the next step,
respectively. The solid line indicated by an arrow is linear approximation for all data points and the diagonal linear line represents the identity map.

Table 2. Achievements of acquisition.
Number of achievements
Number ofexperiments

Simulation(trials)

Hardware(trials)

0.3

15

11
(68)

5
(820)

0.5

15

12
(73)

7
(1,529)

1.0

15

12
(293)

10
(2,355)

Rdesired

trajectories nicely, which were explicitly specified by Rdesired
parameter.

Endo, Morimoto, Nakanishi, Cheng, and Matsubara / Learning CPG-based Biped Locomotion with a Policy Gradient Method

225tion and a lateral motion due to a yaw rotational movement
around the center of turning. In addition, a smaller Rdesired
would lead to larger interference between the two motions.
Thus, the learning algorithm tried to utilize both 22 roll and 22 pitch
information to adapt to different Rdesired , suggesting that thelearning algorithm appropriately optimized the policy to negotiate with environmental changes.
We implemented the acquired 35 feedback controllers in
Table 2 on the hardware as in the case of straight walking andconfirmed that additional 2,062 trials on average in numericalsimulations were required for the policy to perform circular
walking in the physical environment. This result also suggests
the learning process gradually improves robustness against
perturbation in the physical environment. However, the rate ofsuccessful walking in the physical system decreases according with the decrease of Rdesired . The main reason would bemodeling error in numerical simulation, especially for groundreaction forces. Even in these cases, we can improve the policy through online learning with the hardware system, which
is discussed in the following section.
Fig. 14. Trajectories of COM with different Rdesired . The robot started walking at 6X1 Y 7 3 601 07 in the direction of thepositive X axis.

We implemented all of the acquired policies with different
numbers of trials on the real robot (over two days of experimentation). We investigated the numbers of achievements andthe required numbers of trials in numerical simulations, as in
the case of straight walking experiments. In Table 2, the number of required trials to acquire a turning walk was much lessthan in the case of straight walking. A possible reason is thatthe walking velocity of a turning walk is relatively smaller than
the straight walking owing to slip between the stance leg andthe ground, which is caused by modeling error of frictional
forces in the horizontal direction. As robustness for stepping
motion in place is guaranteed by (14), the robot does not fall
over even when the robot does not have forward walking velocity. Thus, learning movement with lower walking velocity
can be regarded as an easier task. Therefore, the acquisitionof a turning walk becomes easier in comparison with straightwalking in numerical simulations.
Figure 15 shows typical results of the acquired CPG feedback policies for all Rdesired considered. As in the case ofstraight walking, we could not observe policy convergence for
failed experiments. The policies, which could not succeed on
the hardware experiment, had similar profiles to a typical policy. However the output amplitude was slightly smaller or significantly larger than the policy that is applicable to the hardware.
The smaller Rdesired utilizes 22 pitch more information compared with the straight walking case. The reason would be that
we cannot clearly decompose a turning walk into sagittal mo-

4.6. Online Learning Based on Obtained Policy
As shown in Tables 1 and 2, several policies obtained in numerical simulations could not achieve walking with the physical robot in the real environment. However, even under these
conditions, we could make use of an additional online learningon the real hardware. This is because the computational cost of
the numerical simulation is largely due to the dynamics calculation, rather than the learning algorithm itself. Thus, online
learning can be adapted. In this section, we attempt to improve
the obtained policies of the numerical simulations that could
not originally produce steady walking in the hardware experiments to an online learning scheme.
In the case of the reward function, we are required to provide forward walking velocity and body height. Therefore, it isdesirable for the robot to measure this information only by using equipped onboard sensors. We can derive the walking velocity from the relative velocity of the stance leg with respect
to the pelvis. This can be an estimate with inverse kinematics and numerical differentiation of the measured joint anglesof the stance leg. We introduced a first-order low-pass filter
with a cutoff frequency of 1 Hz to smooth out the velocity estimation. The body height was measured by the joint anglesof the stance leg and the absolute body inclination, which wasderived from integration of an internal gyration sensor.
Despite delayed and inaccurate reward information, the online learning algorithm successfully improved the initial policy and performed steady walking within 200 trials (which
took 2.5 h to perform). Figure 16 shows an example of online
learning for straight walking where k S 3 490, kh 3 1090 andh 5 3 09275. (Note that the value of the accumulated reward differs from the simulated result in Figure 9 owing to a different
time duration 16 s for one trial.) Figure 17 shows the circular

226

THE INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH / February 2008

Fig. 15. Typical acquired policy with different Rdesired . The smaller Rdesired utilizes 22 pitch more information compared with thestraight walking case.

Fig. 16. Additional online learning for straight walking. The
black and gray lines are the accumulated reward for one trial
and the running average of the accumulated reward for 20 trials, respectively.

Fig. 17. Additional online learning for circular walking. The
black and gray lines are the accumulated reward for one trial
and the running average of the accumulated reward for 20 trials, respectively.

Endo, Morimoto, Nakanishi, Cheng, and Matsubara / Learning CPG-based Biped Locomotion with a Policy Gradient Method
walking case where k S 3 1090, kh 3 1090 and h 5 3 09275 witha time duration of 20 s for one trial. The gray line indicates therunning average of the accumulated reward for 20 trials. These
results show the efficiency of learning speed, which is in practice applicable to the physical system.

V 6x9 wc 7 3

In this paper, we have proposed an efficient learning framework for CPG-based biped locomotion control using the policy gradient method. We have decomposed the walking motion into a stepping motion in place, with a propulsive motion, while the feedback pathways for the propulsive motion
were acquired through the proposed policy gradient method.
Despite the considerable number of hidden variables, the proposed framework has successfully obtained a steady walking
pattern for straight walking within 1,000 trials, on average, inthe simulation. The acquired feedback controllers have been
implemented on a 3D hardware robot and demonstrated robust
walking in the physical environment. We have discussed velocity control and stability for straight steady walking, as wellas an extension to circular walking. Finally, we have demonstrated the possibility of online learning with the hardware robot.
To the best of our knowledge, our study is the first successful result to acquire biped locomotion that can be applied to afull-body hardware humanoid robot.
In this paper, we have only considered a policy gradient method based on Kimura’s update rule. A comparison with other alternative policy search approaches such as
value-function-based reinforcement learning (Doya 2000) andGPOMDP developed by Baxter and Bartlett (2001) will form
part of our future work.

Acknowledgments
We would like to thank Seiichi Miyakoshi of the Digital Human Research Center, AIST, Japan and Kenji Doya of the Initial Research Project, Okinawa Institute of Science and Technology, for productive and invaluable discussions. We would
like to thank all of the persons concerned in the hardware development of the humanoid platform for supporting this research.

Appendix: Function Approximator for the Value
Function and the Policy
We use a NGnet (Doya 2000) to model the value function andthe mean of the policy. The variance of the policy is modeledby a sigmoidal function (Kimura and Kobayashi 19981 Peters
et al. 2003). The value function is represented by the NGnet:

4 kc bk 6x71

(30)

k31

where
bk 6x7 3

5. Conclusion

K
1

227

k 6x7
1
K
l31 l 6x7

k 6x7 3 e

4skT 6x4ck 7

(31)

and k is the number of the basis functions. The vectors ck andsk characterize the center and the size of the kth basis function,
respectively. The mean  and the variance  of the policy are
represented by the NGnet and the sigmoidal function,K
1

4 ij bi 6x71

(32)

1
1
1 6 exp644 j 7

(33)

j 3

i31

and
j 3

respectively. We assigned basis functions k 6x7 at even intervals in each dimension of the input space 64290 
22 roll 1 22 pitch  2907. We used 225 (315  15) basis functions
to approximate the value function and the policy.

References
Aoi, S. and Tsuchiya, K. (2005). Locomotion control of abiped robot using nonlinear oscillators. Autonomous Robots, 19(3): 219–232.
Baxter, J. and Bartlett, P. L. (2001). Infinite-horizon policygradient estimation. Journal of Artificial Intelligence Research, 15: 319–350.
Benbrahim, H. and Franklin, J. A. (1997). Biped dynamic
walking using reinforcement learning. Robotics and Autonomous Systems, 22: 283–302.
Cohen, A. H. (2003). Control principle for locomotion—
looking toward biology. Proceedings of the 2nd International Symposium on Adaptive Motion of Animals and Machines (AMAM’03), Kyoto, Japan (CD-ROM, TuP-K-1).
Cohen, A. H. and Boothe, D. L. (1999). Sensorimotor interactions during locomotion: principles derived from biologicalsystems. Autonomous Robotics, 7(3): 239–245.
Doya, K. (2000). Reinforcement learning in continuous timeand space. Neural Computation, 12: 219–245.
Endo, G. et al. (2004). An empirical exploration of a neuraloscillator for biped locomotion control. Proceedings ofthe 2004 IEEE International Conference on Robotics
and Automation (ICRA’04), New Orleans, LA, pp. 3036–
3042.
Endo, G. et al. (2005a). Learning CPG sensory feedback with
policy gradient for biped locomotion for a full-body humanoid. Proceedings of the 20th National Conference on

228

THE INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH / February 2008

Artificial Intelligence (AAAI-05), Pittsburgh, PA, pp. 1267–
1273.
Endo, G. et al. (2005b). Experimental studies of a neural oscillator for biped locomotion with QRIO. Proceedings ofthe 2005 IEEE International Conference on Robotics andAutomation (ICRA’05), Barcelona, Spain, pp. 598–604.
Grillner, S. et al. (1995). Neural networks that co-ordinate locomotion and body orientation in lamprey. Trends in NeuroSciences, 18(6): 270–279.
Hase, K. and Yamazaki, N. (1997). A self-organizing model
to imitate human development for autonomous bipedal
walking. Proceedings of the 6th International Symposiumon Computer Simulation in Biomechanics, Tokyo, Japan,
pp. 9–12.
Hase, K. and Yamazaki, N. (1998). Computer simulation of
the ontogeny of biped walking. Anthropological Science,
106(4): 327–347.
Hirai, K. et al. (1998). The development of Honda Humanoid
Robot. Proceedings of the 1998 IEEE International Conference on Robotics and Automation, Leuven, Belgium,
pp. 1321–1326.
Hirukawa, H. et al. (2004). Humanoid robotics platforms developed in HRP. Robotics and Autonomous Systems, 48(4):165–175.
Ishiguro, A., Fujii, A. and Hotz, P. E. (2003). Neuromodulated
control of bipedal locomotion using a polymorphic CPG
circuit. Adaptive Behavior, 11(1): 7–17.
Kimura, H. and Kobayashi, S. (1998). An analysis of actor/critic algorithms using eligibility traces: reinforcement
learning with imperfect value function. Proceedings ofthe 15th International Conference on Machine Learning(ICML-98), Madison, WI, pp. 278–286.
Kimura, H., Fukuoka, Y. and Cohen, A. H. (2007). Biologically inspired adaptive dynamic walking of a quadruped
robot. Philosophical Transactions of The Royal Society A
365(1850): 153–170.
Konda, V. R. and Tsitsiklis, J. N. (2003). On actor–critic algorithms. SIAM Journal on Control and Optimization, 42(4):1143–1166.
Kuroki, Y. et al. (2001). A small biped entertainment robot. Proceedings of the 2001 IEEE-RAS International Conference on Humanoid Robots (Humanoids2001), Tokyo,
Japan, pp. 181–186.
Matsubara, T. et al. (2006). Learning CPG-based biped locomotion with a policy gradient method. Robotics and Autonomous Systems, 54: 911–920.
Matsuoka, K. (1985). Sustained oscillations generated by mutually inhibiting neurons with adaptation. Biological Cybernetics, 52: 345–353.

McGeer, T. (1990). Passive dynamic walking. International
Journal of Robotics Research, 9(2): 62–82.
McMahon, T. A. (1984). Muscles, Reflexes, and Locomotion.
Princeton, NJ, Princeton University Press.
Miyakoshi, S. et al. (1998). Three dimensional bipedal stepping motion using neural oscillators—towards humanoid
motion in the real world. Proceedings of the 1998 IEEE/RSJ
International Conference on Intelligent Robots and Systems(IROS’98), Victoria, BC, pp. 84–89.
Mori, T. et al. (2004). Reinforcement learning for a CPGdriven biped robot. Proceedings of the 19th National Conference on Artificial Intelligence (AAAI’04), San Jose, CA,
pp. 623–630.
Morimoto, J. et al. (2005). Poincare-map-based reinforcement learning for biped walking. Proceedings of the 2005
IEEE International Conference on Robotics and Automation (ICRA’05), Barcelona, Spain, pp. 2381–2386.
Nishiwaki, K. et al. (2000). Design and development of research platform for perception–action integration in humanoid robot: H6. Proceedings of the 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS’00), Takamatsu, Japan, pp. 1559–1564.
Orlovsky, G. N., Deliagina, T. G. and Grillner, S. (1999). Neuronal Control of Locomotion: From Mollusc to Man. Oxford, Oxford University Press.
Park, I. et al. (2005). Mechanical design of humanoid robot
platform KHR-3 (KAIST Humanoid Robot—3: HUBO).
Proceedings of the 2005 IEEE-RAS International Conference on Humanoid Robots (Humanoids2005), Tsukuba,
Japan, pp. 321–326.
Peters, J., Vijayakumar, S. and Schaal, S. (2003). Reinforcement learning for humanoid robots—policy gradients andbeyond. Proceedings of the 3rd IEEE International Conference on Humanoid Robots (Humanoids2003), Karlsruhe
and Munich, Germany (CD-ROM).
Sutton, R. S. et al. (2000). Policy gradient methods for reinforcement learning with imperfect value function. Advances in Neural Information Processing Systems, 12:1057–1063.
Taga, G. (1995). A model of the neuro-musculo-skeletal system for human locomotion I. Emergence of basic gait. Biological Cybernetics, 73: 97–111.
Tedrake, R., Zhang, T. W. and Seung, H. S. (2004). Stochastic policy gradient reinforcement learning on a simple 3D
biped. Proceedings of the 2004 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS’04),
Sendai, Japan, pp. 2849–2854.
Williamson, M. (1998). Neural control of rhythmic arm movements. Neural Networks, 11(7–8): 1379–1394.