Reinforcement Learning for Balancer Embedded Humanoid Locomotion
Akihiko Yamaguchi, Sang-Ho Hyon, and Tsukasa Ogasawara

Abstract— Reinforcement learning (RL) applications inrobotics are of great interest because of their wide applicability,
however many RL applications suffer from large learning costs.
We study a new learning-walking scheme where a humanoid
robot is embedded with a primitive balancing controller forsafety. In this paper, we investigate some RL methods forthe walking task. The system has two modes: double stanceand single stance, and the selectable action spaces (sub-action
spaces) change according to the mode. Thus, a hierarchical RL
and a function approximator (FA) approaches are compared insimulation. To handle the sub-action spaces, we introduce thestructured FA. The results demonstrate that non-hierarchical
RL algorithms with the structured FA is much faster than thehierarchical RL algorithm. The robot can obtain appropriate
walking gaits in around 30 episodes (20∼30 min), which isconsidered to be applicable to a real humanoid robot.

I. I NTRODUCTION
Designing behavior by only its objective is essential to
future robots, since this ability enables the end-users to easilyteach their wish to the robots. Reinforcement learning (RL)
is such a technology, so, RL applications in robotics are ofgreat interest. Additionally, model-free RL algorithms areapplicable without the dynamics model of the environment.
Hence, a lot of works have been done in applying RL
methods to robot control [1]∼[11]. In this paper, we focus
on the RL application to humanoid locomotion.
Past research on RL applications to locomotion utilizes
central pattern generators (CPGs) [4], [5], [7], or a propertyof passive dynamic walking [1], [6]. These methods restrict
the behavior of the robot to certain patterns or dynamics. But
such restriction is desirable for walking, which reduces thelearning time greatly.
In contrast, we study a new learning-walking scheme
where a humanoid robot embedded with a primitive balancing controller [12], [13] learns to walk (Fig. 1). This scheme
has two advantages: (1) the robot can learn in safety, and (2)the size of a state-action space can be reduced. The balancing
controller restricts the behavior of the robot to avoid falling
down. Using the remaining DoF (degree of freedom), therobot learns to walk. Actually, the robot can still move its
center of mass and a foot, which is capable to walk. However,Part of this work was supported by a Grant-in-Aid for JSPS, Japan Society
for the Promotion of Science, Fellows (22·9030)
A. Yamaguchi (JSPS Research Fellow) and T. Ogasawara are with
the Graduate School of Information Science, Nara Institute of Science and Technology, 8916-5, Takayama, Ikoma, Nara 630-0192, JAPAN{akihiko-y, ogasawar}@is.naist.jp

S. Hyon is with the Department of Robotics, Ritsumeikan University, 1-1-1 Noji Higashi, Kusatsu, Shiga 525-8577 JAPAN gen@fc.ritsumei.ac.jp
A. Yamaguchi and S. Hyon are also with the Computational Neuroscience Laboratories, Advanced Telecommunications Research InstituteInternational (ATR), Kyoto 619-0288, JAPANFig. 1.
Illustration of our learning-walking scheme. (a) A humanoid
robot embedded with a primitive balancing controller learns walking.
(b) Selectable actions in the double and the single stance modes. (c) Subaction spaces.

the dynamics of the robot with the controller becomes socomplex that we cannot easily identify the dynamics model.
Thus, it is difficult to design optimal walking gaits. Even so,
model-free RL methods are applicable to this case.
Compared to the CPG approaches, our scheme is considered to be safer since falling down is automatically prevented
by the balancing controller. In some passive dynamic walking
approaches, the falling down of the robot is avoided due toits hardware property. For instance, falling down of the robot
in [1] is a rare occasion. However, such kind of avoidance
restricts the variety of robot behavior a lot. In contrast, ourscheme is a software approach. That is, we can choose to
use the balancing controller or not, which keeps the varietyof the humanoid behavior.
In this paper, we investigate some applicable RL methods
for our scheme. The features of our RL task are as follows:(F1) The robot learns from scratch to obtain better performance than hand-coded policies.
(F2) The robot uses an on-line learning method (or batchmode of small sizes) to handle (F1) and reduce learningcost.
(F3) The robot has two modes: double stance and single stance, and the selectable action spaces (we call
sub-action spaces) change according to the mode (Fig.
1(b), (c))
To handle (F3), we consider a hierarchical RL approach([10], [14], [15]) and a structured function approximator(FA) approach. Kirchner applied a hierarchical version of

Q-learning (HQL) to a similar task, the forward movementof a six-legged robot [10]. However, in our task, only twolayers are needed to handle (F3). In this case, we can define asingle FA into which the sub-FAs (corresponding to the subaction spaces) are structured. Then we apply a normal (nonhierarchical) RL method to the structured FA. In general,if available prior knowledge of task is almost the same, ahierarchical RL method is inferior to a normal RL method
since the former one has to limit its algorithm for converge.
Thus, the structured FA is considered to be a better approach.
Thus, we compare Peng’s Q(λ)-learning [16] and fitted Q
iteration [17], [18] with the structured FA, and Cohen’shierarchical RL (HRL) algorithm [15].
Our simulation results demonstrate that the FA approach ismuch superior to the hierarchical approach in both learning
speed and the performance of the acquired walking gaits.
The robot can obtain appropriate walking gaits in around 30
episodes (20∼30 min), which is considered to be applicable
to a real humanoid robot.
In the rest of this paper, we describe the RL algorithms
used in this paper in section II, then define the FAs in section
III. We present the simulation results in section IV, andconclude the paper in section V.
II. R EINFORCEMENT L EARNING A LGORITHMS
The purpose of RL is for a learning system (agent) whose
input is a state xn ∈ X and a reward Rn ∈ R, and whose
output is an action un ∈ U, to acquire the policy π(xn ) :X[∑→ U that maximizes
the expected discounted return]∞E k=1 γ k−1 Rn+k where n ∈ N = {0, 1, . . . } denotes
the time step and γ ∈ [0, 1) denotes a discount factor. In
value-function-based RL algorithms, an action value function
Q(x, u) : X × U → R is learned to represent the expecteddiscounted return by taking an action u from a state x. Then,the optimal action rule is obtained from the greedy policy
π(x) = arg maxu Q(x, u).
Another approach to find the optimal policy is searching directly in the policy space. The Natural Actor Critic(NAC) [3] is a typical example. However, this kind of
approach strongly depends on the initial value of the policy
parameter, especially in large domains. In our learning-fromscratch case, we assume that the value-function-based RL
methods may obtain better policy since they can have richer
policy parameterization. Thus, we test Peng’s Q(λ)-learningalgorithm [16] and fitted Q iteration algorithm [17], [18].
In general, fitted Q iteration requires appropriate samples,but it is difficult to obtain such samples with a random
policy. In our preliminary experiments, though we applied
fitted Q iteration in batch mode of small sizes, we were
aware that Q(λ)-learning is better in the early stage oflearning. A possible reason is that since we reduce the setof basis functions of a FA to improve the learning speed,
the system loses the Markov property. Thus, we mainly use
Q(λ)-learning, and apply fitted Q iteration to the same FA
only with sample sequences of higher return.
In the rest of this section, we introduce these RL algorithms. Additionally, we briefly describe Cohen’s hierarchical

RL (HRL) algorithm [15].
A. Peng’s Q(lambda)-Learning Algorithm
The Peng’s Q(λ)-learning algorithm [16] is an on-line RL
method. The update procedure for a generic FA Q(x, u) of
the parameter θ ∈ Θ is written as follows:
en = Rn + γVn (xn+1 ) − Vn (xn ),e0n = Rn + γVn (xn+1 ) − Qn (xn , un ),θn+1 = θn + αen Tr n + αe0n ∇θ Qn (xn , un ),

(1a)
(1b)
(1c)

Tr n+1 = (γλ)(Tr n + ∇θ Qn (xn , un )),(1d)where ∇θ Q(x, u) denotes the derivative of Q(x, u) w.r.t.
the parameter θ, α denotes a step-size parameter, Tr denotes the eligibility trace (Tr 0 = 0 ∈ Θ), and Vn (x) ,maxu Qn (x, u). This update procedure is applied after eachaction.
B. Fitted Q Iteration Algorithm
The fitted Q iteration algorithm [17], [18] is a batch mode
RL method to learn from sample trajectories whose element
is a four-tuple Fn = (xn , un , xn+1 , Rn ). The idea of fitted Q
iteration is as follows: first, we build a training set from thecurrent FA and a set of four tuples. Then, we train the next
FA with the training set by a supervised learning method.
Iterating these two steps, the FA will converge to the action
value function.
The action value FA Q0 is initialized so that Q0 (x, u) = 0
for all (x, u) ∈ X × U. In the N -th iteration, the training set{in , on } is built from the current FA QN −1 and the set offour tuples {Fn } byin = (xn , un ),on = Rn + γ max QN −1 (xn+1 , u).
u

(2)
(3)

Then, QN is trained with {in , on }. We implement thissupervised learning with a gradient descent for the leastsquares.
As mentioned above, we combine fitted Q iteration andQ(λ)-learning. At the end of each action, we apply the update
rule of Q(λ)-learning and store the sample. At the end of
every NFQI episodes, we execute an iteration of fitted Q
iteration with the samples in the top Nsmpl episodes ranked
by the return of the episode.
C. Cohen’s Hierarchical Reinforcement Learning
As a hierarchical RL method, we use Cohen’s hierarchical
RL (HRL) [15]. Many hierarchical RL methods require that atask can be decomposed into sub-tasks. But, in our case, weonly design a single reward function. Thus, the hierarchical
RL methods requiring sub-tasks are not applicable to ourcase. However, Cohen’s HRL uses a single reward function,
which is suitable for our case. Thus, we choose it.
Though Cohen’s HRL is developed for discrete stateaction spaces, we extend this algorithm in a straightforward
way so that a FA is available for each module.
For our task, we construct a two layer modular structure.
Specifically, it has one higher module and several lower

modules. Each lower module has its unique sub-action space.
The higher module selects a lower module as an action.
III. F UNCTION A PPROXIMATORS
Next, we describe some FAs for the action value functions
Q(x, u). For a continuous state space X and a discrete action
space U, we use a linear FA because of its stability. If boththe state and the action spaces are continuous, we employ
wire-fitting [19]. The remarkable feature of wire-fitting isthat we can maximize Q(x, u) w.r.t. u by evaluating only on
a fixed number of points. We define the structured FA for our
walking task, where the action space consists of discrete sets
and continuous vector spaces whose selectability depends on
the state.
A. Linear Function Approximator
For a continuous state x ∈ X and a discrete actionu ∈ U, we let Q(x, u) = θu> φ(x), where φ(x) =(φ1 (x), . . . , φ|K| (x))> denotes the feature vector of a statex, K = {φk | k = 1, 2, ..} denotes a set of basis functions,and θu ∈ R|K|×1 denotes a parameter related to an action u.
> >The parameter vector is defined as θ = (θ1> , . . . , θ|U∈|)|K||U |×1
R
. The derivative of the Q(x, u) w.r.t. θ is given> >by ∇θ Qn (x, u) = (δ1u φ>
1 , . . . , δ|U|u φ|U | ) where δ denotes
the Kronecker’s delta.
As basis functions, we use Normalized Gaussian Network(NGnet) [20] which is sometimes used as the basis functionsof FAs in RL applications [11]. In NGnet, φk (x) is given byG(x; µk , Σk )φk (x) = ∑,000k ∈K G(x; µk , Σk )(4)

where G(x; µ, Σ ) denotes a Gaussian with mean µ andcovariance matrix Σ . In the case of a linear FA, K is predefined and {µk , Σk | k ∈ K} are treated as fixed parameters.
In the following, a linear FA with NGnet is referred to as
LFA-NGnet.
B. Wire-Fitting
For a continuous state x ∈ X and a continuous actionu ∈ U, wire-fitting is defined as:∑−1
qi (x)i∈W (di + )∑Q(x, u) = lim
,
(5)
−1
(d
+
)
→0+
i∈W i
[
]
di = ku − ui (x)k2 + C max(qi0 (x)) − qi (x) . (6)0i ∈WHere, a pair of the functions qi (x) : X → R and ui (x) :X → U (i ∈ W) is called a control wire; wire-fitting isregarded as an interpolator of the set of control wires W. C
is a smoothing factor of the interpolation; we choose C =0.001 in the experiments. Any FA is available for qi (x) andui (x). Regardless of the kind of the FAs, one of qi (x), i ∈ W
is equal to maxu Q(x, u) and the corresponding ui (x) is thegreedy action at x.
We use NGnet for qi (x) and a constant vector for ui (x),that is, we let qi (x) = θi> φ(x) and ui (x) = Ui , where φ(x)is the feature vector of the NGnet. The parameter vector θ is>>defined as θ> = (θ1> , U1> , θ2> , U2> , . . . , θ|W|
, U|W|), and thegradient ∇θ Q(x, u) can be calculated analytically.

As an exploration policy in using wire-fitting, we use theBoltzmann-like selection method proposed in [21]. A control
wire i is considered to be a discrete action whose action
value is qi (x), and one of the control wires is chosen by
Boltzmann selection. Then the corresponding ui (x) is theselected action. We refer to this method as WF-Boltzmann.
C. Structured Function Approximator
Next, we define a FA for our walking task where theaction space consists of discrete sets and continuous vector
spaces whose selectability depends on the state. Such anaction space can be defined as a direct sum of discretesets andFvector spaces. Thus, we denote the action spaceas U = p∈P Up , where Up is a sub-action space and P isa set of sub-space indexes. Again, see Fig. 1(b), (c) as an
example of a set of sub-action spaces. In the following, we
denote u = (p, up ) ∈ U for convenience. We use P(x) ⊆ P
to express the selectable sub-action spaces at a state x.
We simply define a FA over the action space U by
structuring (combining) sub-action value functions. First ofall, we define each sub-action value function Qp (x, up ) over
X × Up . We choose the LFA-NGnet for a discrete set Up ,and wire-fitting for a continuous space Up . Then we define
the overall Q as∑
Q(x, u) ,δpp0 Qp0 (x, up0 ) = Qp (x, up ),
(7)
p0 ∈P

where u = (p, up ). We let θp the parameter of a sub-action
value function Qp . The parameter vector of the Q can be> >) . The derivative of Q w.r.t. θ
defined as θ = (θ1> , . . . , θ|P|is given by(∇θ Q(x, u)> = δp1 ∇θ1 Q1 (x, u1 )> ,). . . , δp|P| ∇θ|P| Q|P| (x, u|P| )> ,(8)
where u = (p, up ).
The greedy action at x can be given by
u? =

arg
max
F

u∈

Q(x, u).

(9)p∈P(x) Up

This can be evaluated as follows: (1) calculating ûp =arg maxup ∈Up Qp (x, up ) for all p ∈ P(x), (2) calculatingu? = arg maxu∈{ûp } Q(x, u).
As an exploration policy, we define a two-stage action
selection method so that the RL agent can explore as broadly
as possible and it has a scalability for any kind of sub-FAs. In
the first stage, for each p ∈ P(x), select a sub-action ûp from
Up based on Qp (x, up ). Here, we use Boltzmann selection
if Qp is the LFA-NGnet and WF-Boltzmann selection if Qp
is wire-fitting.
In the second stage, select an action u from {ûp |p ∈ P(x)}based on their action values {Qp (x, ûp )}. We use a weighted
version of Boltzmann selection to consider the size of Up sothat the RL agent can broadly explore. That is,
wp exp( τ1 Qp (x, ûp ))
,
1
0
0
0
p0 ∈P(x) wp exp( τ Qp (x, ûp ))π(ûp |x) = ∑(10)    

z
x

(a)

Fig. 3.

y(b)Illustration of the sub-action spaces of the walking task.

(c)Fig. 2.
SARCOS biped humanoid robot developed by NICT/ATR.
(a) Hardware. (b) DoF configuration. (c) Simulation model.

where wp denotes the weight to compensate the size of Up .
We decide the weights {wp } so that they are proportional to
the size of the action set Up if Up is discrete, or to the numberof the control wires of Qp if Qp is wire-fitting. Note that in
the early stage of learning, the probability of ûp is nearly
proportional to wp , which makes the exploration appropriate
for the size of Up .
IV. E XPERIMENTS : L EARNING WALKING OF A
H UMAN -S IZE H UMANOID ROBOT
We apply the RL methods mentioned above to the walking
task of a human-size biped humanoid robot shown in Fig. 2.
The humanoid robot has 50 DoF and torque controllability
with hydraulic actuation [22]. Its height is 1.58 m, and its hipheight is 0.82 m when at an upright posture. It weighs 93.7
kg. Its DoF configuration is shown in Fig. 2(b). The arms
and legs each have 7 DoFs, and the neck and torso eachhave 3 DoFs. In this section, we demonstrate a simulationcomparison of the RL methods. In the experiments, weuse a dynamics simulator with a presice model (Fig. 2(c))
including a well-tuned contact model.
A. Robotic System Setup for RL
The robot is embedded with the balancing controller [13].
It regulates the center of mass (CoM) to the center ofthe supporting region through the optimal force control.
For biped walking case, the desired CoM, as well as the
position of the swinging foot should be varied. Although thecontroller has compliant stabilization and terrain-adaptation
abilities, its performance is not satisfactory for dynamic
situations because of sensory delays and limitations in the
low-level joint controllers. Thus, we investigate which RL
method is most appropriate for this situation.
The state given to the RL agent consists of the stancemode, {left, double, right}, the position of the CoM, and theprevious stance mode. We also tested the velocity of the CoM
instead of the previous stance mode. But, the velocity is so
sensitive to sensor noise and is therefore not suitable for thereal robot case. Thus, the state x is defined as
x = (mode st , xcm , ycm , mode pst ).

 

(11)

In the following experiments, we allocate 405 Gaussians on
a 3 × 9 × 5 × 3 grid as the basis functions of the NGnet.

Though the robot with the balancing controller can movethe position of the CoM and the swinging foot, we preimplement some primitive CoM movements so that the RL
agent can learn the task easily. Thus, the available subaction spaces are defined as illustrated in Fig. 3. Specifically,
the available sub-action space in the double stance mode isdefined as follows.
Uchst = {to-R, to-L} : Changing the stance mode to right orleft. Specifically, the position of CoM is moved to above
the left/right foot, then the other foot is raised up. These
actions are executed in 1.0 second respectively.
And the available sub-action spaces in the single stance modeare defined as follows.
Udwnft = {down} : Putting down the swinging foot. Thisaction is executed in 1.0 second.
Usf = {(∆xsf , ∆ysf ) | ∆xsf , ∆ysf ∈ R} : Moving xsf andysf which denote the x and y positions of the swinging
foot. ∆xsf and ∆ysf denote their differences. This action
is executed in 0.1 second.
Namely, the action space is U = Uchst t Udwnft t Usf , andthe selectable action spaces at a state x can be written by{{chst}
if mode st = double,
P(x) =(12){dwnft, sf} if mode st = left or right.
B. RL Methods Configurations
For the discrete action spaces Uchst and Udwnft , we define
Qchst and Qdwnft as the LFA-NGnet, respectively. For thecontinuous action space Usf , we define Qwf
sf as wire-fitting.
For comparison, we also define Qdisc
asthe
LFA-NGnet oversf
Usfdisc defined by discretizing Usf with a 3 × 3 grid. The
parameters of every LFA-NGnet are initialized to zero. About
Qwf
sf , {θi |i ∈ W} are initialized to zero, while {Ui |i ∈ W}
are initialized with points of a 3 × 3 grid on Usf .
In this experiments, we compare the following combinations of the RL algorithms and the sub-FAs1 .
S-WF-QL : Q(λ)-learning for Qwf where Qchst , Qdwnft , andQwf
sf are structured.
S-WF-QLFQI : The combination of Q(λ)-learning and fitted
Q iteration for Qwf .
S-DISC-QL : Q(λ)-learning for Qdisc where Qchst , Qdwnft ,and Qdiscare structured.
sf
S-DISC-QLFQI : The combination of Q(λ)-learning andfitted Q iteration for Qdisc .
S-DISC-Q0LFQI : The combination of Q(0)-learning andfitted Q iteration for Qdisc .
1 We used the RL library, SkyAI: skyai.sourceforge.net

HRL : Cohen’s HRL for a two layer modular structure where
the lower modules learn Qchst , Qdwnft , and Qdisc
sf . The
higher module learns the policy to select a lower module.
The S-DISC-Q0LFQI is compared to verify the effect of theeligibility trace (λ). The reason why Qdisc
is used in thesf
HRL rather than Qwf
is
due
to
the
stabilityofthe linear FA.
sf
For every RL method, we set γ = 0.95. We
use a decreasing step size parameter α = max(0.05,
0.3 exp(−0.002Neps )) for Q(λ)-learning and the HRL. Neps
denotes a number of episodes. For fitted Q iteration, we use
a constant step size parameter α = 0.05. For Q(λ)-learning,
we set λ = 0.9 and apply the replacing trace [23] to makethe eligibility trace stable (see also [24]). For the combinationof Q(λ)-learning and fitted Q iteration, we set NFQI = 3 andNsmpl = 10. As the exploration policy, we use Boltzmann(or Boltzmann-like) selection, with a decreasing temperatureτ = 1.0 exp(−0.002Neps ).
C. Task Setting
The objective of the walking task is to move forward
along the x-axis as far as possible. Though the balancing
controller is embedded, the robot still has a probability offalling down. The balancing controller employed in thispaper does not consider the swinging leg motions explicitly.
When the swinging leg moves too fast and CoP locates near
the supporting edge, then the robot can lose the stability.
Thus, we design the reward function as follows:r(t) = rmv (t) − rsc (t) − rfd (t),rmv (t) = 200vcmx (t)δt,
rsc (t) = 1δt,
{ 50 if falling-down,
rfd (t) =0otherwise,

Fig. 4. Resulting learning curves of the walking task. Each curve shows themean of the return over 10 runs per episode. The horizontal line (MANUAL)
shows the performance of a manually manipulated walking with a keyboard
interface.

(13a)
(13b)
(13c)
(13d)

where rmv means a reward for moving, rsc means a step
cost, and rfd means a penalty for falling down2 . The reward
function r(t) is integrated from the start time to the finish
time of an action, and given to the RL agent. Each episodestarts with the initial state where the robot is standing up(first snapshot in Fig. 6) and stationary, and ends if t > 75 s
or the robot is falling down.

Fig. 5. Trajectory of the CoM position. Small arrows indicate the directionof the movement. Note that the origin of the CoM is the center of the feet.

D. ResultFig. 4 shows the resulting learning curves of the walking
task (the mean of the return over 10 runs per episode).
The horizontal axis is logarithmic scale. The horizontal line
(MANUAL) shows the performance of a manually manipulated walking with a keyboard interface3 . Fig. 5 shows the
trajectory in an episode of the CoM position of a walking
gait acquired by the S-WF-QLFQI, and Fig. 6 shows thecorresponding snapshots.
The HRL and the S-DISC-Q0LFQI are very slow, thatis, they take a lot of episodes to acquire performance. The
possible reason is that the update rule of the HRL is similar to
2 Specifically, the falling-down is defined as (|φ| > 25.0◦ ) ∨ (|θ| >25.0◦ ) where φ and θ denote the x and y-Euler angles respectively.
3 The maximum return value in 30 trials is plotted.

Fig. 6.

Animation snapshots.

Q(0)-learning which may be poor for the walking task. About
the S-DISC-Q0LFQI, Q(0)-learning is dominant in the earlystage of learning rather than fitted Q iteration since suitablesamples are not obtained yet. Thus, the S-DISC-Q0LFQI is
considered to be as slow as the HRL.

All of the methods using Q(λ)-learning are much fasterthan these two methods. Thus, the eligibility trace (λ) is
considered to be very effective for the walking task.
The acquired performance of the S-WF-QLFQI is thebest, that is, it converges to the highest value of the return.
It substantially exceeds the performance of the manually
manipulated walking. Compared with Qdisc , the FA Qwf has
an ability to acquire better performance since Qwf directly
approximate over the continuous action space Usf by wirefitting. However, there is no performance difference between
the S-DISC-QL and the S-WF-QL because of the instabilityof wire-fitting caused by its nonlinearity. Thus, we consider
the reason for the best performance of the S-WF-QLFQIis due to both using wire-fitting and updating by fitted Q
iteration which is more stable than Q(λ)-learning. We should
also note that the S-WF-QLFQI is slightly slower than the
S-WF-QL. A possible reason is that Q(λ)-learning and fitted
Q iteration conflict since the definition of the action valuefunction is slightly different in the two algorithms.
V. C ONCLUSION AND F UTURE W ORK
We investigated some applicable RL methods for the
new learning-walking scheme where a humanoid robot isembedded with a primitive balancing controller. This scheme
has two advantages: (1) the robot can learn in safety, and(2) the size of the state-action space can be reduced. We
considered a hierarchical RL approach and FA approaches,and compared them in simulation. The results demonstrated
that Cohen’s hierarchical RL algorithm did not work well;it took long learning time. On the other hand, the structured
FA was defined for our situation. The RL methods basedon Peng’s Q(λ)-learning could obtain a suitable policy much
faster than the HRL. Especially, applying the combinationof fitted Q iteration and Q(λ)-learning to the structured FA
acquired the best performance. The reason is considered asboth the accuracy of wire-fitting and the stability of fitted Q
iteration.
Though we investigated for the walking task, our findings
are considered to be applicable to similar situations. Thatis, for a task where the action space consists of discrete sets
and continuous vector spaces whose selectability depends on
the state, the structured FA approach should be consideredrather than hierarchical RL methods. The hierarchical RL
methods have wider applicability, but in a specific situation,
specialized methods sometimes work more efficiently likethe structured FA.
We did not compare the policy gradient methods in thispaper, since we assumed the value-function-based RL methods can obtain better policy. However, testing some policy
gradient methods, such as NAC [3], is part of our futurework.
R EFERENCES[1] R. Tedrake, T. Zhang, and H. Seung, “Stochastic policy gradientreinforcement learning on a simple 3d biped,” in the IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS’04),
vol. 3, 2004, pp. 2849–2854.
[2] H. Kimura, T. Yamashita, and S. Kobayashi, “Reinforcement learningof walking behavior for a four-legged robot,” in Proceedings of the40th IEEE Conference on Decision and Control, 2001.

[3] J. Peters, S. Vijayakumar, and S. Schaal, “Reinforcement learningfor humanoid robotics,” in Humanoids2003, IEEE-RAS International
Conference on Humanoid Robots, 2003.
[4] T. Matsubara, J. Morimoto, J. Nakanishi, M. Sato, and K. Doya,“Learning CPG-based biped locomotion with a policy gradientmethod,” Robotics and Autonomous Systems, vol. 54, no. 11, pp.
911–920, 2006.
[5] L. Righetti and A. Ijspeert, “Programmable Central Pattern Generators:
an application to biped locomotion control,” in the IEEE Internactional
Conference in Robotics and Automation (ICRA’06), 2006.
[6] K. Hitomi, T. Shibata, Y. Nakamura, and S. Ishii, “Reinforcement
learning for quasi-passive dynamic walking of an unstable bipedrobot,” Robotics and Autonomous Systems, vol. 54, no. 12, pp. 982–
988, 2006.
[7] Y. Nakamura, T. Mori, M. Sato, and S. Ishii, “Reinforcement learning
for a biped robot based on a CPG-actor-critic method,” Neural
Networks, vol. 20, no. 6, pp. 723–735, 2007.
[8] T. Matsubara, J. Morimoto, J. Nakanishi, S. Hyon, J. G. Hale,and G. Cheng, “Learning to acquire whole-body humanoid CoM
movements to achieve dynamic tasks,” in the IEEE Internactional
Conference in Robotics and Automation (ICRA’07), 2007, pp. 2688–
2693.
[9] A. Yamaguchi, J. Takamatsu, and T. Ogasawara, “Constructing action
set from basis functions for reinforcement learning of robot control,”in the IEEE Internactional Conference in Robotics and Automation(ICRA’09), Kobe, Japan, 2009, pp. 2525–2532.
[10] F. Kirchner, “Q-learning of complex behaviours on a six-legged
walking machine,” Robotics and Autonomous Systems, vol. 25, no.
3-4, pp. 253–262, 1998.
[11] J. Morimoto and K. Doya, “Acquisition of stand-up behavior by areal robot using hierarchical reinforcement learning,” Robotics andAutonomous Systems, vol. 36, no. 1, pp. 37–51, 31 July 2001.
[12] S. Hyon, J. Hale, and G. Cheng, “Full-body compliant humanhumanoid interaction: Balancing in the presence of unknown external
forces,” Robotics, IEEE Transactions on, vol. 23, no. 5, pp. 884–898,
2007.
[13] S. Hyon, “Compliant terrain adaptation for biped humanoids without
measuring ground surface and contact forces,” Robotics, IEEE Transactions on, vol. 25, no. 1, pp. 171–178, 2009.
[14] A. G. Barto and S. Mahadevan, “Recent advances in hierarchicalreinforcement learning,” Discrete Event Dynamic Systems, vol. 13,
no. 4, pp. 341–379, 2003.
[15] S. Cohen, O. Maimon, and E. Khmlenitsky, “Reinforcement learningwith hierarchical decision-making,” in ISDA ’06: Proceedings of the
Sixth International Conference on Intelligent Systems Design andApplications. USA: IEEE Computer Society, 2006, pp. 177–182.
[16] J. Peng and R. J. Williams, “Incremental multi-step Q-learning,” inInternational Conference on Machine Learning, 1994, pp. 226–232.
[17] D. Ernst, P. Geurts, and L. Wehenkel, “Iteratively extending time
horizon reinforcement learning,” in Proceedings of the 14th European
Conference on Machine Learning, N. Lavra, L. Gamberger, andL. Todorovski, Eds. Dubrovnik, Croatia: Springer-Verlag Heidelberg,
September 2003, pp. 96–107.
[18] ——, “Tree-based batch mode reinforcement learning,” Journal of
Machine Learning Research, vol. 6, pp. 503–556, 2005.
[19] L. C. Baird and A. H. Klopf, “Reinforcement learning withhigh-dimensional, continuous actions,” Wright Laboratory, WrightPatterson Air Force Base, Tech. Rep. WL-TR-93-1147, 1993.
[20] M. Sato and S. Ishii, “On-line EM algorithm for the normalized
Gaussian network,” Neural Computation, vol. 12, no. 2, pp. 407–432,
2000.
[21] A. Yamaguchi, J. Takamatsu, and T. Ogasawara, “Constructing
continuous action space from basis functions for fast and stablereinforcement learning,” in the 18th IEEE International Symposium
on Robot and Human Interactive Communication (RO-MAN’09),
Toyama, Japan, 2009, pp. 401–407.
[22] M. Kawato, “From ‘understanding the brain by creating the brain’
towards manipulative neuroscience,” Phil. Trans. R. Soc. B, vol. 363,
no. 1500, pp. 2201–2214, 2008.
[23] S. P. Singh and R. S. Sutton, “Reinforcement learning with replacing
eligibility traces,” Machine Learning, vol. 22, no. 1-3, pp. 123–158,
1996.
[24] J. N. Tsitsiklis and B. V. Roy, “An analysis of temporal-difference
learning with function approximation,” IEEE Transactions on
Automatic Control, vol. 42, no. 5, pp. 674–690, 1997.