Learning Strategy Fusion to Acquire Dynamic Motion
Akihiko Yamaguchi (JSPS Research Fellow), Jun Takamatsu, and Tsukasa Ogasawara
Graduate School of Information Science
Nara Institute of Science and Technology, 8916-5, Takayama, Ikoma, Nara 630-0192, JAPAN
Email: {akihiko-y, j-taka, ogasawar}@is.naist.jp

Abstract—This paper proposes a method to fuse learning
strategies (LSs) in a reinforcement learning framework. In
this method, some LSs are integrated for learning a single
task of a single robot. The LSs consists of (1) LS-scratch:learning a policy from scratch, (2) LS-accelerating: learning a
policy from a previously learned policy by accelerating motionspeed parameters, and (3) LS-freeing: learning a policy froma previously learned policy by increasing the DoF (degree of
freedom). The proposed LS fusion method enables (A) in theearly stage of learning, LS fusion can select a suitable DoF
configuration from a predefined set of DoF configurations, and(B) after a behavior module that learns from scratch converges,
the LSs are applied to improve the policy. As a result, a robot canlearn a complex task by starting with a simplified configuration,and then transferring the learned behaviors while increasing thedifficulty. We introduce WF-DCOB proposed by Yamaguchi etal. for the LSs. We verify the proposed LS fusion method with acrawling task of a humanoid robot. The simulation experiments
demonstrate the advantage of the proposed method compared tolearning with a single learning module.

I. I NTRODUCTION

Fig. 1. Example learning process of the learning strategy (LS) fusion method.
Multiple LSs are applied to a single task multiple times. LS-scr, LS-accel,and LS-free denote LS-scratch, LS-accelerating, and LS-freeing respectively.
UCB means the UCB-Boltzmann selection method which chooses a behavior
module (B1, B2,...) that decides the actual movement of the robot. Behavior
modules that have potentially high performance are used and trained preferentially.

Designing a behavior by only its objective is essential
for future robots, since this ability enables the end-users toteach their wish to the robots easily. Reinforcement Learning(RL) is such a technology. There are many RL applications
to robotics research [1]∼[7]. However, RL methods require
a lot of learning cost in large domains, such as a motion
learning task of a humanoid robot. Many researchers aretackling to this issue, and some effective learning strategies(LSs) are proposed: dimension reduction [3], model utilization[4], hierarchical structure [5], imitation of the others [6], andtransferring (reusing) previously learned knowledge [7].
Though these LSs are defined and used individually, we
think that humans probably use multiple LSs for a single task
multiple times. Imagine learning a tennis swing. We first swing
a tennis racket slowly. After a good swing form is acquired,we speed up the swing, then continue to learn. In an RL sense,an accelerating LS is used in this learning; this LS transfers
the policy of the slow swing to the policy of the faster swing.
Note that we may use this LS multiple times. Other example
is found in an infant’s learning walking model [8], where theinfant starts from a lower DoF (degree of freedom), then learns
in a higher DoF. In an RL sense, a freeing LS is used whichtransfers the policy of the lower DoF to the policy of the higher
DoF. In addition, we can naturally infer that the accelerating
LS may be also used in learning walking. Though these are
only a few examples, we consider that using multiple LSs fora task multiple times is a key technique to learn a policy in alarge domain.
In this paper, we propose the learning strategy fusion
method, which automatically applies multiple LSs to a task
multiple times during its learning process. In addition, we
propose some LSs that are compatible with the LS fusion
method.
To accomplish the LS fusion method, we develop a systemthat maintains multiple policies rather than a single policy,
in order to test several configurations of an LS and return
to a previous policy if a transfer LS does not improve theperformance. In our system, each policy is maintained bya behavior module that consists of an RL method (an RL
algorithm and a function approximator) and its parameters. In
this system, an LS is defined as a method that generates abehavior module.
Now the problems that should be solved to accomplish LS
fusion are: (1) how to select a behavior module that actuallycontrols the robot and updates its policy from the samples,(2) how to design each LS, and (3) when the LSs are applied.
In the following three paragraphs, we describe the outline of
the solutions.
We employ the upper-confidence-bound (UCB) value of the
return to evaluate each behavior module. Here, the return isthe amount of reward in an episode, and the UCB value isthe sum of the return and its standard deviation. The returnindicates the performance of a behavior module, thus the UCB
value indicates the performance including an expectation of
the improvement. In addition, rather than selecting a behavior
module that has the maximum UCB value, we utilize theBoltzmann selection method to introduce randomness. We
refer to this method as the UCB-Boltzmann selection method.
In this paper, we define three LSs:
LS-scratch generates a behavior module that learns a task
from scratch.
LS-accelerating generates a behavior module by accelerating
the motion of a source behavior module.
LS-freeing generates a behavior module by increasing the DoF
of a source behavior module.
As an RL method of LS-scratch, we utilize WF-DCOB proposed by Yamaguchi et al. [1], which learns a policy to selecta continuous action and is efficient in a learning-from-scratchcase. In addition, the parameters of the function approximatorin WF-DCOB encode the speed of motion and the target joint
angles, which enables us to define LS-accelerating and LSfreeing easily.
If the system can apply the LSs in every stage of thelearning, the number of behavior modules becomes large.
Therefore, we introduce two principles in applying the LSs.
One is preventing to apply an LS of the same configuration.
The other is that only when a behavior module is trained
enough, a transfer LS (LS-accelerating or LS-freeing) is applied to it.
Fig. 1 illustrates an example learning process where the LS
fusion method is applied to a crawling task of a humanoid
robot. First, a behavior module is generated by LS-scratch(B1). After B1 is trained enough, B2 is generated by applying
LS-accelerating to B1. Similarly, B3 is generated by applying
LS-freeing to B2. A behavior module is selected by the
UCB-Boltzmann selection method. If the performance of atransferred module is not improved, UCB-Boltzmann selects
a previous behavior module.
We apply the proposed method to a crawling task of a small
size humanoid robot. The simulation experiments demonstrate
the advantage of LS fusion compared to learning with a single
behavior module.
In Section II, we discuss the related works. In Section III,
we briefly introduce RL. We propose the LS fusion method in
Section IV, and define the LSs in Section V using WF-DCOB.
In Section VI, the simulation experiments are presented. Finally, we conclude in Section VII.
II. R ELATED W ORKS
LS fusion is considered to be a learning architecture thatconsists of multiple RL modules for a single robot and a single
task, and allows behavior transfer during a single learning
process. The most remarkable feature compared to the otherworks is that the LSs transfer not only policy parameters, butalso physical limitation of the policy. LS-freeing changes the
DoF. LS-accelerating changes not only the speed parameters,
but the constraints of the speed parameters. Namely, theseLSs modify the task domain (the state-action space and the
dynamics), which has a probability to increase performance.
Uchibe et al. [9] proposed the Cooperative-CompetitiveConcurrent Learning with Importance Sampling (CLIS) architecture, where multiple RL modules sharing the same sensorymotor system learn for the same task simultaneously by using
importance sampling. The similarity to LS fusion is employing
multiple modules for a single task. CLIS architecture isconsidered to be transferring samples obtained from a module
to the other modules. Thus, the architecture allows concurrent
learning. On the other hand, LS fusion transfers a policy
in a behavior module to a new module where some motion
parameters, such as DoF and speed parameters, are changed.
Thus, the new behavior module has a probability to increase
the performance of the old module.
Fernández et al. [10] proposed a method to probabilisticallyreuse the policies learned in the other tasks. Later, Fernández
et al. [11] extended their Policy Reuse so that each policy istransferred when reusing. LS fusion is considered to be reusing
policies, so these two approaches are similar. The differenceis that the methods in [10], [11] transfer a policy between
several tasks, while LS fusion transfers a policy in a single
task. In addition, LS fusion modifies the motion parameters
including the physical limitation, while the method in [10]does not change the state-action space and the dynamics.
These differences also appear in the other transfer learning
methods (e.g. [7], [12]).
III. R EINFORCEMENT L EARNING
The purpose of RL is for a learning system (agent) whose
input is a state xn ∈ X and a reward rn ∈ R, andwhose output is an action un ∈ U, to acquire the policyπ(xn ) : X → U that] maximizes the expected discounted return[∑∞E k=1 γ k−1 rn+k where n ∈ N = {0, 1, . . . } denotes
the time step and γ ∈ [0, 1) denotes a discount factor. In
value-function-based RL algorithms, an action value function
Q(x, u) : X × U → R is learned to represent the expecteddiscounted return by taking an action u from a state x. Then,the optimal action rule is obtained from the greedy policy
π(x) = arg maxu Q(x, u). We use the Peng’s Q(λ)-learningalgorithm [13], which is an on-line RL method. That is to say,the update procedure is applied after each action.
IV. L EARNING S TRATEGY F USION
LS fusion consists of two elements: the LSs and the LS
fusion algorithm. Each LS generates new behavior modules
according to the strategy. The LS fusion algorithm consists ofthree parts; (1) generating the new behavior modules by using
the LSs, (2) selecting a behavior module to be executed, and(3) executing and training the selected behavior module. The
overview of each part is as follows:(1) Behavior modules of different DoF configurations are
generated by LS-scratch. If there is a behavior module
trained enough, the transfer LSs are applied to the behavior
module. Note that the generation of the same setup (e.g.

LS-scratch of the same DoF) is forbidden in order toprevent that too much behavior modules are generated.
(2) A behavior module is selected from both the existing
behavior modules and the new ones generated by the LSs.
A behavior module desired to be selected is one that is agood performance or has a potential of the performance
improvement. Thus, the Boltzmann selection method with
the UCB value is introduced.
(3) The selected behavior module is used to decide the actual
movement of the robot and update its policy from thesamples obtained through the execution.
The UCB-Boltzmann selection method uses both the mean ofrecent return data RB and its deviation σ B , where B denotes abehavior module. σ B can estimate the potential improvementof the performance. The deviation σ B is also used to judge if
a behavior module is trained enough.
A. Algorithm
We assume that several pairs of a control command space
and a state space {(Ũ, X )} are predefined; they have different
DoF configurations. Here, defining Ũ and X means giving
conversions between (Ũ, X ) and the whole body’s commandand state spaces (Ũw , Xw ) (i.e. the joints are not constrained).
Specifically, we use linear conversions with constant matricesCŨ and CX such that ũw = CŨ ũ, x = CX xw where ũ ∈ Ũ,ũw ∈ Ũw , x ∈ X , and xw ∈ Xw .
Each learning strategy LS is defined as a function
GEN(LS , Ũ, X ) that returns a set of new behavior modules.
The LS fusion algorithm is designed for an episodic task,
given as Algorithm 11 . Here, NLSD denotes an interval of
executing the LSs (LSD means LS Decision). NLSD > 1 isneeded to compute the valid reward statistics (we choose 10 in
the experiments). The LS fusion algorithm selects a behavior
module from both the existing B and new behavior modules
generated by the LSs. Note that only the selected new behavior
module is added into B.
Thus, the key element of LS fusion is each GEN andthe UCB-Boltzmann selection method. Note that LS fusion
works with any LS for that GEN is defined. The rest of thissection describes the reward statistics and the UCB-Boltzmann
selection method. In the next section, we introduce the LSs
used in this paper.
B. Reward Statistics
We P
evaluate the performance of a behavior module byr
R , Tt t , where {rt |t = 1, 2, . . . } denotes the observed
reward sequence in an episode, and T denotes the amountof time in the episode. The definition of R depends on thetask. In general, a sum of reward (return) is expected to bean evaluation of the performance, but in our crawling task,
dividing the return by the amount of time is suitable to selecta better behavior module, especially in the early stage oflearning.
1 In implementing this algorithm, the size of B is limited to 20 to preventthe large memory usage. If the size exceeds the limit, a behavior module thathas the minimum UCB value except for Bnext is removed from B.

Algorithm 1: Learning strategy fusion
1: B ← {} /∗ a set of behavior modules ∗/2: for Neps = 1, 2, . . . do /∗ Neps : episode number ∗/3:
if Neps mod NLSD = 0 then4:/∗ select a behavior module: ∗/5:
Bnew ← {}6:
for each (Ũ, X ) do
7:
for each LS do
8:
Bnew ← Bnew ∪ GEN(LS , Ũ, X )9:select Bnext from B ∪ Bnew by UCB-Boltzmann selection
10:
if Bnext ∈ Bnew then B0 ← B ∪ {Bnext } else B0 ← B
11:perform the episode with Bnext where Bnext is updated by
12:Q(λ)-learning
update the reward statistics RBnext , R2 Bnext , σ maxBnext

Since each behavior module is updated to obtain better
policy, its performance changes with episodes. Thus, we
compute the mean and the standard deviation of recent R databy forgetting the old data, and use them to select a behavior
module. Let RNeps the observation of R at an Neps -th episode.
The reward statistics RB , R2 B are updated byRB ← αR RNeps + (1 − αR )RB ,(1)

2
R2 B ← αR RN+ (1 − αR )R2 B ,
eps

(2)

where αR is a learning rate. The standard deviation of R can2
be obtained by σ B = (R2 B − RB )1/2 . In addition, at the endof each episode, σ maxB is updated for some LSs that storesthe maximum σ B for each B.
The reward statistics RB , R2 B are initialized by zero if
B is generated by LS-scratch. If B is generated by LSaccelerating or LS-freeing, the statistics are initialized by thesource behavior module’s values.
C. UCB-Boltzmann Selection
The UCB value of a behavior module B is defined by
RUCBB , RB + fUCB σ B ,(3)where fUCB is a real constant value that decides the weight of
expected improvement (typically 1 or 2).
According to Boltzmann selection, the probability to select
B is defined as
1π(B) ∝ exp(
RUCBB ),
(4)
τlsd
where τlsd is a temperature parameter to adjust the randomness. We decrease τlsd with τlsd = τlsd0 exp(−δτlsd Neps ).
V. L EARNING S TRATEGIES USING WF-DCOB
This section defines each LS, which generates behavior
modules. The defined LSs are LS-scratch, LS-accelerating, andLS-freeing. As an RL method of LS-scratch, we utilize WFDCOB [1], which learns a policy to select a continuous action
and is efficient in a learning-from-scratch case. In addition, theparameters of the function approximator in WF-DCOB encode
the speed of motion and the target joint angles, which enables
us to define LS-accelerating and LS-freeing easily.

A. WF-DCOB
WF-DCOB is an extension of the discrete action set DCOB
[2]. DCOB is designed to improve RL methods that are applied
for motion learning tasks of multi-legged robots especially in alearning-from-scratch case. DCOB is constructed from a set ofbasis functions (BFs) given to approximate a value function.
Each action in DCOB is designed to be a trajectory that is
Directed to the Center Of a target BF (which is the origin of
the name).
WF-DCOB extends DCOB to search continuous actions
around each discrete action of DCOB. In WF-DCOB, wirefitting [14] is used for function approximation (thus, it namedWF-). WF-DCOB aims to acquire higher performance than
DCOB with keeping the learning stability and the learning
speed in learning-from-scratch cases. The key idea of WFDCOB is restricting the exploration around the actions in
DCOB to make the learning process stable and keep thelearning speed. WF-DCOB has the parameters that encodea speed of motion and target joint angles. We can defineLS-accelerating and LS-freeing so that they change theseparameters. Therefore, WF-DCOB is suitable not only for LSscratch, but for the transfer LSs.
In brief, WF-DCOB learns an action value function Q(x, u),where x ∈ X is a continuous state and u ∈ U is a continuous
action. Note that x is a state of the robot, but u is not a command of the robot. Instead, WF-DCOB has an action converter
that changes an action u = (g, q trg ) into a command sequence
for the robot, where g (> 0) is called an interval factor which
decides a speed of motion2 , and q trg is a vector of target
joint angles. Q(x, u) is approximated by wire-fitting whoseparameter vector is θ> = (θ1> , U1> , θ2> , U2> , . . . ). Here, θi is
a scalar parameter that encodes an action value, Ui is a vector
defined as Ui = (gi , qitrg ) that encodes an action. In addition,a set of constant parameters IR = {(gis , gie )|i = 1, 2, . . . } isdefined to constrain gi . See Appendix for the details.
B. Learning Strategies
Next, we define the LSs, namely, defines a function
GEN(LS , Ũ, X ) for each learning strategy LS . Every behavior
module B has information, LS (B) : a learning strategy withwhich the behavior module is generated, Ũ (B) : a command
space and X (B) : a state space where the behavior module
learns, and K(B) : a set of BFs.
1) LS-Scratch: The function GEN(LS-scr, Ũ, X ) generates
a behavior module in the learning-from-scratch manner. But,
GEN generates no behavior module if there is a behavior
module of the same setup; that is, B is forbidden to include
a behavior module B such that LS (B) = LS-scr, Ũ (B) = Ũ,and X (B) = X . The reason of preventing a generation of thesame setup is that the probability that the new behavior module
obtains better performance than the old one is not high. A new
behavior module Bnew uses WF-DCOB with Q(λ)-learning,where a default (predefined) set of BFs is employed. If a
default set of BFs is not predefined for X , a new behavior
2 Actually, g indicates the slowness; a smaller g generates a faster motion.

module is not generated3 . The parameters of wire-fitting are
initialized as described in Appendix.
2) LS-Accelerating: LS-accelerating generates behavior
modules from source behavior modules by accelerating the
speed parameters of the policy, where a new behavior moduleand its source behavior module have the same command andthe state space. Also, the generation with the same setup
is prevented. The acceleration is performed by multiplying
the interval factor of WF-DCOB by a real constant value
faccel < 1.
Specifically, in the function GEN(LS-accel, Ũ, X ), for every
Bsrc such that Ũ (Bsrc ) = Ũ and X (Bsrc ) = X , a new behavior
module Bnew is generated if the conditions are satisfied:(1) B does not include a behavior module B such that
LS (B) = LS-accel and Src(B) = Bsrc ,(2) σ Bsrc /σ maxBsrc < σth ,where Src(B) denotes the source behavior module of B, σ Bdenotes a deviation of B’s reward, and σ maxB denotes its
maximum. Thus, the condition (2) checks if Bsrc almost
converged, namely, is trained enough. σth is a threshold.
Bnew uses the same BFs as Bsrc , namely K(Bnew ) = K(Bsrc ) .
The parameters of wire-fitting of Bnew are copied from Bsrc
except that {Ui(Bnew ) } is multiplied by faccel . Specifically, foreach i ∈ W (Bsrc ) ,θi(Bnew ) = θi(Bsrc ) ,(5)

Ui(Bnew ) = (gi(Bnew ) , qitrg(Bnew ) ) = (faccel gi(Bsrc ) , qitrg(Bsrc ) ).

(6)

Additionally, the set of constraint range IR = {(gis , gie )|i =1, 2, . . . } is also modified: gis(Bnew ) = faccel gis(Bsrc ) , gie(Bnew ) =faccel gie(Bsrc ) . Thus, LS-accelerating transfers not only thepolicy parameters, but also the limitation of the policy.
3) LS-Freeing: LS-freeing generates behavior modules
from source behavior modules by freeing the DoF of eachsource behavior module to a larger DoF based on a predefined
freeing direction F . Each freeing direction F includes the
(F )
(F )
(F )
(F )
information, Ũsrc, Xsrc, Ũdest, and Xdest
which denote thespaces of a source behavior module and the spaces of adestination behavior module.
In the function GEN(LS-free, Ũ, X ), for every pair of
(F )
(F )
(F )
(F, Bsrc ) such that Ũdest= Ũ, Xdest= X , Ũ (Bsrc ) = Ũsrc, and(Bsrc )(F )X
= Xsrc , a new behavior module Bnew is generated if
the conditions are satisfied:(1) B does not include a behavior module B such that
LS (B) = LS-free, Ũ (B) = Ũ, X (B) = X , and Src(B) =Bsrc ,(2) The same as the condition (2) of LS-accelerating.
The condition (1) is to prevent to generate with the same setup.
Bnew is initialized so that its action value function is almost
the same as that of Bsrc . To do this, first, the freeing matrices
DŨ and DX are calculated so that the conversions ũdest =(F )DŨ ũsrc , xdest = DX xsrc are performed where ũdest ∈ Ũdest
,
(F )
(F )
(F )
ũsrc ∈ Ũsrc , xdest ∈ Xdest , and xsrc ∈ Xsrc . We define thesematrices as(F ) ,
DŨ = C]Ũ (F ) CŨsrc
dest

3 In the following experiments, X

DX = CX (F ) C]X (F ) ,16 is the case.

dest

src

(7)

where ] denotes a pseudo-inverse.
Bnew ’s BFs are calculated as
new )
src )
µ(B
= DX µ(B,kk

The parameters of the>Σk(Bnew ) = DX Σk(Bsrc ) DX,

(8)

for each k ∈ K(Bsrc ) . The parameters of wire-fitting are
initialized as
θi(Bnew ) = θi(Bsrc ) ,(9)Ui(Bnew ) = (gi(Bnew ) , qitrg(Bnew ) ) = (gi(Bsrc ) , DŨ qitrg(Bsrc ) ),(10)

for each i ∈ W (Bsrc ) . In this case, IR is not changed.

Fig. 2.

Simulation model of a humanoid robot.

VI. E XPERIMENTS
In this section, we apply LS fusion to a crawling task of ahumanoid robot on simulation. Fig. 2 shows the simulation
model. Its height is 0.328m. It weighs 1.20kg. Each joint
torque is limited to 1.03Nm, and a PD-controller is embedded
on it. The following experiments are performed on a dynamics simulator, ODE4 , with a time step 0.2ms. LS fusion is
implemented with the RL library, SkyAI5 .
A. Space Configurations
We use six sets of DoF configurations (Fig. 3): 3-DoF(Ũ3 , X3 ), 4-DoF (Ũ4 , X4 ), 5-DoF (Ũ5 , X5 ), 6-DoF (Ũ6 , X6 ),7-DoF (Ũ7 , X7 ), and 16-DoF (Ũ16 , X16 ). In 3, 5, 6, and 7DoF configurations, some joints are coupled to be bilaterally
symmetric. In the 3-DoF, each set of joint pairs {q1 , q3 , q4 , q6 },{q8 , q13 }, {q9 , q10 , q14 , q15 } is coupled respectively. In the 4DoF, each set of joint pairs {q1 , q3 }, {q4 , q6 }, {q8 , q9 , q10 },{q13 , q14 , q15 } is coupled respectively, which means one DoF
for each leg. In the 5-DoF, each set of joint pairs {q1 , q4 },{q3 , q6 }, {q8 , q13 }, {q9 , q14 }, {q10 , q15 } is coupled respectively. In the 6-DoF, a coupled joint pair {q7 , q12 } is added to
the 5-DoF. In the 7-DoF, a coupled joint pair {q2 , q5 } is added
to the 6-DoF. In the 16-DoF, only the head link is fixed, while
the other joints move independently.
In ND -DoF configuration, its command input space is a ND dimensional vector space that represents target joint angles. Its
state space is
x = (c0z , qw , qx , qy , qz , q>ND ,>ċ0x , ċ0y , ċ0z , ωx , ωy , ωz , q̇>
ND )

(11)

where (c0x , c0y , c0z ) denotes the position of the center-of-massof the body link, (qw , qx , qy , qz ) denotes the rotation of the
body link in quaternion, (ωx , ωy , ωz ) denotes the rotational
velocity of the body link, and qND denotes the joint angle
vector of the ND -DoF. The reason for the absence of c0x andc0y from the state x is that a policy for the crawling task does
not have to depend on the global location of the robot.
The default BFs are allocated as follows. For the state space
X3 , we allocate BFs on a 5 × 5 × 5 grid over the q3 space.
For the state space X4 , we allocate BFs on a 4 × 4 × 4 × 4 grid
over the q4 space. For the state space X5 , we allocate 202
BFs by a dynamics-based allocation method used in [2]. For
4 Open Dynamics Engine: www.ode.org
5 SkyAI: skyai.org

Fig. 3. DoF configurations and possible freeing directions. Each encircled
number shows an index of dimension; joints with the same number arecoupled. Each arrow shows that freeing is possible in this direction.

the state space X6 and X7 , we allocate 300 and 600 BFs overthe (qy , q6 ) and the (qy , q7 ) space respectively by a springdamper allocation method6 . For the state space X16 , we do
not prepare a default set of BFs since the DoF is too large.
The possible freeing directions between these DoF configurations are defined as shown in Fig. 3. Each arrow shows thatfreeing is possible in this direction.
B. Task Setup
The objective of the crawling task is to move forward as fast
as possible. According to the objective, the reward is designed
as follows:r(t) = rmv (t) − rrt (t) − rsc (t) − rfd (t)(12)

where rmv (t) = 50(ċ0x (t)ez1 (t) + ċ0y (t)ez2 (t)), rrt (t) =5|ωz (t)|, rsc (t) = 2 × 10−5 kũ(t)k; rmv (t) is a reward forforward movement, (ez1 , ez2 , ez3 )> is a z-component of therotation matrix of the body link, rrt is a penalty for rotation,rsc (t) is a step cost, rfd (t) is a penalty for falling down.
6 First, we allocate BFs randomly. The covariance matrix of each BF isconstrained to Σ = σ 2 1 where 1 is an unit matrix. Then, they are re-arranged
so that the centers of the BFs spread as widely as possible and σ becomes aslarge as possible without overlapping.

rfd (t) takes 4 if the body or the head link touches the ground,
otherwise it takes 0. The penalty for falling down is given oncein each action. Each episode begins with the initial state∫ t where
the robot lies down and stationary, and ends if 0 r(t0 )dt0
6 −40 or t > 20[s].
C. Learning Method Configurations
We choose the parameters of LS fusion (denoted as LSF
in the experiments) as follows: fUCB = 2, αR = 0.05,
NLSD = 10, τlsd0 = 20, δτlsd = 0.004, σth = 0.2,
and faccel = 0.95. LS-scratch generates a behavior module whose parameters are set as τ0 = 2, δτ = 0.02 for
Boltzmann selection of the decreasing temperature parameterτ = τ0 exp(−δτ NepsB ) where NepsB denotes a number of
episodes performed by the behavior module B. The parametersof WF-DCOB are as follows: Cp (x) = qND , Cd (x) = q̇ND
for x ∈ XND , and IR = {(0.05, 0.1), (0.1, 0.2), (0.2, 0.3)} forevery configurations. LS-accelerating and LS-freeing generate
a behavior module of the parameters τ0 = 0.1, δτ = 0.02
since the behavior module starts from an almost converged
policy. Every behavior module uses Peng’s Q(λ)-learning with
γ = 0.9, λ = 0.9, and a decreasing step size parameterα = 0.3 exp(−0.002NepsB ).
As a comparison, some configurations of WF-DCOB are
also applied. Note that WF-DCOB for the crawling task issuperior to conventional methods though its performance isalmost the same as that of DCOB [1], [2]. We employ fiveconditions that are denoted as WF-DCOB-{3,4,5,6,7}. Each
number indicates a DoF. All of them use Peng’s Q(λ)-learningwith the same parameters as a behavior module of LS-scratch.
D. Results
We execute 10 runs for each configuration. Fig. 4 shows
the learning curves of the first five runs of LSF (ex0,...,ex4).
In this figure, each circle shows the return acquired by abehavior module generated by LS-scratch (we refer to it asa scratch behavior module). Namely, the other points on thesolid curve are obtained by behavior modules generated byLS-accelerating and LS-freeing. In four out of five runs, thelearning curves converge to higher values than that of thescratch behavior modules. This result means that the transfer learning by LS-accelerating and LS-freeing successfully
improves the performance of the motion. Meanwhile, in ex2,
the performance is not improved by the transfer learning. A
possible reason is that the scratch behavior module acquires ahigh performance motion in the early stage of the learning, andthere is no room for improvement. Anyway, the remarkable
point is that the LS fusion algorithm selects a suitable sequenceof the LSs including the selection of a DoF configuration.
Let us see a detailed learning process. Fig. 5 shows alearning curve and a behavior module transition in a run
obtained from LSF (ex0 in Fig. 4). The returns obtained byscratch behavior modules are also plotted by circles. In theearly stage of the learning (0 . . . 500-th episode), the scratch
behavior modules dominate. One of them seems converging
to a return about 230 around 350-th episode. The converged

Fig. 4. Learning curves of five runs obtained from LSF. Each solid line
shows the return per episode, and each circle shows the return acquired by abehavior generated by LS-scratch.

Fig. 5. Learning curve and module transition in a run obtained from LSF(ex0 in Fig. 4). The dotted line shows the return per episode, each circle shows
the return acquired by a behavior generated by LS-scratch, and the solid line
shows the index of the selected behavior module in each episode.
TABLE IP ROFILES OF MOTIONS IN F IG . 6.
EpisodeReturn

200
300
400
500
600
700
800
1000
1500

6.12
231.55210.00229.46248.01
328.49361.10361.87374.33

Procedure
S(3)S(3)S(3)→F(3→16)
S(3)
S(3)→F(3→5)→A→F(5→6)→F(5→16)
S(3)→F(3→5)→A→F(5→16)→A→A
S(3)→F(3→5)→A→F(5→16)→A→A→A
S(3)→F(3→5)→A→F(5→16)→A→A→A
S(3)→F(3→5)→A→F(5→16)→A→A→A

module uses the 3-DoF configuration, with which a behavior
module can learn policy quickly because of the lower dimension. Then, LS-accelerating and LS-freeing are applied. The
behavior module used in the final stage of the learning is obtained through the following LS sequence: S(3-DoF)→F(3→5DoF)→A→F(5→16-DoF)→A→A→A, where S denotes LSscratch, A denotes LS-accelerating, and F denotes LS-freeing.

Therefore, these results demonstrate that using LS fusion
enables (1) in the early stage of learning, an agent canselect a suitable DoF configuration, and (2) after a scratch
behavior module converges, LS-accelerating and LS-freeing
can improve the policy.
VII. C ONCLUSION AND F UTURE W ORK

Fig. 6. Snapshots of motions during learning (ex0 in Fig. 4). Every snapshot
is taken at 3-FPS during first 15 frames in each episode.

We proposed the learning strategy (LS) fusion method
where some LSs are integrated for learning a single task bya single robot. As the LSs, we developed LS-scratch, LSaccelerating, and LS-freeing; they are implemented with WFDCOB [1]. In the LS fusion algorithm, the upper-confidencebound (UCB) value of the return and Boltzmann selection
method are employed to select a behavior module.
The simulation experiments of a crawling task of a small
size humanoid robot demonstrated the advantage of LS fusioncompared to learning with single learning modules. Namely,using LS fusion enables (1) in the early stage of learning,
LS fusion can select a suitable DoF configuration, and (2)after a behavior module learning from scratch converges, LSaccelerating and LS-freeing can improve the policy.
Our LS fusion architecture does not share the samples
among the behavior modules. Introducing a kind of importancesampling to share the samples like CLIS [9] may improve thelearning speed, which is one of our future tasks.
ACKNOWLEDGMENTS
Part of this work was supported by a Grant-in-Aid for
JSPS, Japan Society for the Promotion of Science, Fellows(22·9030).
A PPENDIX
WF-DCOB
We first introduce the function approximator wire-fitting[14], then describe WF-DCOB.

Fig. 7. Resulting learning curves of the crawling task. Each curve shows
the mean of the return per episode over 10 runs.

The final convergent value of the return is around 370, thus,
the performance is improved by the transfer learning. Fig. 6
shows snapshots during learning (ex0 in Fig. 4), and TABLE I
shows the profiles of motions in Fig. 6. These results show
how the performance of motion is improved. Please see also
the accompanying video.
Fig. 7 shows the resulting learning curves of the crawlingtask (the mean of the return per episode over 10 runs).
Among WF-DCOBs, WF-DCOB-3 converges fastest and tothe highest value of return. A possible reason is that in addition
to the lowest dimension, the joint coupling of the 3-DoF is
suitable for the crawling task. On the other hand, LSF reaches
at a higher value than that of WF-DCOB-3. The reason isconsidered to be an effect of LS fusion; though the return
value of WF-DCOB-3 is not improved after convergence,
LSF improves its policy by applying LS-accelerating and LSfreeing.

A. Wire-Fitting
Wire-fitting is a function approximator over a continuous
state space X and a continuous action space U which iscompatible with value-function-based RL algorithms [14]. It
is defined as∑(di + )−1 qi (x)∑Q(x, u) = lim+ i∈W
,
(13)
−1
→0
i∈W (di + )[]
(qi0 (x)) − qi (x) . (14)di = ku − ui (x)k2 + C max
0i ∈W

Here, a pair of the functions qi (x) : X → R and ui (x) :X → U (i ∈ W) is called a control wire; wire-fitting isregarded as an interpolator of the set of control wires W.
C is a smoothing factor of the interpolation (we choose
C = 0.001 in the experiments). Obviously, qi (x) is related
to an action value, and ui (x) is related to an action. Anyfunction approximator is available for these functions. Regardless of the kind of the function approximators, wirefitting has the features: maxu Q(x, u) = maxi∈W (qi (x)), andarg maxu Q(x, u) = ui? (x) where i? = arg maxi∈W (qi (x)).
Namely, the greedy action at a state x is calculated only byevaluating qi (x) for i ∈ W. We use a normalized Gaussian
network (NGnet) [15] for qi (x) and a real value vector forui (x), that is, let qi (x) = θi> φ(x) and ui (x) = Ui , where
φ(x) is the output of the NGnet. The k-th element of φ(x) is
defined as
G(x; µk , Σk )φk (x) = ∑
,
(15)
0
0
k0 ∈K G(x; µk , Σk )where G(x; µ, Σ ) denotes a Gaussian with mean
µ and covariance matrix Σ , and K denotes a setof BFs. Thus, the parameter vector θ is defined by>>
θ> = (θ1> , U1> , θ2> , U2> , . . . , θ|W|
, U|W|), and the gradient∇θ Q(x, u) can be calculated analytically.
B. WF-DCOB
WF-DCOB [1] explores continuous actions around eachdiscrete action of DCOB [2] where wire-fitting is used toapproximate a value function over continuous action. WFDCOB consists on two elements; an action converter and an
extended wire-fitting. Action converter converts an output ofan RL agent into a sequence of a control command of a robot.
In the following, we describe only key points. Please referto the original paper for the details; [1] for WF-DCOB, and[2] for DCOB.
1) Assumptions: WF-DCOB assumes the follows:(A) Each BF k ∈ K has a fixed center µk ∈ X .
(B) Q, Cp (x), and Cd (x) are defined. Q: a space in which areference trajectory is calculated (e.g. a joint angle space).
Cp (x): a function that extracts q ∈ Q from a state x ∈ X
as q = Cp (x) : X → Q. Cd (x): a function that extracts
the derivative of q ∈ Q from a state x∈X as q̇ = Cd (x).
(C) A low-level controller ũ(t) = Ctrl (x(t), q d (t + δt)) tofollow a trajectory q d (t) is given where ũ is in Ũ and δtdenotes a control time-step.
In this paper, we let Q = Ũ for simplicity. In this case, ũ(t) =Ctrl (x(t), q d (t + δt)) = q d (t + δt).
2) Action Converter: Input of the action converter is acontinuous action defined by u = (g, q trg ) ∈ R × Q , Ucnv
where g ∈ R is called an interval factor that decides a speedof motion, and q trg ∈ Q is target joint angles. The action
converter outputs a control command of a robot so that thejoint angles of the robot move toward q trg . Actually, the output
command for a single action terminates in a small time intervalwith which the state moves into the nearest BF.
3) Extended Wire-Fitting: DCOB discretizes Q space by
the centers of the BFs, and the interval factor space by somereal numbers. Searching continuous actions around each actionof DCOB is accomplished through three steps: (1) prepare
wire-fitting that has the same number of the control wires
as the size of DCOB, (2) initialize the Ui by corresponding
action in DCOB, (3) apply a Q(λ)-learning to wire-fitting.
Specifically, theparameters are initialized as θi = 0, Ui =gs +ge(gi , qitrg ) = ( i 2 i , Cp (µki )) for each i ∈ W.
Additionally, to relax the instability of Q(λ)-learning withwire-fitting, each Ui is constrained to around the corresponding action in DCOB. Letting Ui , (gi , qitrg ), the parameter

constraint is defined as
if gi < gis then gi ← gis
if gi > gie then gi ← gie
if kqitrg − Cp (µki )k > dQn (ki ) then(16)

(qitrg − Cp (µki ))qitrg ← Cp (µki ) + dQ(k)
inkqitrg − Cp (µki )kwhere gis , gie denote the constraint range of gi , ki denotes anindex of the corresponding BF, dQ
n (ki ) denotes a distance inQ space between the BF ki and its nearest BF. The set of therange {(gis , gie )|i = 1, 2, . . . } is predefined as IR . Thus, thenumber of the control wires is |W| = |IR ||K|. This constraint
is applied after each Q(λ)-learning update.
R EFERENCES[1] A. Yamaguchi, J. Takamatsu, and T. Ogasawara, “Constructing
continuous action space from basis functions for fast and stablereinforcement learning,” in the 18th IEEE International Symposium on
Robot and Human Interactive Communication (RO-MAN’09), Toyama,
Japan, 2009, pp. 401–407.
[2] ——, “Constructing action set from basis functions for reinforcement
learning of robot control,” in the IEEE International Conferenceon Robotics and Automation (ICRA’09), Kobe, Japan, 2009, pp.
2525–2532.
[3] J. Morimoto, S. Hyon, C. Atkeson, and G. Cheng, “Low-dimensionalfeature extraction for humanoid locomotion using kernel dimension
reduction,” in the IEEE International Conference on Robotics andAutomation (ICRA’08), 2008, pp. 2711–2716.
[4] A. M. Farahmand, A. Shademan, M. Jägersand, and C. Szepesvári,“Model-based and model-free reinforcement learning for visual servoing,” in the IEEE International Conference on Robotics and Automation(ICRA’09), Kobe, Japan, May 2009, pp. 2917–2924.
[5] Y. Takahashi and M. Asada, “Multi-layered learning systems for visionbased behavior acquisition of a real mobile robot,” in Proceedings of
SICE Annual Conference 2003, 2003, pp. 2937–2942.
[6] J. Kober and J. Peters, “Learning motor primitives for robotics,” in theIEEE International Conference on Robotics and Automation (ICRA’09),
2009, pp. 2509–2515.
[7] J. Zhang and B. Rössler, “Self-valuing learning and generalization withapplication in visually guided grasping of complex objects,” Robotics
and Autonomous Systems, vol. 47, no. 2-3, pp. 117–127, 2004.
[8] G. Taga, R. Takaya, and Y. Konishi, “Analysis of general movements
of infants towards understanding of developmental principle for motor
control,” in the IEEE International Conference on Systems, Man, andCybernetics, 1999 (SMC ’99), vol. 5, 1999, pp. 678–683.
[9] E. Uchibe and K. Doya, “Competitive-cooperative-concurrent reinforcement learning with importance sampling,” in the International Conference on Simulation of Adaptive Behavior: From Animals and Animats,
2004, pp. 287–296.
[10] F. Fernández and M. Veloso, “Probabilistic policy reuse in a reinforcement learning agent,” in AAMAS ’06: Proceedings of the fifthinternational joint conference on Autonomous agents and multiagent
systems. New York, NY, USA: ACM Press, 2006, pp. 720–727.
[11] F. Fernández, J. Garcı́a, and M. Veloso, “Probabilistic Policy Reuse
for inter-task transfer learning,” Robotics and Autonomous Systems,
vol. 58, no. 7, pp. 866–871, 2010.
[12] M. G. Madden and T. Howley, “Transfer of experience between
reinforcement learning environments with progressive difficulty,”
Artificial Intelligence Review, vol. 21, pp. 375–398, June 2004.
[13] J. Peng and R. J. Williams, “Incremental multi-step Q-learning,” inInternational Conference on Machine Learning, 1994, pp. 226–232.
[14] L. C. Baird and A. H. Klopf, “Reinforcement learning with highdimensional, continuous actions,” Wright Laboratory, Wright-Patterson
Air Force Base, Tech. Rep. WL-TR-93-1147, 1993.
[15] M. Sato and S. Ishii, “On-line EM algorithm for the normalized
Gaussian network,” Neural Computation, vol. 12, no. 2, pp. 407–432,
2000.