社団法人 電子情報通信学会
THE INSTITUTE OF ELECTRONICS,
INFORMATION AND COMMUNICATION ENGINEERS

信学技報
TECHNICAL REPORT OF IEICE.

Fusing Learning Strategies to Learn Various Tasks
with Single Configuration
Akihiko YAMAGUCHI† , Jun TAKAMATSU† , and Tsukasa OGASAWARA†† Graduate School of Information Science, Nara Institute of Science and Technology
8916-5, Takayama, Ikoma, Nara 630-0192, JAPAN
Abstract This paper proposes a method to fuse learning strategies (LSs) in reinforcement learning framework.
Generally, we need to choose a suitable LS for each task respectively. In contrast, the proposed method automatesthis selection by fusing LSs. The LSs fused in this paper includes a transfer learning, a hierarchical RL, and a model
based RL. The proposed method has a wide applicability. When the method is applied to a motion learning task,such as a crawling task, the performance of motion may be improved compared to an agent with a single LS. The
method also can be applied to a navigation task by hierarchically combining already learned motions, such as a
crawling and a turning. This paper demonstrates a maze task of a humanoid robot where the robot learns not only
a path to goal, but also a crawling and a turning motions.
Key words Learning System, Reinforcement Learning, Modularization, Motion Learning, Humanoid Robot

1. Introduction
Designing a behavior by only its objective is essential for
future robots, since this ability enables the end-users to teachtheir wish to the robots easily. Reinforcement Learning (RL)
method is such a technology. There are a lot of RL applications to robotics research [1]〜[4]. However, RL methods
require a lot of learning cost in large domains, such as motion
learning of a humanoid robot.
Many researchers have tackled this issue and proposed effective methods; for instance, dimension reduction [2], hierarchical RL [3], transfer learning [5], model utilization [6], imitation learning [4]. We refer to such a method as a learning
strategy (LS).
These LSs improve RL, however, their eﬀectiveness or applicability depends on a task. For instance, a dimension
reduction by using a pattern generator may improve cyclic
motions; meanwhile, this way may restrict the capability tolearn episodic motions such as jumping. The dependency of
LSs on tasks means that the end-users should select a proper
LS for each task. This selection may be diﬃcult for ordinary
users.
Thus, we aim to make a system in which multiple LSs
are applied to each task of a single robot multiple times,
where the ordering of the LSs is automatically decided. For
this purpose, the highly modularized learning system mainly
consists of the following four elements:
Behavior modules: deciding the behavior of a robot.
Fundamental module types: constructing behavior modules, such as a module of an RL method.
LS modules: generating behavior modules from the fundamental module types according to its learning strat-

Fig. 1 Overview of the highly modularized learning system. Every small circles, ellipses, and rectangles indicate modules.
UCB-B denotes UCB-Boltzmann selection method.

egy.
UCB-Boltzmann selection method: choosing a behavior
module actually used in each learning stage.
Fig. 1 illustrates the overview of the system. The system
maintains a set of module types. The LS modules generate
behavior modules from the types (Module Instances). Eachbehavior module is used to decide a behavior of the robot
and learn a task. Some of the LS modules generate behavior modules from existing behavior modules (e.g. a transfer
LS). The system has multiple behavior modules for each task.
Each behavior module has upper conﬁdence bound (UCB)
as an evaluation value which is calculated from reward observation. In each learning stage, a behavior module that actually decides the movement of the robot is chosen from thewhole set of behavior modules by UCB-Boltzmann selection
method. UCB-Boltzmann selection method is Boltzmann—1—selection method with UCB, which enables a probabilistic
exploration. The selected behavior module is trained during
the learning stage. Thus, each behavior module is generated
through a sequence of LSs, and the selection of behavior
modules means the selection of a proper LS sequence for thetask. The core algorithm is referred to as LS fusion method.
This paper also deﬁnes the following LSs:
LS-scratch generates a behavior module that learns a task
from scratch using WF-DCOB [1].
LS-accelerating generates a behavior module by accelerating the motion of a source behavior module.
LS-freeing generates a behavior module by increasing the
DoF of a source behavior module.
LS-planning generates a behavior module that uses amodel-based RL method such as Dyna [6].
LS-hierarchy generates a hierarchical action space module.
Here, the LS-accelerating and the LS-freeing are transfer
learning methods [5]. A remarkable feature of these transfer LSs is that the LSs transfers not only policy parameters,
but also physical limitation of the policy.
The proposed LS fusion is applied to a maze task of thesimulated humanoid robot where the robot learns not only
a path to goal, but also a crawling and a turning motions.
The rest of this paper is organized as follows. Section 2.
describes LS fusion method, Section 3. deﬁnes the LSs, Section 4. demonstrates the experiments, and Section 5. concludes this paper.

2. Learning Strategy Fusion
LS fusion consists of two elements: (1) the LSs, and(2) UCB-Boltzmann selection. There are two types in LSs;
behavior LSs that generate new behavior modules, and supplementary LSs that generate supplementary modules otherthan behaviors, such as model modules and hierarchicalaction-space modules. Let LS bhv denote the set of the behavior LSs, LS spl denote the set of the supplementary LSs.
UCB-Boltzmann selection method chooses a behavior fromboth the existing behavior modules and the new ones generated by the behavior LSs. The selected behavior module isused to actually control the robot, and the module is updated
its policy from samples.
We design LS fusion so that the behavior and the supplementary modules are generated and learned through thefollowing ﬂow:
(1) The supplementary LSs generate modules if applicable.
(2) The behavior LSs generate new behavior modules.
Speciﬁcally, the LS-scratch generates new behaviors
which may have diﬀerent DoF conﬁgurations. If there
are behaviors trained enough, the LS-freeing and theLS-accelerating generate new behavior modules bytransferring the trained behaviors.
(3) UCB-Boltzmann selection method chooses a behavior
module from both the existing behavior modules andthe new ones generated in (2).
(4) Several episodes are performed using the selected behavior module, and the policy of the behavior module
is updated from samples. The supplementary modules
are also updated if possible.
(5) (1). . . (4) are repeated.
Here, UCB (upper conﬁdence bound) uses both the meanof a reward summation RB and its deviation σ B . The deviation σ B can estimate the potential improvement of theperformance. The deviation σ B is also used to judge if abehavior module is trained enough.
2. 1 LS Fusion Algorithm
We assume that several pairs of a control command space
and a state space {(Ũ, X )} are predeﬁned; they have different DoF conﬁgurations. Here, deﬁning Ũ and X means
giving conversions between (Ũ, X ) and the overall (full DoF)
command and state spaces (Ũw , Xw ). Speciﬁcally, we assume
linear conversions with constant matrices CŨ and CX suchthat ũw = CŨ ũ, x = CX xw where ũ ∈ Ũ , ũw ∈ Ũw , x ∈ X ,and xw ∈ Xw .
Each behavior learning strategy LS is deﬁned as a function GENbhv (LS , U, X , Task ) that generates behavior modules, and each supplementary learning strategy is deﬁned asa function GENspl (LS , Task ) that generates supplementary
modules.
The LS fusion algorithm is deﬁned for an episodic task.
Algorithm 1 shows the overall algorithm(*1). Here, NLSSp isan interval of executing the supplementary LSs (LSSp means
LS Supplementary), NLSBh is an interval of executing the behavior LSs (LSBh means LS Behavior). NLSBh > 1 is needed
to compute the valid reward statistics (we choose NLSSp = 20
and NLSBh = 10 in the experiments). UCB-Boltzmann selection method chooses a behavior module from both theexisting B and new behavior modules generated by the behavior LSs. Note that only the selected new behavior module
is added into B.
Thus, the key element of LS fusion is each LS (GENbhv ,GENspl ) and UCB-Boltzmann selection method. Note that
LS fusion works with any LS for that GENbhv or GENspl is deﬁned. The rest of this section describes the reward statistics
and UCB-Boltzmann selection method. In the next section,
we specify the LSs used in this thesis.
2. 2 Reward Statistics
WePevaluate the performance of a behavior module byr
R , Tt t , where {rt |t = 1, 2, . . . } denotes the observed reward sequence in an episode, and T denotes total time in theepisode. The deﬁnition of R depends on a task. In general,a sum of reward (return) may be used, but in our crawlingtask, this deﬁnition is suitable to select a better behavior
module, especially in the early stage of learning.

(*1)：In implementing this algorithm, the size of B is limited to 20
per a task to prevent the large memory usage. If the size exceeds thelimit, a behavior module that has the minimum UCB of RUCB except
for Bnext is removed from B.

—2—where τlsd is a temperature parameter to adjust randomness.
We decrease τlsd with τlsd = τlsd0 exp(−δτlsd Neps ).

Algorithm 1: Learning strategy fusion
Input:Task Task , behavior modules B,
state-space modules {X }, action-space modules: {U},dynamics-model modules {Mdyn }, reward-model modules{Mrwd }/∗ {Mdyn } and {Mrwd } may be empty ∗/1: for Neps = 1, 2, . . . do /∗ Neps : episode number ∗/
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:

15:

if Neps mod NLSSp = 0 thenfor each LS ∈ LS spl do{X }, {U }, {Mdyn }, {Mrwd } ← GENspl (LS , Task )if Neps mod NLSBh = 0 then/∗ select a behavior module: ∗/Bnew ← {}for each (U, X ) dofor each LS ∈ LS bhv do
Bnew ← Bnew ∪ GENbhv (LS , U, X , Task )Select Bnext from B∪Bnew by UCB-Boltzmann selection
if Bnext ∈ Bnew then B0 ← B ∪ {Bnext } else B0 ← B
return Bnext , B0
Perform the episode with Bnext :Bnext is updated by its own learning algorithm{Mdyn }, {Mrwd } are updated if possible
Update the reward statistics RBnext , R2 Bnext , σ maxBnext

Since each behavior module is updated to obtain better
policy, its performance changes with episodes. Thus, we
compute the mean and the standard deviation of R with
forgetting the old data, and use them to select a behavior
module. Let RNeps the observation at an Neps -th episode.
The reward statistics RB , R2 B are updated byRB ← αR RNeps + (1 − αR )RB ,

(1)

2
2
B ← αR RNeps + (1 − αR )R B ,

(2)

R2

where αR is a learning rate. The standard deviation of R
2
can be obtained by σ B = (R2 B − RB )1/2 . Additionally, atthe end of each episode, σ maxB is updated which is used insome LSs.
The reward statistics RB , R2 B are initialized by zero if B
is generated by the LS-scratch. If B is generated by the LSaccelerating or the LS-freeing, the statistics are initialized by
the source behavior module’s values.
2. 3 UCB-Boltzmann Selection
We employ UCB of R to evaluate the priority of search.
Additionally, we use Boltzmann selection to probabilisticallyselect a behavior module.
The UCB of R is deﬁned by
RUCBB , RB + fUCB σ B(3)where fUCB is a real constant value that decides the weightof expected improvement (typically 1 or 2).
According to Boltzmann selection, the probability to select
B is deﬁned as
π(B) ∝ exp(

1
RUCBB )
τlsd

(4)

3. Learning Strategies
This section deﬁnes the LSs, namely, deﬁnes a function
GENbhv (LS , U , X , Task ) or GENspl (LS , Task ) for each learning strategy LS . Every behavior module B has information,
Task (B) : a task that B is learning, LS (B) : a learning strategy with which the behavior module is generated, U (B) : anaction space and X (B) : a state space where the behavior
module learns, and K(B) : a set of BFs.
The LS-scratch requires that an RL method is (1) eﬃcientin a learning-from-scratch case, and (2) suitable to deﬁne theLS-accelerating and the LS-freeing. Thus, we utilize WFDCOB [1] for these LSs, since WF-DCOB is eﬃcient in alearning-from-scratch case like a discrete action set but explores continuous actions stably. Additionally, WF-DCOB
explicitly has parameters related to a speed of motion andtarget joint angles, which is suitable for the LS-accelerating
and the LS-freeing.
In WF-DCOB, a continuous action space is generated froma set of BFs given to approximate a value function, thenwire-ﬁtting [7] is utilized to learn over the continuous action space. An element in the continuous action space isdenoted as u = (g, q trg ), where g ∈ R is called an interval factor that decides a speed of motion, and q trg ∈ Q is
the target joint angles of a trajectory. WF-DCOB’s conﬁgurations are Cp (x): a function that extracts q ∈ Q from astate x ∈ X , Cd (x): a function that extracts the derivative of q ∈ Q from a state x ∈ X as q̇ = Cd (x), andIR : a set of ranges of the interval factors. The action
space deﬁned by these conﬁgurations is learned using wireﬁtting. Wire-ﬁtting is an interpolator whose parameters are>>
Θ> = (θ1> , U1> , θ2> , U2> , . . . , θ|W|
, U|W|). Here, θi encodes anaction value, and Ui encodes an action. The LS-accelerating
and the LS-freeing are realized by modifying the action parameters {Ui }. Please see [1] for the detail of WF-DCOB.
3. 1 LS-Scratch
The function GENbhv (LS-scr, U , X , Task ) generates a behavior module if U is a command space Ũ and there is no
behavior module of the same setup. Namely, a module isgenerated if B does not include a behavior module B suchthat LS (B) = LS-scr, U (B) = Ũ, and X (B) = X . The reason of preventing a generation of the same setup is that theprobability that the new behavior module obtains better performance than the existing one is not high. A new behavior
module Bnew uses WF-DCOB with Q(λ)-learning, where adefault (predeﬁned) set of BFs is employed. If a default setof BFs is not predeﬁned for X , a new behavior module is
not generated(*2). WF-DCOB’s conﬁgurations Cp , Cd , Ctrl ,IR are assumed to be predeﬁned for each (Ũ, X ). Since thisbehavior module learns from scratch, the parameters of wireﬁtting are initialized as the WF-DCOB’s manner.
(*2)：In the following experiments, X16 is the case.

—3—3. 2 LS-Accelerating
The LS-accelerating generates behavior modules fromsource behavior modules that have the same command andstate spaces. Also, the generation with the same setup is prevented, and this LS works with only a command space. The
acceleration is performed by multiplying the interval factorof the source module’s WF-DCOB by a real constant value
faccel < 1.
Speciﬁcally, in the function GENbhv (LS-accel, Ũ, X , Task ),for each Bsrc such that U (Bsrc ) = Ũ and X (Bsrc ) = X , a
new behavior module Bnew is generated if the conditions are
satisﬁed:(1) B does not include a behavior module B such that
LS (B) = LS-accel and Src(B) = Bsrc ,(2) σ Bsrc /σ maxBsrc < σth ,where Src(B) denotes the source behavior module of B, σ Bdenotes a deviation of B’s return, and σ maxB denotes its
maximum. Thus, the condition (2) checks if Bsrc almost
converged, namely, is trained enough. σth is a threshold.
Bnew uses the same BFs with Bsrc , namely K(Bnew ) =(Bsrc )K
. The parameters of wire-ﬁtting of Bnew are copied(B)
from Bsrc except for {Ui new } that is multiplied by faccel .
(Bsrc )
Speciﬁcally, for each i ∈ W,(Bnew )

θi

(Bsrc )

= θi

,

(5)

(B
)
(B
)
trg(Bnew )(B)
trg(Bsrc )Ui new = (gi new , qi
) = (faccel gi src , qi).

(6)Additionally, the set of constraint range IR = {(gis , gie )|i =
s(B
)
s(B
)
e(B
)
1, 2, . . . } is also modiﬁed: gi new = faccel gi src , gi new =e(Bsrc )faccel gi. Thus, the LS-accelerating transfers not only
the policy parameters, but also the limitation of the policy.
3. 3 LS-Freeing
The LS-freeing frees the DoF of a source behavior module to larger DoF based on a predeﬁned freeing direction F .
(F )
(F )
Each freeing direction F includes information, Ũsrc , Xsrc ,
(F )
(F )
Ũdest , and Xdest which denote the spaces of a source behavior module and the spaces of a destination behavior module.
This LS works with only a command space.
In the function GENbhv (LS-free, Ũ, X , Task ), for each pair
(F )
(F )
(F )
of (F, Bsrc ) such that Ũdest = Ũ, Xdest = X , U (Bsrc ) = Ũsrc ,(F)
and X (Bsrc ) = Xsrc , a new behavior module Bnew is generated if the conditions are satisﬁed:(1) B does not include a behavior module B such that
LS (B) = LS-free, Ũ (B) = Ũ, X (B) = X , and Src(B) =Bsrc ,(2) The same as the condition (2) of the LS-accelerating.
The condition (1) is to prevent to generate with the same
setup.
Bnew is initialized so that its action value function is almost the same as that of Bsrc . To do this, ﬁrst, the freeing matrices DŨ and DX are calculated so that the conversions ũdest = DŨ ũsrc , xdest = DX xsrc are performed where
(F )
(F )
(F )
(F )
ũdest ∈ Ũdest , ũsrc ∈ Ũsrc , xdest ∈ Xdest , and xsrc ∈ Xsrc .
We deﬁne these matrices as
DŨ = C] (F ) CŨ (F ) ,Ũdestsrc

DX = CX (F ) C] (F ) ,
dest

Xsrc

(7)

where ] denotes a pseudo-inverse. The parameters of the
Bnew ’s BFs are calculated as
(B

)

(B

)

µk new = DX µk src ,>Σk new = DX Σk src DX
,
(B

)

(B

)

(8)

for each k ∈ K(Bsrc ) . The parameters of wire-ﬁtting are initialized as(Bnew )

θi

(Bsrc )

= θi

,

(B
)
(B
)
trg(Bnew )(B)
trg(Bsrc )Ui new = (gi new , qi
) = (gi src , DŨ qi
),

(9)
(10)

for each i ∈ W (Bsrc ) . In this case, IR is not changed.
3. 4 LS-Planning
The function GENbhv (LS-pln, U , X , Task ) generates a behavior module if U is a discrete action space. Also, thegeneration with the same setup is prevented. A new behavior module uses Dyna-MG [6] where we use Q(λ)-learninginstead of Q(0)-learning. Here, a default set of BFs is employed to linearly approximate the value function, and thereward and the dnyamics models.
3. 5 LS-Hierarchy
The function GENspl (LS-hier, Task ) generates a hierarchical action space H. The LS-hierarchy assumes that eachtask has a category label. A hierarchical action space is generated as a set of subtasks that have a same category label.
Thus, the LS-hierarchy does not construct a hierarchical action space in a fully automatic manner. When an action in
H is selected, the LS fusion algorithm is also used to execute the subtask. If a subtask is a continuing task (theHumanoidML-crawling and the HumanoidML-turning tasks
are the case), we need to conﬁgure a duration of the subtask.

4. Experiments – HumanoidMaze
LS fusion is applied to a maze task of the simulated humanoid robot. In this task, the robot learns from scratch,namely, it learns not only a path to goal, but also a crawling
and a turning motions. This task is referred to as HumanoidMaze. In order to train the robot, we specify a task sequence;
0-th to 1499-th episode: the crawling task, 1500-th to 2499th episode: the turning-left task, 2500-th to 3499-th episode:the turning-right task, and 3500-th to 3999-th episode: themaze task.
In this learning, each LS is expected to be used in thefollowing scenario:(1) The primitive tasks (crawling and turning) are learned
with the LS-scratch, the LS-freeing, and the LSaccelerating.
(2) The LS-hierarchy generates a hierarchical action spacein which the primitive tasks are treated as the subtasks.
(3) The LS-planning generates a behavior module for themaze task where the hierarchical action space is used.
4. 1 Task Setup
Fig. 2 shows the simulation model of the robot. Its heightis 0.328m. It weights 1.20kg. Each joint torque is limited
to 1.03Nm, and a PD-controller is embedded on it. The following experiments are performed on a dynamics simulator,—4—

Fig. 2 Simulation model of a humanoid robot.

Fig. 3 DoF configurations and possible freeing directions. Each
encircled number shows an index of dimension; joints withthe same number are coupled. Each arrow shows that freeing is possible in this direction.

ODE(*3), with a time step 0.2ms. LS fusion is implemented
with the RL library, SkyAI(*4).
We use six sets of DoF conﬁgurations (Fig. 3): 3-DoF(Ũ3 , X3 ), 4-DoF (Ũ4 , X4 ), 5-DoF (Ũ5 , X5 ), 6-DoF (Ũ6 , X6 ),7-DoF (Ũ7 , X7 ), and 16-DoF (Ũ16 , X16 ). In 3, 5, 6, and 7DoF conﬁgurations, some joints are coupled to be bilaterally
symmetric. In the 4-DoF, each leg has one DoF. In the 16DoF, only the head link is ﬁxed, while the other joints moveindependently.
The possible freeing directions between these DoF conﬁgurations are deﬁned as shown in Fig. 3. Each arrow showsthat freeing is possible in this direction.
The objective of the crawling task is to move forward as
fast as possible. The objective of the turning task is to turnaround the z-axis as fast as possible. In the both tasks, each
episode begins with the initial state where the robot lies down
and stationary, ends if t > 20[s] or the sum of reward is lessthan −40, which is caused by falling-down penalty. The objective of the maze task is to ﬁnd a path from a start to a(*3)：Open Dynamics Engine: www.ode.org

goal. When the robot reaches the goal, reward 10 is given
and the episode is terminated.
4. 2 Learning Method Configurations
We choose the parameters of LS fusion (denoted as LSF
in the experiments) as follows: fUCB = 2, αR = 0.05,
NLSBh = 10, NLSSp = 20, τlsd0 = 20, δτlsd = 0.004, σth = 0.2,
and faccel = 0.95. The LS-scratch generates a behavior module whose parameters are set as τ0 = 2, δτ = 0.02 for Boltzmann selection of the decreasing temperature parameter τ =τ0 exp(−δτ NepsB ) where NepsB denotes a number of episodes
performed by the behavior module B. The parameters of
WF-DCOB are as follows: Cp (x) = qND , Cd (x) = q̇ND forx ∈ XND , and IR = {(0.05, 0.1), (0.1, 0.2), (0.2, 0.3)} for every conﬁgurations. The LS-accelerating and the LS-freeing
generate a behavior module of the parameters τ0 = 0.1,δτ = 0.02 since the behavior module starts from an almost
converged policy. Every behavior modules use Peng’s Q(λ)learning with γ = 0.9, λ = 0.9, and a decreasing step size
parameter α = min(0.05, 0.3 exp(−0.002NepsB )).
4. 3 Results
We execute 10 runs. Fig. 4 shows the learning curves of
the ﬁrst ﬁve runs of LSF (ex0,...,ex4). In the learning stageof the crawling and the turning tasks, we can ﬁnd that the
scenario (1) is achieved. Fig. 5 shows a learning curve and abehavior module transition in a run obtained in ex0 of Fig. 4.
This graph shows how the set of behavior modules increases.
Fig. 6 shows the resulting learning curves of the task (themean of the return per episode over 10 runs). In learning themaze task, one out of ten runs fails to acquire a path to goal.
The major factor is considered to be a poor connection between the crawling policy and the turning policy. Using suchpolicies as low-level actions may remove the Markov property
from the maze task. Thus, the robot fails in the maze task.
Such a failure will be avoided by mixing the primitive tasks
in the primitive learning stage. This diﬃculty is not speciﬁc to LS fusion, but may arise in the other hierarchical RL
methods. Thus, this failure does not dismiss the claim that
LS fusion has a scalability to complex tasks.
Fig. 7 shows snapshots of an acquired behavior at the endof the maze task. We can see that the robot moves fromstart to goal by using the crawling and the turning motions.

5. Conclusion
This paper proposed the learning strategy (LS) fusion
method where some LSs are integrated for learning a single
task by a single robot. As the LSs, we deﬁned the LS-scratch,the LS-accelerating, the LS-freeing, the LS-planning, and theLS-hierarchy. In the LS fusion algorithm, upper conﬁdencebound (UCB) and Boltzmann selection method are employed
to decide when and which LS is applied.
The proposed LS fusion was veriﬁed in a maze task of thesimulated humanoid robot. In this task, the robot learned
not only a path to goal, but also a crawling and a turning
motions.

(*4)：SkyAI: skyai.sourceforge.net—5—Fig. 7 Snapshots of an acquired behavior at the end of the HumanoidMaze task (taken in 1-FPS).
Fig. 4 Learning curves of five runs obtained from LSF. Each solidline shows the return per episode, and each circle shows
the return acquired by a behavior generated by the LSscratch.

Fig. 5 Learning curve and module transition in a run obtained
from LSF (ex0 in Fig. 4). The dotted line shows the return per episode, each circle shows the return acquired bya behavior generated by the LS-scratch, and the solid line
shows the index of the selected behavior module in each
episode.

Fig. 6 Resulting learning curves of the HumanoidMaze task
where the task changes at 1500, 2500, 3500-th episode.
Each curve shows the mean of the return over 10 runs per
episode.

Our LS fusion architecture has a wide applicability; introducing the other LSs, such as using importance samplingtechnique [8] and imitation learning, is our future work.
Acknowledgements Part of this work was supported
by a Grant-in-Aid for JSPS, Japan Society for the Promotion of Science, Fellows (22·9030).
Bibliography[1] A. Yamaguchi, J. Takamatsu and T. Ogasawara: “Constructing continuous action space from basis functions forfast and stable reinforcement learning”, the 18th IEEE International Symposium on Robot and Human Interactive
Communication (RO-MAN’09), Toyama, Japan, pp. 401–
407 (2009).
[2] J. Morimoto, S. Hyon, C. Atkeson and G. Cheng: “Lowdimensional feature extraction for humanoid locomotion using kernel dimension reduction”, the IEEE Internactional
Conference in Robotics and Automation (ICRA’08), pp.
2711–2716 (2008).
[3] Y. Takahashi and M. Asada: “Multi-layered learning systems for vision-based behavior acquisition of a real mobile
robot”, Proceedings of SICE Annual Conference 2003, pp.
2937–2942 (2003).
[4] J. Kober and J. Peters: “Learning motor primitives for
robotics”, the IEEE Internactional Conference in Robotics
and Automation (ICRA’09), pp. 2509–2515 (2009).
[5] L. Torrey and J. Shavlik: “Transfer learning”, Handbook of
Research on Machine Learning Applications (Eds. by E. Soria, J. Martin, R. Magdalena, M. Martinez and A. Serrano),
IGI Global, chapter 11 (2009).
[6] R. S. Sutton, C. Szepesvári, A. Geramifard and M. Bowling:“Dyna-style planning with linear function approximationand prioritized sweeping”, Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence, pp. 528–536(2008).
[7] L. C. Baird and A. H. Klopf: “Reinforcement learning withhigh-dimensional, continuous actions”, Technical Report
WL-TR-93-1147, Wright Laboratory, Wright-Patterson Air
Force Base (1993).
[8] E. Uchibe and K. Doya:“Competitive-cooperativeconcurrent reinforcement learning with importance sampling”, In Proc. of International Conference on Simulationof Adaptive Behavior: From Animals and Animats, pp. 287–
296 (2004).

—6—