June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Advances in Complex Systems
Vol. 16, Nos. 2 & 3 (2013) 1350015 (38 pages)c World Scientiﬁc Publishing Company

DOI: 10.1142/S021952591350015X

COMBINING CORRELATION-BASEDAND REWARD-BASED LEARNING IN NEURALCONTROL FOR POLICY IMPROVEMENTPORAMATE MANOONPONG∗,†,‡ , CHRISTOPH KOLODZIEJSKI∗,§ ,FLORENTIN WÖRGÖTTER∗,¶ and JUN MORIMOTO∗,†,∗Bernstein Center for Computational Neuroscience,

The Third Institute of Physics,
University of Göttingen, Göttingen 37077, Germany†ATR Computational Neuroscience Laboratories,

2-2-2 Hikaridai Seika-cho,
Soraku-gun, Kyoto 619-0288, Japan‡poramate@physik3.gwdg.de§kolo@physik3.gwdg.de¶worgott@physik3.gwdg.dexmorimo@atr.jp
Received 21 February 2012
Revised 6 February 2013
Accepted 20 February 2013
Published 24 April 2013

Classical conditioning (conventionally modeled as correlation-based learning) andoperant conditioning (conventionally modeled as reinforcement learning or rewardbased learning) have been found in biological systems. Evidence shows that these two
mechanisms strongly involve learning about associations. Based on these biological ﬁndings, we propose a new learning model to achieve successful control policies for artiﬁcial
systems. This model combines correlation-based learning using input correlation learning(ICO learning) and reward-based learning using continuous actor–critic reinforcement
learning (RL), thereby working as a dual learner system. The model performance is evaluated by simulations of a cart-pole system as a dynamic motion control problem and amobile robot system as a goal-directed behavior control problem. Results show that themodel can strongly improve pole balancing control policy, i.e., it allows the controller
to learn stabilizing the pole in the largest domain of initial conditions compared to theresults obtained when using a single learning mechanism. This model can also ﬁnd a
successful control policy for goal-directed behavior, i.e., the robot can eﬀectively learn
to approach a given goal compared to its individual components. Thus, the study pursued here sharpens our understanding of how two diﬀerent learning mechanisms can be
combined and complement each other for solving complex tasks.
Keywords: Classical conditioning; operant conditioning; associative learning; reinforcement learning; pole balancing; goal-directed behavior.

1350015-1

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

1. Introduction
In biological systems, two classes of conditioning for associative learning are known[5]. One is classical conditioning [50] involving presentations of a conditional
stimulus (CS) along with a signiﬁcant or unconditional stimulus (US). The US
generally drives an unconditional response (UCR), usually a reﬂex (e.g., salivationin dogs when they encounter food). Once the US and CS become associated, animals begin to perform a behavioral response to the CS rather than the US where
this response is called a conditional response (CR). This modiﬁcation basically
happens only if the CS is a predictor for the US [56]. Thus, the CS normally precedes the US ([50, 70], but see Ref. [5] for detailed clariﬁcation). Another type ofconditioning is operant or instrumental conditioning [59, 63]. It mainly involvesa reinforcer (i.e., a US) associated with behavior modiﬁcation instead of anotherstimulus. The probability of a speciﬁc behavior is increased or decreased throughpositive or negative reinforcement at each time that the reinforcement is generated.
Although these conditioning or learning mechanisms are diﬀerent from eachother, a number of studies on animal leaning suggest that they may act in combination [5, 15, 35, 47, 55], rather than separately or alternatively, to obtain an
appropriate behavior. Experiments that have supported this idea were presented
in, e.g., Refs. [11, 39, 68]. Williams and Williams [68] observed a pigeon pecking
at an illuminated key in a Skinner box. The results suggest that the desired keypecking behavior CR may be shaped by not only operant conditioninga but also byclassical conditioning; since imposing an omission schedule on the key-light, keypeck association did little to revoke the conditional pecking response. Hence, it
seems that the existing occasional pairing of the key-light CS with the food US was
adequate to drive the pecking behavior (CR), which thus emerged from classicalconditioning. Lovibond [39] performed experiments in rabbits by providing separately trained conditional stimuli during reinforced operant responding. His results
showed that the strength of an operant response can be inﬂuenced by simultaneously
presenting a classically CS. Brembs and Heisenberg [11] conducted experiments inthe fruit ﬂies (Drosophila). Their results showed that there is a situation where
both operant and classical predictors play their roles at the same time, such thatthe situation can be more easily learned than in the separate case.
In animal training, evidence also reveals that many animals including rodents,
dogs, pigeons, dolphins, seals, and whales, can eﬀectively learn to do some sophisticated tasks when they are trained using a combination of these mechanisms [25].
For instance, marine animal trainers use a whistle as predictive information to “tell”
their animals that a reward (e.g., food) is forthcoming. Thus, marine animals learn
to associate the sound of whistle and food (i.e., learning via classical conditioning).
When the animals perform a desired behavior (e.g., come, jump, ﬂip, etc.), theya In this situation, the animal was induced to respond to the key in association with a reward (i.e.,food). This procedure is also called autoshaping.

1350015-2

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

ﬁrst hear the sound indicating that they have performed appropriately and thenthey receive food (i.e., learning via operant conditioning). After several repetitions,the animals will perform a certain behavior as soon as they hear the sound where
they expect to receive food afterwards.
Classical conditioning is often modeled as a form of correlation-based (diﬀerential Hebbian) learning [32, 37, 52, 70] in computational neuroscience. This approach
uses the correlations between external stimuli (i.e., the US and CS) for synaptic
plasticity leading to an anticipatory action (see Sec. 2 for more details). Operant
conditioning is often modeled as reward-based learning or reinforcement learning(RL, e.g., temporal diﬀerence (TD)-learning [7, 62, 67]) in computer science. Thisapproach uses predeﬁned rewards and punishments in the environment as evaluation allowing an agent to maximize or optimize its own expected cumulative
future reward (or expected return). As a consequence, this leads to a corresponding
behavior (see Sec. 3 for more details).
These two conditioning concepts or learning mechanisms have been widely
applied to artiﬁcial agents (robots) for solving various tasks including the generation of self-organizing behavior and autonomous systems [8]. Generally, muchresearch has separately used them to enable agents to learn solving their tasks[7, 10, 20, 38, 40, 42, 51, 53, 62]. In this study, we point out that these two learning
frameworks can complement each other leading to policy improvement. Correlationbased learning can quickly ﬁnd a correlation between a state and an unwanted
condition (i.e., reﬂex or failure recognized by an immediate reﬂex signal), but cannot evaluate whether a given state or weight change predicts something “good” or“bad” which will happen many steps away in the future. Consequently, it cannot
properly learn solving some diﬃcult tasks (e.g., delayed reward tasks) and cannot
explicitly derive a goal-directed policy. On the other hand, reward-based learning
can derive a policy according to the (delayed) reward signal but using it without any
prior knowledge (predeﬁned control parameters), environment or system models, orappropriate guidance generally takes many learning trials to improve the control
performance. Therefore, we combine correlation-based learning (using input correlation learning (ICO learning) [52]) and RL (using continuous actor–critic RL [19])in parallel to let ICO learning extract important features directly used to guidethe learning strategy of continuous actor–critic RL. If we can extract important orproper features for the task, a model of the policy can be simple and the policy canbe easily improved.
To investigate this hypothesis, to show how these two biologically-inspired learning mechanisms can be combined as a neural learning system, and to present its
performance, we chose pole balancing and goal-directed behavior control problems
as two diﬀerent case studies or testbeds. Generally, we are not interested in solving
these two tasks per se. Instead, we would like to show that the proposed combination can solve model-free control problems and is not limited to a speciﬁc task.
Additionally, we would like to suggest that this combination can be an advantageous but simple way (i.e., only combining them in parallel without modifying their
1350015-3

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

learning mechanisms) to solve (dynamic) sensorimotor control problems with continuous signals. Through the performed experiments we hope that this model may
help to better understand interactions between the two learning mechanisms. To acertain extent, the model might be related to neural learning in biological systems
and it may provide a computationally oriented perspective on animal learning.
Finally, we would like to emphasize that this combinatorial learning frameworksuggests how a prior knowledge can be provided to RL and how RL can be guided
and shaped for policy improvement. To our knowledge, this kind of combinatorial
learning method (which is simple, partially related to neural learning mechanisms
in the brain — see the Discussion and Conclusion section below — and leads topolicy improve) has not been investigated and presented so far.
This paper is organized as follows. In Sec. 2 we present the neural circuit of ICO
learning while the neural circuit of continuous actor–critic RL is given in Sec. 3. In
Sec. 4, we introduce our learning model which combines both learning mechanisms
inspired by biological ﬁndings. This model will lead to policy improvement. In Sec. 5,
we demonstrate its performance using the pole balancing and goal-directed behavior
control problems and provide a comparison of diﬀerent learning mechanisms. This
paper ﬁnishes in Sec. 6 with discussion and conclusions.

2. Correlation-Based Learning
For correlation-based learning, we used ICO rule [52] (see Fig. 1) since this learning rule allows implementation of fast and stable learning and it has been also
successfully applied to real robots for obtaining adaptive behavior [40, 42, 53].

Predictive
signals

Neural ICO learning

xN

x2

x

N

O ICO

2

x
1

x1
x
Reflexsignald/dt

d/dt

0= 1.0d/dt
Excitatory synapse
Plastic synapse

x0

Fig. 1. Neural circuit of ICO learning for the multiple plastic synapses of predictive inputs. The
learning rule is derived from diﬀerential Hebbian learning. Here, the output neuron (or learnerneuron) was modeled as a simple linear neuron (see text for details). It generates a continuous
signal for controlling a system.
1350015-4

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

ICO learning is a form of online unsupervised learning where its rule for synaptic
adaptation is based on the cross-correlation of the two types of input signals: Multiple predictive signals (here considered as CS) which are earlier occurring stimuli
and a single reﬂex signal (here considered as a US) which arrives later with certain delays and drives an unwanted response (or reﬂex). The learning goal of ICO
learning is to use a predictive signal ((observable) state of the system) to predict
the occurrence of a reﬂex signal (some exogenous immediate feedback, e.g., reaching a failure state), thereby allowing an agent to react earlier. In other words, thislearning mechanism enables the agent to learn to perform an anticipatory action
to avoid the reﬂex. For example, heat radiation (predictive signal) precedes a pain
signal (reﬂex signal) when touching a hot surface. Thus, we learn an anticipatory
action to avoid the late unwanted stimulus (i.e., avoiding to touch the hot surface).
Normally, the synaptic adaptation of ICO learning changes through heterosynaptic interactions [27] as a consequence of the order of the arriving inputs. If thepredictive inputs are followed by the reﬂex input, the plastic synapses of the predictive inputs get strengthened but they get weakened if the order is reversed. Hence,
this form of plasticity depends on the timing of correlated neural signals. Formally,
we have
OICO (t) = ρ0 x0 (t) +Nρk (t)xk (t)

(1)

k=1

as the output neuron (OICO ) driven by a linear combination of the reﬂex input(x0 ) and the multiple predictive inputs (xk ). N denotes the number of predictive
inputs. ρ0 is the synaptic strength of the reﬂex input. This synaptic strength is set
to a positive value, e.g., 1.0, and remains unchanged, like an innate reﬂex. During
learning, the plastic synapses (ρk ) get changed by diﬀerential Hebbian learning[32, 37] using the cross-correlation between both inputs (i.e., x0 and xk ). This isexpressed as:dρk (t)dx0 (t)= µxk (t),dtdt

k = 1, . . . , N.

(2)

µ is the learning rate which deﬁnes how fast a system can learn. It is generally set
to a value smaller than 1.0. This learning mechanism leads to weight stabilization
as soon as x0 = 0 [52], meaning that the reﬂex has been successfully avoided. As
a result, we obtain behavioral and synaptic stability at the same time without any
additional weight-control mechanisms.
Due to the learning rule, ICO learning can be considered as a model-free method
since its does not require a system or environment model. However, one should notethat ICO learning requires the proper design of a reﬂex into the system from the
beginning. This means that we have to set up a feedback system which has a desired
state and an error signal (x0 → 0) which drives learning. If the tasks become more
complex where a reﬂex cannot be properly designed or no correlation between areﬂex signal and a predictive signal exists at all, ICO learning will fail.
1350015-5

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

3. Reward-Based Learning
For reward-based learning, we used continuous actor–critic RL [19] (see Fig. 2)
since it is capable of generating (multidimensional) continuous actions thus providing a smooth control performance. It is also practical for continuous state-action
problems [19], like dynamic motion control [20, 46]. In addition, it is based on abiological learning model [70] where its learning rule for synaptic adaptation considers an association between stimuli and/or actions with the reinforcement thatan agent receives. Formally, the reinforcement is a reward or a punishment whichis “evaluative feedback” deﬁned by the designers of a system. Thus, this kind of

Neural actor-critic reinforcement learning
Sensory
signals

xN

Actor
wN

x2

w2

x1

w1

ORL

xN

x
Inhibitory synapse

x2

Excitatory synapse

x

x1

x

yM
Sensory
signals

RBF
HMx

Critic

y2
x

xN
x2

Plastic synapse

y1

H2

x1

M

x
V (t)
V (t- )

H1

2

Rewardsignal

1

R
Fig. 2. Neural circuit of continuous actor–critic RL. The learning mechanisms of the actor andcritic are based on TD learning. The actor component was modeled as a stochastic neural network
while the critic unit was modeled as a radial basis function (RBF) neural network (see text for
details). Note that in this framework, the actor provides a continuous output signal for controllinga system.
1350015-6

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learninglearning mechanism is minimally supervised because an agent is not told explicitly
what actions to take in a particular situation. Rather it must work this out for itself
on the basis of the reinforcement.
Continuous actor–critic RL is divided into two sub-mechanisms: The learningof an action function (actor) and the learning of an evaluation function (critic).
The action part is the controller of an agent. In this study, it was designed as astochastic unit proposed in Ref. [24]. If we consider one-dimensional output, its
output (ORL ) is speciﬁed by:
ORL (t) = ε(t) +Nwk (t)xk (t),

(3)

k=1

where N denotes the number of sensory inputs (xk ) which, here, are comparable
to the predictive inputs of ICO learning. ε is an exploration term. According toRef. [19], it is varied based on a modulation schemeb given by:Vmax − V (x(t)).
(4)ε(t) = ξσ(t) · min 1, max 0,
Vmax − Vmin
σ is the Gaussian distributed noise with zero mean and standard deviation of one. V
is a value function (see its equation below) that estimates the expected cumulative
future reward or the expected return where the reward is used to estimate how good
it is for an agent to be in a given state. Vmax and Vmin are the maximal and minimalvalues of V . This way, the exploration is large if V is close to Vmin . On the other
hand, the exploration is small (close to zero) if it is close to Vmax meaning thatlearning shows good prediction or the performance is improved. ξ is an additional
scale factor. It is introduced in order to be able to amplify the exploration level.
The stochastic unit is related to two biological learning concepts, called behavior
oscillation [26] and successive approximation [59] (see also Ref. [24] for more details).
During learning, the synaptic weights (wk ) of the actor change over time. They arebasically changed by a stochastic RL algorithm [24]. Instead of using the error ofa direct reward, which is one of the learning parameters and originally used in
the stochastic RL algorithm, here we used the TD error [7, 19] (i.e., the error of
an internal reward [7]). By doing so, delayed reward control problems can also be
solved [7, 19]. The equation of the weight adaptation is described by:
dwk (t)= αδ(t)xk (t)ε(t), k = 1, . . . , N,(5)dt
where α is the learning rate and generally set to a value smaller than 1.0. δ isan approximation to the TD error in continuous time described as an internal
reinforcement signal provided by the critic (see below).
b The scheme follows the intuition that an agent should explore a lot if its expected cumulative

future reward V is small (close to Vmin ). This means that it has a poor control policy. On the other
hand, it should exploit or follow the control policy if V is close to Vmax . However, this normallyworks if Vmin and Vmax could be estimated.
1350015-7

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

For the critic network, according to Ref. [45] we used a radial basis function(RBF) neural network as a function approximator which attempts to construct theapproximation of the value function V . It is governed by:
V (x(t)) =Mvj (t)yj (x(t)),

(6)

j=1

where yj are the outputs from the normalized Gaussian basis functions given by:aj (x(t)),
yj (x(t)) = M
l=1 al (x(t))

T2

aj (x(t)) = e−sj (x(t)−cj ) .

(7)

The vectors cj and x deﬁne the center and the input feature, respectively. sj is
the diagonal matrix of the inverse covariance of the RBF neural network. M is thenumber of hidden neurons. According to Ref. [19], vj are synaptic weights which
are updated by:
dvj (t)= λδ(t)yj (x(t)), j = 1, . . . , M,(8)dt
where λ is the learning rate. It is generally set to a value smaller than 1.0. According
to Ref. [19], the TD-error δ is basically computed from the prediction as follows:1
V (x(t)) + V̇ (x(t)),(9)τ
where R is an external reinforcement signal provided by designers. τ is the time
constant of a discount factor. V is the value function [see Eq. (6)] and V̇ is its
derivative with respect to time. Note that using the Euler discretization, the TD
error in continuous time is compatible to the conventional TD error [62]: δ(t) =R(t) + γV (x(t)) − V (x(t − 1)) where γ = 1 − ∆t
τ is the discount factor and ∆t is
the time step of the Euler diﬀerentiation.
δ(t) = R(t) −4. Combining Correlation-Based and Reward-Based Learning
In the previous sections we have presented ICO learning and continuous actor–
critic RL. It is known that ICO learning can quickly learn a correlation between
a failure state recognized by an immediate reﬂex signal and a failure avoidance
behavior (or also called reﬂex avoidance behavior) controlled by predictive signals[52]. However, it cannot evaluate whether a given state or weight change predicts
something “good” or “bad” which will happen many steps away in the future. As aconsequence, this makes it diﬃcult for the controller to properly learn solving somediﬃcult tasks (see Sec. 5). On the other hand, continuous actor–critic RL can makepredictions through its evaluation process such that it can solve the tasks but ingeneral it is slower than ICO learning (see Sec. 5). In addition, due to its stochastic
process it requires several learning repetitions to ensure that a successful control
policy has been achieved.
Thus, in this section, we introduce a combinatorial learning model. It makes useof the advantage of each learning mechanism, resulting in an appropriate acquisition
1350015-8

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

Learned weights

Neural ICO learning

Agent

Feedback

Neural actor-critic
reinforcement learning

Agent

Feedback

Fig. 3. Sequential combination model. ICO learning ﬁrst learns to ﬁnd a solution controllinga system without any prior knowledge. Afterwards the learned weights from ICO learning areprovided to continuous actor–critic RL for initialization. Finally, continuous actor–critic RL serves
as an add-on learning process to enhance the performance of a controller (see Ref. [43] for moredetails).

of the control policy that outperforms either ICO learning or continuous actor–critic
RL alone (see Sec. 5). Basically, there are two ways of combining ICO learning andcontinuous actor–critic RL: sequential or parallel.
Sequential combination (see Fig. 3), which we investigated previously [43], isachieved by initially using ICO learning to extract reward-related features for continuous actor–critic RL. Afterwards continuous actor–critic RL uses the extracted
features as priors (i.e., initial control parameters) to improve the control policy ofthe system. However, the drawback of this learning scheme is that it is technicallyinconvenient since we need to let the ICO learning mechanism learn the wholefeature space ﬁrst such that reward-related features are properly extracted.
In contrast, the parallel combination, proposed in this study and later called
here combinatorial learning (see Fig. 4), is technically more convenient since itallows these two learning mechanisms to simultaneously learn, thereby working asa dual learner system. By doing so, they receive sensory feedback from the agent
in parallel and adapt their weights accordingly. Their output signal contributes
equally to the control of the agent. Thus, the ﬁnal output (OCOM ) is described as:
OCOM (t) = ζ · (OICO (t) + ORL (t)),(10)

Neural ICO learning OICO
OCOM

Agent

Neural actor-critic
reinforcement learning ORL
Feedback
Fig. 4. Combinatorial learning model. It combines ICO learning and continuous actor–critic RL
in a parallel manner for controlling an agent. In this learning scheme, each learning mechanism
develops its weights independently, but they are coupled by sensory feedback. This way, theybasically coadapt the control parameters (i.e., weights) leading to the improvement of the control
policy.
1350015-9

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

where OICO and ORL are the outputs of ICO learning and continuous actor–critic
RL, respectively. ζ is a scale factor which is introduced to ensure that the sum
is a valid control signal. The complete algorithm of combinatorial leaning withpseudocode is shown in Table 1.
In this learning scheme, ICO learning and continuous actor–critic RL can complement each other due to their learning principles. ICO learning [see Eq. (2)]relies on the predeﬁned reﬂex signal, while continuous actor–critic RL [see Eq. (5)]depends on the TD error (δ) based on the estimated value function and the reward.
As a consequence of the reﬂex avoidance learning principle, weight adaptation of
ICO learning is initially more relevant than that from continuous actor–critic RL(until the value function is properly estimated). Thus, in some situations, like a polebalancing task (see Sec. 5.1), ICO learning quickly updates weights (i.e., control

Table 1.

Combinatorial learning algorithm.

Initialize ρk , wk , and vj to 0.0; ε = Gaussian random number
Repeat:At time step t(1) observe reﬂex signal x0 and sensory signals xk which are the state x(2) compute control output
OICO ← ρ0 x0 +

N
X

ρk xk

k=1

ORL ← ε +NXw k xkk=1

OCOM ← ζ · (OICO + ORL )(3) perform action(4) observe reward R, new state x and new reﬂex signal x 0(5) obtain value function by computingTaj ← e−sj (x−cj )
aj
yj ← P M
l=1 al
V ←

M
X

2

vj yj

j=1

ii
h
h
max −V (x)(6) compute ε ← ξσ · min 1, max 0, VV−V(7) compute δ ← R + γV (x ) − V (x)(8) update control parameters

max

min

ρk ← ρk + µxk (x 0 − x0 )wk ← wk + αδxk ε
vj ← vj + λδyj
Until: Successful control policy is found or the maximum number of trials
is reached.
1350015-10

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learningparameters) to enhance or guide the entire learning process that includes continuous
actor–critic RL. At the same time, ICO learning also utilizes the exploration strategy of continuous actor–critic RL to indirectly adapt its weights. In other situations,like a goal-directed behavior task (see Sec. 5.2), ICO learning plays roles on guiding continuous actor–critic RL to receive a reward and shaping a control policy.
However, if reﬂex signal and TD error disagree, ICO learning may interfere with
continuous actor–critic RL (see also Sec. 6 for more discussion on this).
Beside this, one important property of our approach is that we directly use
sensory inputs as the state of a system (i.e., continuous state) without resortingto the explicit discretization of states and actions. Thus, this approach is capableof generating a continuous action leading to smooth control performance. It isalso practical for continuous state-action problems (e.g., pole balancing and goaldirected behavior shown below) in particular in the domain of model-free control
problems because of the learning rules (i.e., correlation-based learning and rewardbased learning) which do not require a system or environment model.
5. Experiments and Results
We tested the performance of our combinatorial learning in two diﬀerent tasks:
A dynamic motion control task using a simulated cart-pole system and a goaldirected behavior control task using a simulated mobile robot system. In each ofthem we compared the performance of three control schemes: ICO learning, continuous actor–critic RL, and combinatorial learning. In addition, we also investigatedinteractions between ICO learning and continuous actor–critic RL by observing
learning curves in order to understand their roles in combinatorial learning. It is
important to note that the aim of this study is not to claim that the combination
outperforms other/older methods for solving the tasks or model-free optimal control
problems. Thus, comparing our combinatorial learning with other baseline methods (like, dynamic programming) will go beyond the scope of this work. Instead,
we emphasize here that a combination is better than its individual components by
utilizing the learning properties of ICO learning and continuous actor–critic RL.
5.1. Dynamic motion control
In this section, we demonstrate the performance of combinatorial learning (seeFig. 4) applied to a pole balancing problem [7] (see Fig. 5). The task was to balance an inverted pendulum, which is mounted on a cart moving freely in a onedimensional interval, and to simultaneously avoid the interval boundaries. This
cart-pole system was simulated on a desktop PC and updated by using a fourthorder Runge–Kutta method with a time step of 0.01 s. It provides four state variables: The angle of the pole with the vertical (θ), the pole angular velocity (θ̇), theposition of the cart on the track (x), and the cart velocity (ẋ). Similar to Ref. [7],the cart was bound to move in the interval −2.4 ≤ x ≤ 2.4 [m] and the angle was
1350015-11

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

Cart-pole system
F
Gain

Neurallearning

X
X=0
Feedback ( , , X ,X)

Fig. 5.

Cart-pole system for a dynamic motion control task (see text for details).

allowed to vary in the interval −12 ≤ θ ≤ 12 [◦ ]. The dynamics of the cart-polesystem is modeled by:


µp θ̇
−F − mlθ̇2 sin θ + µc sgn(ẋ)−
g sin θ + cos θ
M +m
ml


,
(11)
θ̈ =
2
4 m cos θ
−
l
3
M +m
ẍ =

F + ml(θ̇2 sin θ − θ̈ cos θ) − µc sgn(ẋ)
,
M +m

(12)

where g = 9.8 m/s2 denotes gravitational acceleration, M = 1.0 kg and m = 0.1 kg
are mass of the cart and pole, respectively. l = 0.5 m is half of the pole length.
µc = 5.0 × 10−4 and µp = 2.0 × 10−6 are friction coeﬃcient of the cart and pole,
respectively. F is a continuous force applied to the cart which is directly derived
from the output of learning mechanisms with an ampliﬁer gain of 10.0. Note that
all these parameters and the cart-pole equations are generally used [7, 49].
In fact, this task is diﬃcult in its own right due to the limited boundaries ofthe pole angle and in particular the cart position. The boundaries are used as astandard benchmark setup in most control studies [7, 49]. In addition, its vertical
upright equilibrium point to be balanced is inherently unstable (i.e., as any small
disturbance may cause the pole to fall on the either side). From this setup, balancing
the pole at critical initial conditions (e.g., θ = 11 deg, x = 2.1 m) close to theboundaries is already diﬃcult to ﬁnd successful control policies by using a simple
reward function.
In this setup, the four state variables (x, ẋ, θ, θ̇) of the system were used as
sensory feedback (x1,2,3,4 ) to ICO learning [see Eq. (1)] and continuous actor–critic
RL [see Eqs. (3) and (7)]. For ICO learning, these state variables were scaled onto
the interval [−1, 1] similar to Ref. [49] and the reﬂex signal [x0 , see Eq. (1)] wasgiven just before the system failed. The signal shows a positive activation (+1.0) if
x < −2.35 m or θ > 11.5◦ , a negative activation (−1.0) if x > 2.35 m or θ < −11.5◦,1350015-12

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learningand 0 otherwise. Here, we set the learning rate [µ, see Eq. (2)] of ICO learning to
0.1. Note that the weights [ρ1,2,3,4 , see Eq. (2)] are changed only for the positive
derivatives of the reﬂex signal, otherwise they remain unchanged. This is to avoid
negative correlations resulting in poor performance.
For continuous actor–critic RL, we allocated three bases for x, three for ẋ, sixfor θ, and three for θ̇ according to a boxes approach (see Ref. [7] for more details).
This leads to 3 × 3 × 6 × 3 = 162 bases employed as the centers of the critic network.
Thus, the network has in total 162 hidden neurons [M = 162, see Eqs. (6) and (7)]which cover the state space of the system. The size or width of the Gaussian basis
functions was simply set to twice the distance between its center and the centerof its nearest neighbor. The reward signal [R, see Eq. (9)] was set to −1 at failure(i.e., cart hits the boundaries or pole falls to ± 12 deg) and 0 otherwise [7]. Here,
Vmax and Vmin of the modulation scheme controlling the level of the exploration [ε,see Eq. (4)] were set to 0 and −1, respectively. In this setup, we set the scale factor(ξ) of the exploration to 5.0. This is to obtain a better performance. Thus, largechanges of the weights of continuous actor–critic can occur. The control parameters
α [see Eq. (5)], λ [see Eq. (8)], τ [see Eq. (9)], and ζ [see Eq. (10)] were set to 0.5,
0.5, 0.2, and 0.5, respectively.
We let the combinatorial learning mechanism learn to balance the pole on 25×49
initial conditions (θ, x) while θ̇ and ẋ were initially set to small random values using
a Gaussian distribution with zero mean and a standard deviation of 0.1% of signal
ranges which represents the system noise. Note that the control parameters (i.e.,
synaptic weights) of ICO learning and continuous actor–critic RL were initially set
to 0.0. During a run each trial started with a given initial state and ended either in“success” (which occurs when the pole is kept in balance for at least 5 × 104 time
steps or 500 s) or “failure” (which occurs when the pole falls 12 deg to either side or
the cart moves 2.4 m to either side). Runs at each initial condition were terminatedon failure or when a successful trial was achieved or the maximum number of trials
was reached (here 1000 trials). The system was reset to the same initial state at
failure. We repeated this for 25 experiments at each initial condition.
The performance of combinatorial learning is shown in Fig. 6(a). It can be seenthat this learning mechanism was able to ﬁnd successful control policies which canbalance the pole and avoid the ends of the interval in a very large (x, θ)–domain of
initial conditions [see Fig. 6(a)]. The system was successfully stabilized for ≈ 96%
of all initial conditions. This is because, on one hand, ICO learning utilizes the
exploration strategy of continuous actor–critic RL to explore its parameter spacesuch that a proper weight combination is obtained. At the same time continuous
actor–critic RL is also guided by the built-in reﬂex of ICO learning to adapt its
weights in a proper way. The remaining part (black areas), at which learning failed,
is because of physical limitation. For example, if the system stands close to the rightwall and the pole falls to the right, the cart momentum cannot be high enough to
support the pole. Thus, it crashes into the wall before. The results we obtained
here are comparable to the ones shown in Ref. [49] where this work employed
1350015-13

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

Combinatorial learning

Success

Success
rate %

Continuous actor-critic RL rate %
100

100

2
1.5
1
0.5
0
-0.5
-1
-1.5
-2

2
1.5
1
0.5
0
-0.5
-1
-1.5
-2

90
80
70
50
40
30

-10

-5

0

deg

~
~ 96%

20

5

0

10

10

80
70
60

X [m]

X [m]

60

90

50
40
30

~
~ 91%
-10

-5

0

deg

(a)

5

10

20
10
0

(b)

Success
rate %ICO learning

100

2
1.5
1
0.5
0
-0.5
-1
-1.5
-2

90
80
70

X [m]

60
50
40
30

~
~ 75%
-10

-5

0

deg

5

10

20
10
0

(c)
Fig. 6. Performance of three learning mechanisms for the pole balancing problem. (a) Successful
control area (≈ 96%) of combinatorial learning on benchmark initial conditions. (b) Successful
control area (≈ 91%) of continuous actor–critic RL. (c) Successful control area (≈ 75%) of ICO
learning. Black areas represent a domain in which learning failed to solve the problem (i.e., itcannot learn to stabilize the system). A gray scale bar presents the success rate, i.e., the percentage
of success from 25 experiments. Recall that “success” means the pole is kept in balance for atleast 500 s.

similar linear control with four inputs but used an evolutionary algorithm for weightadaptation.
When only ICO learning or continuous actor–critic RL alone was applied, stabilizing the system was accomplished in smaller domains. For ICO learning alone,
the system was balanced for ≈ 75% of all initial conditions [see Fig. 6(c)]. This isbecause ICO learning cannot explore the entire parameter space due to the lack ofan exploration mechanism and it even cannot predict many steps into the future.
It basically develops its weights (i.e., control parameters) with respect to an immediate correlation between predictive and reﬂex signals. In this setup, the built-in
reﬂex occurs only at the last moment that the pole falls or the cart hits the wall.
1350015-14

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

Therefore, at initial conditions in which the system fails, the reﬂex signal cannot
produce a strong cart momentum to turn the pole into an upright position or keep
it balance for a certain period of time (i.e., avoiding the reﬂex). Thus, ICO learning cannot obtain a proper correlation between the predictive and reﬂex signals toachieve a proper weight combination. For continuous actor–critic RL alone, due tothe lack of a prior knowledge, the system was balanced for ≈ 91% of all initial conditions [see Fig. 6(b)]. This experimental result shows that, among the three learning
mechanisms, combinatorial learning, which combines ICO learning and continuous
actor–critic RL, was the best approach with respect to the success rate.
Note that, due mainly to the stochastic process of continuous actor–critic RL
and partly to the introduced system noise which can easily destabilize the system,
combinatorial learning and continuous actor–critic RL alone sometimes had diﬃculty or failed to ﬁnd successful control policies in a given number of trials at, e.g.,x = −2.0 m, θ = 12 deg and around x = 0.0 m, θ = 0 deg, respectively. In contrast,
ICO learning alone was almost 100% success at these initial conditions and evenshowed the very clear boundary between white and black areas [see Fig. 6(c)] since
it is deterministic control where no exploration is involved.
To compare the learning speed of these three learning mechanisms in generalcases, we observed their performance at a noncritical initial condition (e.g., x =1.0 m, θ = −1 deg), where they all can ﬁnd successful control policies, and at a
critical initial condition (e.g., x = −1.8 m, θ = −5 deg) where combinatorial learningand continuous actor–critic RL can ﬁnd the policies but ICO learning cannot. The
result is shown in Fig. 7.
At the noncritical initial condition ICO learning was fastest, combinatorial learning was slower, and continuous actor–critic RL was the slowest [see Fig. 7(a)]. At thecritical initial condition ICO learning failed while continuous actor–critic RL succeeded but required more learning trials compared to combinatorial learning. This
experiment suggests that the fast convergence property of combinatorial learning
is generally derived from ICO learning which can quickly learn to ﬁnd a solution
for a task but cannot properly learn solving a diﬃcult task (i.e., here, stabilizing
the system at a critical initial condition). Furthermore, the capability for solving
a diﬃcult task is basically obtained from continuous actor–critic RL which learns
the task but usually takes many learning trials.
To better understand why the combined mechanism outperforms either ICO
learning or continuous actor–critic RL alone, we also observed learning curves ata critical initial condition (e.g., x = 2.1 m, θ = 11 deg) at which its individualcomponents failed. Figure 8 shows that control parameters (i.e., synaptic weights)
converged to ﬁxed values when combinatorial learning was used [see thick lines in
Figs. 8(a) and 8(b)]. This is because ICO learning and continuous actor–critic RL
tried to ﬁnd a proper weight combination such that a proper force is generated topush the cart for balancing the pole. This proper combination can be seen when theweight for one input in ICO learning increased and that in continuous actor–critic
RL network decreased (e.g., ρx , wx in Fig. 8).
1350015-15

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.
ICO learning

100 Combinatorial
learning

80

Combinatorial
learning

Success rate %Success rate %

100

60
40

Continuous
actor-critic RL

20

80
Continuous
actor-critic RL

60
40
20

0

ICO learning

0
0

100

200

300

400

500

Total learning trials

0

100

200

300

400

500

Total learning trials

(a)

(b)

Fig. 7. (Color online) Comparison of the performance of three learning mechanisms. (a) Successrate according to total learning trials at the noncritical initial condition (x = 1.0 m, θ = −1 deg).
(b) Success rate according to total learning trials at the critical initial condition (x = −1.8 m,
θ = −5 deg). At this critical initial condition, ICO learning failed because during learning its
weights grew more and more. Thus, the output of ICO learning, applied to the cart-pole system,
also strongly increased. As a result, the output disturbed the system rather than balancing it.
Note that we did not limit the output. Recall that success rate is calculated from the percentage
of success in the total 25 experiments after a certain number of trials where “success” means thepole is kept in balance for at least 500 s. Dashed lines indicate the average of the total learning
trials at success.

The behavior of the system controlled by all converged weights is shown in
Fig. 9. Due to the proper weight combination, at the beginning a proper positive
force was generated to push the cart to the right such that the pole could swing to
the left to obtain an upward position. Afterwards, a negative force was generated tobalance the pole and push the cart to the center. Finally, the system was stabilized
at the center where all inputs were converged to zero values, thereby no force was
generated. As a result, the pole was successfully balanced. If the converged weightsof the ICO learning module were only used to control the system while the weightsof the continuous actor–critic RL module were set to 0.0, the controller produced
a very strong positive force to the cart at an early state. As a consequence, thepole fell to −12 deg (see Fig. 10). On the other hand, if the converged weights ofthe continuous actor–critic RL module were only used to control the system while
the weights of the ICO learning module were set to 0.0, the controller produced avery strong negative force at an early state, thereby making the pole quickly fall to
12 deg (see Fig. 11). For these two cases, the system could not be stabilized since
the forces were not properly generated.
When ICO learning alone [see transparent lines in Fig. 8(a)] was used to learn tobalance the pole at the critical initial condition (x = 2.1 m, θ = 11 deg), its control
parameters diverged since the reﬂex signal cannot be avoided (i.e., the pole always
fell). Although ICO learning is designed to learn to avoid a reﬂex signal, in this polebalancing setup the built-in reﬂex occurs only at the last moment that the pole falls
1350015-16

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

Weights (ICO learning)

9

x
6

3

x
0
0

2000

4000

6000

8000

Time [steps]

Weights (actor-critic RL)

(a)

W

4

W

2
0

Wx

-2

Wx

-4
0

2000

4000

6000

8000

Time [steps](b)
Fig. 8. (Color online) Learning curves at a critical initial condition (x = 2.1 m, θ = 11 deg).
(a) Weight changes in ICO learning. (b) Weight changes in continuous actor–critic RL. Thick lines
present the weight changes in each learning mechanism in the combinatorial learning framework
while transparent lines show the weight changes when only ICO learning or continuous actor–
critic RL was used. Using combinatorial learning, the weights became stable after around 6500
time steps (or 70 trials) meaning that the system was successfully stabilized. In contrast, theweights diverged when only ICO learning [see (a)] was used while they changed a lot in the case ofcontinuous actor–critic RL alone [see (b)]. Note that sudden change in wx occurred (e.g., around4400 steps) because there was a high correlation between the TD error and the input (x) while
there were low correlations between the other inputs and the TD error. Here, ρx = ρ1 , ρẋ = ρ2 ,
ρθ = ρ3 , ρθ̇ = ρ4 , wx = w1 , wẋ = w2 , wθ = w3 , wθ̇ = w4 .

or the cart hits the wall. Therefore, in this diﬃcult situation, a reﬂex signal cannot
produce a strong cart momentum to turn the pole into an upright position or keep itbalance for a certain period. Thus, ICO learning cannot obtain a proper correlation
between the predictive and reﬂex signals; thereby its weights just increased more
and more to try to ﬁnd any proper weight combination. However, in this situationsuch a combination leading to a proper force cannot be achieved. While the weights
were increasing, the output of ICO learning, applied to the cart-pole system, also
1350015-17

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

X [m]

1

Force[N][deg/s]

[deg]

2
1
0

X [m/s]

a

a b

b

c

d

c

d

0
-1
10
0

0
-50
-100
20
0
0

300

600

900

1200

Time [steps]
Fig. 9. States of the cart-pole system (x, ẋ, θ, θ̇) and the force under control of the learned
weights (ρx ≈ 8.0774, ρẋ ≈ 1.8395, ρθ ≈ 4.2815, ρθ̇ ≈ 3.0069, wx ≈ −2.5790, wẋ ≈ 1.9225,
wθ ≈ 2.4527, wθ̇ ≈ 3.2216, see e.g., Fig. 8) for the critical initial condition x = 2.1 m, θ = 11 deg.
A series of photos visualizing the cart-pole behavior at particular points is shown above.

increased. As a result, at some point the output disturbed the system rather thanbalancing it. In the case of continuous actor–critic RL alone [see transparent lines
in Fig. 8(b)], the control parameters changed a lot due to the stochastic process
employed which tried to search for a successful control policy. Note that at the earlystate of learning the weights of the ICO learning module in combinatorial learning
became larger than the weights of ICO learning alone due to the stochastic processof the continuous actor–critic RL module in combinatorial learning. It can easily
destabilize the system. Thus, the pole can often fall at the early state. This leads
to the triggering of a reﬂex signal. On the other hand, in the case of ICO learning
alone the pole fell less often at the early state such that the weights grew slower.
Finally, we investigated interactions between these two learning mechanisms.
We ﬁrst started one learning mechanism and then after a number of learning trials(e.g., 100 trials) we activated the other one (see Fig. 12). This is to observe two
eﬀects: (i) Can a later activated learning mechanism assist an earlier activatedlearning mechanism for policy improvement? and (ii) Can the earlier one provide
1350015-18

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

X [m]Combining Correlation-Based and Reward-Based Learning

2.4
2.2

X [m/s]

4
2
0

[deg]

2.0

12
0
-12

[deg/s]Force[N]

0
-150

60
40
20
0

-300

0

1

2

3

4

5

6

7

8

9

10

11

Time [steps]
Fig. 10. States of the cart-pole system (x, ẋ, θ, θ̇) and the force under control of the learned
weights (ρx ≈ 8.0774, ρẋ ≈ 1.8395, ρθ ≈ 4.2815, ρθ̇ ≈ 3.0069, [see Fig. 8(a)] for the critical initialcondition x = 2.1 m, θ = 11 deg. We set wx , wẋ , wθ , and wθ̇ to 0.0.

an appropriate developed control policy to the later one such that a successful
control policy can still be achieved at the end?
Figures 12(a) and 12(b) show learning curves when continuous actor–critic RL
was ﬁrst started and followed by ICO learning after 100 trials [see dashed line in
Fig. 12(a)]. It can be observed that after around 5500 time steps (or 130 trials),
where ICO learning was already activated, the weight (wx ) of continuous actor–
critic RL started to gradually change its growing direction into a diﬀerent way[see e.g., thick line in Fig. 8(b)]. A similar eﬀect also appears for the weight (wẋ )after around 9000 time steps (or 190 trials). This is because ICO learning canquickly ﬁnd a correlation between a state and an unwanted condition (i.e., polefalls) and additionally generates the proper action when the pole falls through its
built-in reﬂex. Thus, it can extract important featuresc serving to guide the learning
strategy of continuous actor–critic RL. As a result, the weights of continuous actor–
critic RL (e.g., wx , wẋ ) gradually changed to their appropriate directions but they
did not change considerable compared to continuous actor–critic RL alone [seetransparent lines in Fig. 8(b)].

c By feature we mean the combination between the weights and input signals.

1350015-19

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

[deg]

X [m/s]

X [m]

2.10
2.09
2.08
0.0
-0.4
-0.8
12.0
11.5

[deg/s]

60
40
20
0

Force[N]

11.0

0
-10
-20
-30
0

1

2

3

4

Time [steps]
Fig. 11. States of the cart-pole system (x, ẋ, θ, θ̇) and the force under control of the learned
weights (wx ≈ −2.5790, wẋ ≈ 1.9225, wθ ≈ 2.4527, wθ̇ ≈ 3.2216, [see Fig. 8(b)]) for the critical
initial condition x = 2.1 m, θ = 11 deg. We set ρx , ρẋ , ρθ , and ρθ̇ to 0.0.

Another interesting eﬀect of the interaction is shown in Figs. 12(c) and 12(d)where ICO learning was ﬁrst started and followed by continuous actor–critic RL
after 100 trials [see dashed line in Fig. 12(d)]. After 115 trials (or around 2500 time
steps), the pole did not fall anymore leading to reﬂex avoidance. As a consequence,
the weights of ICO learning converged. However, the weights of continuous actor–
critic RL still slightly changed due to the TD error. They ﬁnally converged (i.e.,
TD error ≈ 0) after around 17,600 time steps. This experiment shows that, on the
one hand, continuous actor–critic RL seems to highly inﬂuence ICO learning suchthat the weights of ICO learning became stable shortly after continuous actor–critic
RL was activated. On the other hand, ICO learning seems to provide an adequate
control policy or an important feature to continuous actor–critic RL such that it
can quickly adapt its weights to appropriate directions leading to convergence.
5.2. Goal-directed behavior control
Next, we present the performance of combinatorial learning (see Fig. 4) on a different task. Here, we employed it to a goal-directed behavior control problem. The
task was to steer a wheeled mobile robot to move toward and ﬁnally approach adesired object (i.e., its goal) in a given time. In this scenario, we put the robot in
1350015-20

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

x

24
18
12

x

6

12

x
9

x
6
3
0

0
14

W
7

W

0

Wx
-7

Wx
-14

Weights (ICO learning)

30

0

5

10

15

20

25

3
30x10

Weights (actor-critic RL)

Weights (actor-critic RL)

Weights (ICO learning)

Combining Correlation-Based and Reward-Based Learning

Time [steps]

2

W

1

W
0
-1

Wx
Wx

-2
0

5

10

15

3
20x10

Time [steps]

Fig. 12. (Color online) Learning curves at a critical initial condition (x = 2.1 m, θ = 11 deg)when ICO learning and continuous actor–critic RL were not activated at the same time. (a), (b)Weight changes of ICO learning and continuous actor–critic RL in the combinatorial learning
framework. In this experiment, ICO learning was activated after 100 trials (dashed line). (c), (d)Weight changes of ICO learning and continuous actor–critic RL where here continuous actor–critic
RL was activated after 100 trials (dashed line).

a square area where one desired green object and one undesired blue object were
provided. We used the physics simulator LPZROBOTSd to simulate the robot andits environment (see Fig. 13). The simulator was implemented on a desktop PC
with an update time step of 0.01 s.
The mobile robot system provides four state variables, which are two relativeorientations (φG,B ) and two relative positions (DG,B ) of the robot to the locations of
the green (G) and blue (B) objects, and additional eight state variables of infrared(IR) sensors for boundary detection (see Fig. 13). φG,B provide information of
how much the robot’s direction deviates from the objects. They vary in the interval[−180◦, 180◦ ] [see Fig. 13(b)] and show continuous values. If the objects are directlyin front of the robot, φG,B show 0. If they are to the left-hand side of the robot,
φG,B show negative values. If they are to the right-hand side of the robot, φG,B havepositive values [see e.g., Fig. 13(b)]. DG,B provide information of how close the robot
is to the objects. They are mapped onto the interval [0, 1], with 0 representing near,and +1 representing far. If the robot comes close to an object in a certain range[i.e., DG,B > 0.7, see dashed areas in Fig. 13(c)], a reward is given for continuous
d It is based on the open dynamics engine (ODE) for more details of the LPZROBOTS simulator

see http://robot.informatik.uni-leipzig.de/software/.
1350015-21

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

Fig. 13. (Color online) Simulated mobile robot system for a goal-directed behavior control task.
(a) The mobile robot with diﬀerent types of sensors (i.e., relative orientation φ and position D
sensors and infrared IR sensors). (b) The variation of the relative orientation φG of the robot to
the green object. (c) The environmental setup of the robot. The black dot represents the startingor reset position where the robot was initially started or reset after it hit a boundary or reached
one of the objects. Dashed circles show areas where a positive (+1) or negative reward (−1) was
given for continuous actor–critic RL and reﬂex signals were triggered for ICO learning (see text
for details).

actor–critic RL and a reﬂex signal is triggered for ICO learning. The IR sensory
signals are mapped onto the interval [−1, +1], with −1 representing no boundary
detection, and +1 representing hitting a boundary. This IR information was onlyused to reset the robot position on hitting a boundary.
It is important to note that in this setup, only φG,B were used as inputs (i.e.,the state) to the control policy while DG,B were used only to generate reward andreﬂex signals for learning and to reset the robot position when approaching an
object (i.e., DG,B > 0.95). Thus, the robot has insuﬃcient sensor data for reliably
identifying its state in the environment. Furthermore, φG,B overlap with each other,i.e., the robot simultaneously senses its relative orientation to the locations of bothobjects in the whole area. Thus, both sensor signals try to steer the robot toward
the corresponding objects once their synaptic weights have been developed.
Here, for ICO learning, φG,B were used as predictive signals [x1,2 , see Eq. (1)].
Two independent reﬂex signals were conﬁgured: One was for the green object [x0G ,see Eq. (1)] and the other for the blue one [x0B , see Eq. (1)]. They depend on theorientations (φG,B ) and the positions (DG,B ) of the robot to the objects. The reﬂex
signals are triggered as soon as the robot comes close to the objects [i.e., entering
areas inside the dashed circles as shown in Fig. 13(c)], and 0 otherwise. In fact,
the reﬂex signals elicit a turn which is proportional to the deviations deﬁned by
φG,B , i.e., the larger the deviations, the sharper the turn. Thereby, they turn the
robot toward the objects. In other words, ICO learning tries to control the heading
direction of the robot to align with an object. This way, it can implicitly optimize
1350015-22

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

the behavior over the entire path. Since the green and blue objects are far fromeach other, the reﬂex areas do not overlap. Thus, the two reﬂex signals cannot be
triggered at the same time. We set the learning rate [µ, see Eq. (2)] of ICO learning
to 0.005. The weights [ρ1,2 , see Eq. (2)] were initially set to 0.0. They change only if
the positive derivatives of the reﬂex signals are higher than a threshold, otherwise
they remain unchanged. For example, when the robot comes close to the green
object and the reﬂex signal is triggered, the weight (ρ1 ) of the orientation signal
with respect to the green object increases while the weight (ρ2 ) of the blue one
remains unaﬀected and vice versa when the robot comes close to the blue object.
For continuous actor–critic RL, we allocated four bases for φG and φB each.
This leads to 4 × 4 = 16 bases employed as the centers of the critic network.
Thus, the network has in total 16 hidden neurons [M = 16, see Eqs. (6) and (7)]which cover the state space of the system. The size or width of the Gaussian basis
functions was simply set to twice the distance between its center and the centerof its nearest neighbor. The reward signal [R, see Eq. (9)] was set to +1 when
the robot came close to the green object (desired object or goal) and −1 to theblue object (undesired object). In order to promote exploration, we used low-passﬁltered noise for low-frequency probing which was appropriate for the robot. We
also used the modulation scheme for controlling the exploration level where ξ, Vmax
and Vmin were here set to 5.0, 50 and 0, respectively. In addition to this scheme, theexploration term was exponentially reduced as soon as the performance improved(i.e., the robot frequently approached the goal). The control parameters α [seeEq. (5)], λ [see Eq. (8)], τ [see Eq. (9)], and ζ [see Eq. (10)] were set to 0.001, 0.7,
0.2, and 0.5, respectively. The weights [w1,2 , see Eq. (3)] were initially set to 0.0
and changed by Eq. (5).
We let the combinatorial learning mechanism learn to steer the robot toapproach the desired goal (i.e., the green object). Without control, the robot randomly moved around. During a run in each trial, the robot started at a speciﬁc
location [i.e., the black dot shown in Fig. 13(c)]. A run was terminated when the
robot approached one of the objects or hit a boundary as well as when simulation
time was above 15 s. After termination, the robot was reset to the same starting location with a random orientation in the interval [−45◦, 45◦ ]. We repeatedthis 50 experiments where each experiment was terminated after 200 trials. The
performance of combinatorial learning compared to ICO learning and continuous
actor–critic alone is shown in Fig. 14.
As can be seen, combinatorial learning had the highest success rate, continuous
actor–critic RL a lower one, and ICO learning the lowest. With respect to the number of learning trials, combinatorial learning and ICO learning were not signiﬁcantly
diﬀerent. However, they were substantially faster than continuous actor–critic RL.
Among these learning mechanisms, combinatorial learning was the best approach,showing highest success rate with the lowest number of learning trials.
To better understand why the combinatorial learning mechanism outperforms
its individual components in this task, we also plotted learning curves. Figure 151350015-23

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Success rate %100
80

Total learning trials at success

P. Manoonpong et al.

94 %
76 %

60
46 %
40
20
0

140

Error bars represent
the 95% confidence interval

120
100

120 trials

80
60

83 trials
73 trials

40
20

Combinatorial Continuous ICO learninglearning
actor-critic RL(a)0
Combinatorial Continuous ICO learninglearning
actor-critic RL(b)Fig. 14. Comparison of the performance of three learning mechanisms. (a) Success rate in a totalof 50 experiments. Here, “success” means that the robot can approach the green object from thestarting position with diﬀerent random orientations in the interval [−45◦ , 45◦ ] [see e.g., Fig. 13(c)].
(b) Average of the total learning trials at success.

exempliﬁes the learning curves showing the changes of the control parameters (i.e.,
synaptic weights) of ICO learning and continuous actor–critic RL in combinatorial
learning. The weights converged to ﬁxed values [see thick lines in Figs. 15(a)and 15(b)] resulting in a goal-directed behavior. The input and output signals
during this learning experiment and the behavior of the system after the weights
converged are shown in Fig. 16.
When only ICO learning was used, the weights sometimes converged to otherﬁxed values [see transparent lines in Fig. 15(a)] producing an undesired behavior;i.e., the robot moved toward the undesired blue object instead of the desired green
object. When only continuous actor–critic RL was used, the weights sometimes
changed a lot to negative values [see transparent lines in Fig. 15(b)]. As a consequence, the robot moved away from the objects. However, they will ﬁnally converge
but this will require a lot of learning trials, e.g., > 600 trials.
In principle, ICO learning can recognize a correlation only between its inputs(i.e., predictive and reﬂex signals, see e.g., Fig. 1) without recognizing a goal (i.e.,
reward or punishment). Thus, for this task it can only generate an anticipatory
reaction towards objects, rather than a goal-directed behavior. On the other hand,continuous actor–critic RL can achieve this in most cases but requires more learning
trials than ICO learning. By contrast, combinatorial learning allows ICO learningand continuous actor–critic RL to complement each other leading to control policy
improvement (high success rate and fast convergence [see Fig. 14]). This is because
continuous actor–critic RL tries to drive a robot toward a goal with a certain degreeof exploration. At the same time, ICO learning tries to limit the exploration area
(i.e., guiding) since it tries to drive the robot toward the point of interest (green
or blue object) deﬁned by a prior knowledge. Without ICO learning, due to the
exploration, the robot sometimes has diﬃculties to go back to the goal or it requires
1350015-24

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Weights (ICO learning)

Combining Correlation-Based and Reward-Based Learning

0.12
0.09
0.06
0.03

G

0.00

B

0

5

10

15

20

25

30 x104

Time [steps]

Weights (actor-critic RL)

(a)

W

0.10
0.05

G

W B

0.00
-0.05
-0.10
0

5

10

15

20

25

30 x104

Time [steps](b)
Fig. 15. (Color online) Learning curves of a goal-directed behavior. (a) Weight changes in ICO
learning. (b) Weight changes in continuous actor–critic RL. Thick lines present the weight changes
in each learning mechanism in the combinatorial learning framework while transparent lines show
the weight changes when only ICO learning or continuous actor–critic RL was used. Using combinatorial learning, the weights became ﬁnally stable after around 75,000 time steps (or 50 trials)
meaning that the robot can successfully approach the goal. In contrast, the weights sometimes
converged to other ﬁxed values in the case of ICO learning alone [see (a)] producing an undesired
behavior (i.e., the robot went to a blue object) while they sometimes changed a lot in the caseof continuous actor–critic RL alone [see (b)]. Note that here ρφG = ρ1 , ρφB = ρ2 , wφG = w1 ,w φB = w 2 .

more trials. In addition ICO learning can also shape the learning process such thatthe robot can approach the goal on a short path (see below).
To see the guiding and shaping eﬀects from ICO learning, we took the developed weights of ICO learning and continuous actor–critic RL before convergenceoccurred to control the robot and observed its behavior. Then, we compared the
behavior to the one controlled by only the developed weights of continuous actor–
critic RL, i.e., we set the weights of ICO learning to 0.0 while the weights of continuous actor–critic RL remained unchanged. Interestingly, we found three diﬀerent
behaviors (I, II, III, see Figs. 17–19). Recall that in these goal-directed behavior experiments, the two relative orientations (φG,B ) overlap with each other, i.e.,
1350015-25

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

[deg]

180
90

G

0
-90

-180

90
0

B

[deg]

180

-90

-180

Output

1.0
0.5
0.0
-0.5
-1.0
0

5

10

15

20

25

4
30 x10Time [steps](a)(b)
Fig. 16. (Color online) (a) States of the mobile robot system (φG,B ) and the output (OCOM )during learning. Learning curves belonging to these signals are shown in Fig. 15 (see thick lines).
(b) Robot trajectories observed from around 28 × 104 to 30 × 104 time steps [see gray area in(a)]. Positive and negative values of the output means turning right and left, respectively. At
the beginning the robot explored a lot (i.e., large amplitude of the output signal). After learning
converged, the robot did not turn much (i.e., small amplitude of the output signal). It only turned
if it deviated from the goal. As a result, it always approached the goal (green object). In otherwords, the learned control policy drove the robot toward the goal and kept it away from the blue
object; thereby, φB shows most of the time negative values above 90 deg (i.e., heading away fromthe blue object) while φG shows most of the time positive values around 90 deg (i.e., turning
towards the goal).

1350015-26

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

the robot simultaneously sensed its relative orientations to the locations of bothobjects in the whole area. In addition, for continuous actor–critic RL a positive
reward (+1) was given when the robot got into the circle around the green object
while a negative reward (−1) was given when the robot got into the circle aroundthe blue object [see Fig. 13(c)]. Thus, during learning as long as the exploration
term and the TD error existed, the weights of both signals simultaneously changed,
no matter where the robot was.
Figure 17 shows the ﬁrst behavior I where we took the developed weights ataround 3 × 104 time steps [see dashed line in Figs. 17(a) and 17(b)], slightly before
the weights of continuous actor–critic RL became stable, to test the robot. It can beseen that the robot always moved toward the desired green object [see Fig. 17(c)]when the developed weights of ICO learning and continuous actor–critic RL wereused. On the other hand, it sometimes moved to the undesired blue object or wentstraight when only the developed weights of continuous actor–critic RL were used[see Fig. 17(d)] because of exploration as well as a large weight wφB . Note that ρφB

Weights (ICO learning)

0.30
0.25

G

0.20
0.15
0.10
0.05
B

0.00
0

10

20
30
Time [steps]

4
50 x1040

Weights (actor-critic RL)

0.15
W
0.10

W

B

G

0.05
0.00

-0.05
-0.10

0

10

20
30
Time [steps]

404
50 x10Fig. 17. (Color online) Learning curves and robot behaviors I. (a), (b) Weight changes of ICO
learning and continuous actor–critic RL in the combinatorial learning framework. Dashed line
shows the point (i.e., around 3 × 104 time steps or 300 s) where the weights of ICO leaning
and continuous actor–critic RL were used to test the robot. (c) Trajectories of the robot from thestarting position with diﬀerent random orientations in the interval [−45◦ , 45◦ ]. (d) The trajectories
when ICO learning control policy was switched oﬀ, i.e., we set the weights of ICO learning to 0.0
while the weights of continuous actor–critic RL remained unchanged. Note that during the test
we removed the exploration term from the controller in order to clearly see the trajectories.
1350015-27

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

did not become large since when the robot approached the blue object it did notdeviate much from the object. Thus, the positive derivative of the reﬂex signal was
smaller than threshold, thereby ρφB remained unchanged. This experimental resultshows that ICO learning complemented continuous actor–critic RL leading to goaldirected behavior. In other words, ICO learning guided a learning strategy enabling
continuous actor–critic RL to exploit more the positive reward. As a consequence,
convergence ﬁnally occurred.
Figure 18 shows the second behavior II where we took the developed weights ataround 20 × 104 time steps [see dashed line in Figs. 18(a) and 18(b)], slightly before
the weights of continuous actor–critic RL reversed their growing directions, to test
the robot. At this point, it can be seen that when the developed weights of ICO
learning and continuous actor–critic RL were used the robot always approached theundesired blue object [see Fig. 18(c)] where the negative reward (−1) was given.
Thus continuous actor–critic RL could use this reward signal to correct its current control policy. This eﬀect can be observed from the weights of continuous

0.30
0.25

G

0.20
0.15

B

0.10
0.05
0.00
0

10

20

30
40
Time [steps]

4
60 x10

50

0.4

W
G

0.2

W

B

0.0
-0.2
-0.4
0

10

20

30
40
Time [steps]

50

4
60 x10Fig. 18. (Color online) Learning curves and robot behaviors II. (a), (b) Weight changes of ICO
learning and continuous actor–critic RL in the combinatorial learning framework. Dashed line
shows the point (i.e., around 20 × 104 time steps or 2000 s) where the weights of ICO leaning
and continuous actor–critic RL were used to test the robot. (c) Trajectories of the robot from thestarting position with diﬀerent random orientations in the interval [−45◦ , 45◦ ]. (d) The trajectories
when the ICO learning control policy was switched oﬀ, i.e., we set the weights of ICO learning
to 0.0 while the weights of continuous actor–critic RL remained unchanged. Note that during thetest we removed the exploration term from the controller in order to clearly see the trajectories.
1350015-28

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

actor–critic RL which signiﬁcantly reversed their growing directions at around
20 × 104 time steps or 2000 s. On the other hand, when only the developed weights
of continuous actor–critic RL were used, the robot always moved away from theobjects [see Fig. 18(d)]. Therefore, in this situation continuous actor–critic RL had
diﬃculty to obtain any reward signal to correct its current control policy. Due to thestochastic process employed, which tried to search for a successful control policy,
the weights might change to a large degree [see transparent lines in Fig. 15(b)]. As aresult, continuous actor–critic RL might fail to solve the task in a given number oftrials (here, maximal 200 trials). This result suggests that ICO learning shaped or
guided the learning strategy of continuous actor–critic RL such that it can receivea reward (i.e., here a negative one). Then it used this reward to correct the current
control policy. As a consequence, convergence ﬁnally occurred.
Figure 19 shows the third behavior III where we took the developed weights
at two states to test the robot. The early state was around 75 × 103 time steps
where only the weights of continuous actor–critic RL became stable [see dashedline State I in Figs. 19(a) and 19(b)]. They were stable since the exploration termState I

Weights (ICO learning)

0.25

State II
State I

G

0.20
0.15
0.10
0.05

B

0.00
0

10

20

4
50 x103040Time [steps]

Weights (actor-critic RL)

0.3
0.2

W

0.1

W

State II
G

B

0.0
-0.1
-0.2
-0.3

0

10

20

30
40
Time [steps]

4
50 x10Fig. 19. (Color online) Learning curves and robot behaviors III. (a), (b) Weight changes of ICO
learning and continuous actor–critic RL in the combinatorial learning framework. Dashed lines
show two states where the weights of ICO leaning and continuous actor–critic RL were used totest the robot. State I was around 75 × 103 time steps or 750 s and state II was around 50 × 104 or
5000 s. (c) Trajectories of the robot from the starting position with diﬀerent random orientationsin the interval [−45◦ , 45◦ ] at state I. (d) The trajectories at state II. Note that during the test
we removed the exploration term from the controller in order to clearly see the trajectories.
1350015-29

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

was zero. Recall that the exploration term was exponentially reduced as soon as theperformance improved (i.e., the robot frequently approached the goal). The later
state was around 50 × 104 time steps where the weights of ICO leaning became alsostable [see dashed line State II in Figs. 19(a) and 19(b)]. It can be seen that the
robot moved toward the goal in long trajectories [see Fig. 19(c)] when the control
policy at the early state was used. In contrast, it moved on shorter trajectories
when the control policy at the later state was used [see Fig. 19(d)]. This suggeststhat although continuous actor–critic RL was stopped due to the inhibition of its
exploration term, ICO learning still shaped the control policy. This is because thereﬂex signal was not completely avoided since the robot still had large deviations to
the goal when it came close to it. As a consequence, ICO learning improved robot
performance by making it head directly to the goal, thereby leading to shorter
trajectories.
It is important to note that although the resulting weights of the experiments
shown in Figs. 15 and 17–19 converged to diﬀerent values, they generally conρ+wverged to almost the same weight ratio ( ρφφG + wφφG ) of 2.9 ± 0.6. This shows that inBB

combinatorial learning the combination of the weights of these two modules is necessary to successfully solve the task. Using all learned weights even yields a better
result (i.e., the robot moved on the shortest trajectories) compared to using only
the learned weights of either the ICO learning module or the continuous actor–
critic RL module (see Fig. 20). In addition, the weight ratio also suggests thatthe positive reward attracts the system approximately three times larger than thenegative reward repulses it.

(a)

(b)

(c)

Fig. 20. (Color online) Robot behaviors at three diﬀerent control parameter setups of combinatorial learning. For our investigation here, we used the learned weights from the experimentshown in Fig. 19, i.e., the weights at State II. (a) All learned weights were used, i.e., ρφG ≈ 0.245,
ρφB ≈ 0.036, wφG ≈ 0.224, wφB ≈ 0.117. (b) Only learned weights of the ICO learning module
were used while the weights of the continuous actor–critic RL modules were set to 0.0. (c) Only
learned weights of the continuous actor–critic RL module were used while the weights of the ICO
learning module were set to 0.0. Note that in each test the robot started from the starting position with diﬀerent random orientations in the interval [−45◦ , 45◦ ] and we removed the exploration
term from the controller in order to clearly see the trajectories.
1350015-30

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

6. Conclusion
In the following, we will discuss some remaining issues while other relevant discussion points have been treated alongside the experimental section above.
In this study, we introduced a neural combinatorial learning model for policy
improvement. The learning model combines ICO learning and continuous actor–
critic RL in a parallel manner where the ICO learning output and the continuous
actor–critic RL output are equally weighted to control the agent. The equal contribution used here is a simple and straightforward strategy for combining them [seeEq. (10)]. In general, ICO learning alone can quickly learn to solve tasks but has
limitations for more diﬃcult tasks (i.e., here, balancing the pole in critical initialconditions as well as goal-directed behavior). On the other hand, pure continuous
actor–critic RL can often solve the tasks but learns slowly.
Mainly we found that the performance of the controller can be strongly improved
when the combinatorial learning model was applied. To make this model work properly we need to design a proper reﬂex for ICO learning as well as an appropriate
correlation between predictive and reﬂex signals. For example, in the pole balancing task we conﬁgured ICO learning such that the weights were changed only forthe positive derivatives of the reﬂex signal, otherwise they remained unchanged.
This is to avoid a negative correlation resulting in poor learning performance oreven failure. Another condition which would make the current form of the model
problematic is a strong conﬂict between the reward function and the reﬂex. Some
problematic cases would be:(i) The ﬁrst case is if continuous actor–critic RL moves the agent toward a target
due to the reward function, while ICO learning tries to move it away from it
due to the built-in reﬂex.
(ii) Another case is robot navigation in an environment with obstacles when ICO
learning is used to generate a negative tropism behavior (e.g., avoiding obstacles) while continuous actor–critic RL is used to generate a positive tropism
behavior (e.g., approaching a goal). In this scenario, a conﬂict will occur when
the goal is behind an obstacle or directly close to it.
(iii) The last case is a dynamic motion control task like balancing a humanoid
robot (many degrees of freedom system) against an external disturbance (pushing) where ICO learning controls the robot to avoid pushing (e.g., leaning
action) while continuous actor–critic RL wants to keep it balance (e.g., uprightposition).
However, these complex tasks might also be solved but this will require a modiﬁcation of the model or an improvement by:(i) Properly designing the reward function of continuous actor–critic RL and thereﬂex of ICO learning,(ii) Using more appropriate sensory signals or predictive signals,
1350015-31

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

(iii) Transforming the low-dimensional input of the actor part into a higherdimensional one by using nonlinear functions (e.g., an RBF network [46]) or
using a decoder [7],(iv) Using an adaptive critic network or another type of a critic network (e.g., aself-adaptive reservoir computing network [14] having high-dimensional nonlinear dynamics and internal memory) for a better approximation of the valuefunction. These issues (i–iv) are still under investigation and go beyond thescope of this paper.
Besides our approach, there are a number of investigations on combining conventional RL with other (learning) mechanisms or applying other adaptive methods
to it in order to enhance learning capability, reduce learning time, or counteract thecurse of dimensionality. For example, Price and Boutilier [54] developed an imitation
model called “implicit imitation” and integrated it into RL. It basically combines its
own experience with its observations of the behavior of an expert mentor for learning. Doya [18] proposed hybrid RL based on the “actor-tutor” framework, whichuses a model of the system dynamics as the tutor part. There, the actor (or controller) is trained by supervised learning to minimize the diﬀerence between its output and the tutor’s output (desired output). This framework, basically resembling“feedback error learning” [23], was applied to nonlinear control tasks. Centina [12]introduced a supervised reinforcement learning (SRL) architecture for robot control
problems with high-dimensional state spaces. There, a behavior model learned fromexamples is used to dynamically reduce the set of actions available from each state
during the early RL process. In addition to these, other eﬀorts have been made by
developing advanced RL techniques [21, 31, 58] like hierarchical RL [3, 6, 9, 17, 66],
by employing adaptive state representation [28, 34, 57], by using diﬀerent exploration/exploitation techniques [4, 44, 60, 64], and by introducing algorithms forshaping rewards [2, 16, 48]. While all these advanced methods can successfully solve
several (robot) tasks and are eﬀective in their own right, they are quite diﬃcult tomatch to biological neural learning and conditioning paradigms.
Only a few works have developed diﬀerent types of learning models for robot
control where the models mimic principles of these learning or conditioning mechanisms of animal leaning [1, 13, 65]. Alonso et al. [1] introduced the associative
learning based approach (called the Pavlovian and Instrumental Q-learning framework) to deal with generalization in Q-learning. This approach improves RL (i.e.,
Q-learning) by applying the Rescorla-Wagner model [56] as a part of the control
scheme for stimulus-stimulus associations. Its performance was tested in a grid simulator where agents have to approach or avoid appetitive and aversive stimuli. Chang
and Gaudiano [13] presented a neural network based on operant and classical conditioning. It was tested on mobile robots. As a consequence, it allows the robots tosimultaneously learn to approach light sources and avoid obstacles. Touretzky and
Saksida [65] developed a model of operant conditioning that incorporates aspectsof chaining in which behavioral routines are built up from smaller action segments.
1350015-32

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning

The model was implemented on a mobile robot for solving the delay match to sample
task (i.e., the task that involves behavioral sequences). Although all these learningmodels [1, 13, 65] employed learning and conditioning aspects of animal learning,they did not show or provided an understanding of how diﬀerent mechanisms interact or complement each other, resulting in successful control policies. Instead, they
were developed for improving conventional RL models or solving particular robotictasks (i.e., goal-directed behavior control). Therefore, it is still unclear whether
these models can also deal with qualitatively diﬀerent tasks, like dynamic motion
control.
Compared to many of these approaches just summarized, our combinatorial
learning model applied the principles of classical and operant conditioning of animal learning. It was developed using ICO learning (a simpliﬁed model of classicalconditioning) and continuous actor–critic RL (a simpliﬁed model of operant conditioning). They were implemented based on artiﬁcial neural networks; thereby,
making them conceptually closer to biological systems compared to any other solution. Furthermore, ICO learning and continuous actor–critic RL are partially related
to neural learning mechanisms in the brain. Speciﬁcally, ICO learning implements
plain heterosynaptic plasticity associated with modulatory processes found in thebrain [27] and continuous actor–critic RL uses TD learning which is related to
dopaminergic responses in the brain. Especially, some cells in the substantia nigra
and ventral tegmental area (VTA) show a behavior similar to representing the errorof TD-learning [61], which we use for weight adaptation in actor and critic networks.
We demonstrated the capability of this model in solving two diﬀerent tasks: Polebalancing and goal-directed behavior control. This shows that the learning model
is not limited to a speciﬁc task.
In addition to this, our model can be considered as a model-free method since its
learning rules do not require a system or environment model. Instead, ICO learning
requires only a built-in reﬂex as a self-supervised mechanism to quickly ﬁnd thecorrelations between a state and an unwanted condition (i.e., reﬂex action), while
continuous actor–critic RL uses its prediction mechanism including its own experiences and some exploration to obtain a good control policy. Although our models
use a ﬁxed state representation in the critic, one could also extend the critic toadaptive state partitioning [45] since the actor and critic are independently constructed. Our work also shares a connection to Kolter and Ng [33] where theypresented a policy gradient method called the “Signed Derivative” approximation.
The general concept of this approach is similar to our model in the sense that it is amodel-free method which uses intuition to guess the direction where control inputs
aﬀect future state variables. This intuition is used to construct the signed derivative approximation which is directly applied to the update rule of RL. Generally
speaking, the signed derivative approximation can be viewed as an instance of thebuilt-in reﬂex of ICO learning. However, in our model the built-in reﬂex indirectly
aﬀects RL through ICO learning since it is used to guide and shape learning.

1350015-33

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

In summary, the study pursued here sharpens our understanding of how diﬀerent
learning mechanisms (i.e., correlation-based learning and RL) can be appropriately
combined and how they complement each other leading to policy improvement.
This study also suggests that correlation-based learning, on the one hand, can beused to speed up the learning process of RL. On the other hand, it can shape andcorrectly guide RL for searching an optimal policy. While the proposed combination of these two learning mechanisms can improve the performance of the systems,
they are still combined in a simple way [see Eq. (10)]. Thus, for future work, wewill investigate adaptive combinations. One possible option is to employ a learning
mechanism based on a correlation between a direct reward signal and the outputs of
ICO learning and continuous actor–critic RL for adapting their output weights. Thisway, an active output will have a high correlation with the reward signal, thereby
strengthening its weight. The output weights will ﬁnally determine the behaviorof the agent. Another option is to use a hierarchical RL framework [46] to ﬁnd an
optimal combination. Furthermore, we will also apply the combinatorial learning
mechanism to more complex tasks, like the double-pendulum scenario [22, 29, 30],
including ones with high dimensional states and actions (e.g., helicopter control [36]and octopus arm problems [69]). We also aim to use it as online learning for realrobotic tasks, e.g., adaptive walking of hexapod robots [41], dynamic motion control of biped robots, and real robot navigation in complex environments. However,solving such tasks may require a modiﬁcation of some components, e.g., using anonlinear actor and/or an adaptive critic network, which could be easily done dueto the modularity of the framework.

Acknowledgments
This research was supported by Emmy Noether grant MA4464/3-1 of theDeutsche Forschungsgemeinschaft (DFG), Bernstein Center for Computational
Neuroscience II Göttingen (BCCN grant 01GQ1005A, project D1), Japan Society
for the Promotion of Science (JSPS), European Communitys Seventh Framework
Programme FP7/2007-2013 (Theme 3, Information and Communication Technologies) under grant agreement 270273, Xperience, and a part of this research was
supported by “Brain Machine Interface Development” SRPBS, MEXT, MEXT
KAKENHI 23120004 and Strategic International Cooperative Program, JST. P.M.
would like to thank NICT for its support within the JAPAN TRUST International
Research Cooperation Program. We thank Tomas Kulvicius for critical discussions
and Martin Biehl and Frank Hesse for technical advice.

References[1] Alonso, E., Mondragon, E. and Kjäll-Ohlsson, N., Pavlovian and instrumental
Q-learning: A Rescorla and Wagner-based approach to generalization in Q-learning,in Proc. Adaptation in Artificial and Biological Systems (2006), pp. 23–29.
1350015-34

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning[2] Asmuth, J., Littman, M. L. and Zinkov, R., Potential-based shaping in modelbased reinforcement learning, in Proc. 23rd National Conf. Artificial Intelligence(AAAI’08 ) (2008), pp. 604–609.
[3] Bakker, B. and Schmidhuber, J., Hierarchical reinforcement learning based on subgoal
discovery and subpolicy specialization, in Proc. 8th Conf. Intelligent Autonomous
Systems (2004), pp. 438–445.
[4] Banerjee, B. and Kraemer, L., Action discovery for single and multi-agent reinforcement learning, Adv. Complex Syst. 14 (2011) 279–305.
[5] Barnard, C., Animal Behavior : Mechanism, Development, Function, and Evolution(Pearson Education, 2004).
[6] Barto, A. G. and Mahadevan, S., Recent advances in hierarchical reinforcement learning, Discrete Event Dyn. Syst. 13 (2003) 41–77.
[7] Barto, A. G., Sutton, R. S. and Anderson, C. W., Neuron-like adaptive elementsthat can solve diﬃcult learning control problems, IEEE Trans. Syst. Man, Cybern.
13 (1983) 834–846.
[8] Bekey, G., Autonomous Robots From Biological Inspiration to Implementation andControl (MIT Press, Cambridge, 2005).
[9] Botvinick, M. M., Niv, Y. and Barto, A. C., Hierarchically organized behavior andits neural foundations: A reinforcement learning perspective, Cognition 113 (2009)262–280.
[10] Bovet, S., Robots with self-developing brains, Ph.D. thesis, University of Zurich(2007).
[11] Brembs, B. and Heisenberg, M., The operant and the classical in conditioned orientation in drosophila melanogaster at the ﬂight simulator, Learn. Memory 7 (2000)
104–115.
[12] Cetina, V. U., Supervised reinforcement learning using behavior models, in Proc.
Sixth Int. Conf. Machine Learning and Applications (ICMLA 2007 ) (2007), pp. 336–
341.
[13] Chang, C. and Gaudiano, P., Application of biological learning theories to mobile
robot avoidance and approach behaviors, Adv. Complex Syst. 1 (1998) 79–114.
[14] Dasgupta, S., Wörgötter, F. and Manoonpong, P., Information theoretic selforganised adaptation in reservoirs for temporal memory tasks, in Proc. 13th Int.
Conf. Engineering Applications of Neural Networks (EANN 2012 ) (2012), pp. 31–40.
[15] Dayan, P. and Balleine, B., Reward, motivation, and reinforcement learning, Neuron
36 (2002) 285–298.
[16] Devlin, S., Kudenko, D. and Grzes, M., An empirical study of potential-based reward
shaping and advice in complex, multi-agent systems, Adv. Complex Syst. 14 (2011)
251–278.
[17] Dietterich, T. G., Hierarchical reinforcement learning with the MAXQ value function
decomposition, J. Artif. Intell. Res. 13 (2000) 227–303.
[18] Doya, K., Eﬃcient nonlinear control with actor-tutor architecture, in Advances in
Neural Information Processing Systems (1997), pp. 1012–1018.
[19] Doya, K., Reinforcement learning in continuous time and space, Neural Comput. 12(2000) 219–245.
[20] Endo, G., Morimoto, J., Matsubara, T., Nakanish, J. and Cheng, G., Learning CPGbased biped locomotion with a policy gradient method: Application to a humanoid
robot, Int. J. Robot. Res. 27 (2008) 213–228.
[21] Fischer, J., A Modulatory Learning Rule for Neural Learning and Metalearning inReal World Robots with Many Degees of Freedom (Shaker Verlag GmbH, 2003).

1350015-35

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

[22] Gomez, F., Schmidhuber, J. and Miikkulainen., R., Accelerated neural evolution
through cooperatively coevolved synapses, J. Mach. Lear. Res. 9 (2008) 937–965.
[23] Gomi, H. and Kawato, M., Neural network control for a closed-loop system using
feedback-error-learning, Neural Netw. 6 (1993) 933–946.
[24] Gullapalli, V., A stochastic reinforcement learning algorithm for learning real-valued
functions, Neural Netw. 3 (1990) 671–692.
[25] Howery, L. D., Why do animals behave the way they do?, Backyards and Beyond :Rural Living in Arizona 3 (2007) 17–18.
[26] Hull, C. L., A Behavior System: An Introduction to Behavior Theory Concerning theIndividual Organism (Yale University Press, New Haven, CT, 1952).
[27] Humeau, Y., Shaban, H., Bissiere, S. and Lüthi, A., Presynaptic induction ofheterosynaptic associative plasticity in the mammalian brain, Nature 426 (2003)
841–845.
[28] Iida, S., Kuwayama, K., Kanoh, M., Kato, S. and Itoh, H., A dynamic allocation
method of basis functions in reinforcement learning, in Proc. 17th Australian Joint
Conf. Artificial Intelligence (2004), pp. 71–73.
[29] Kassahun, Y., de Gea, J., Edgington, M., Metzen, J. H. and Kirchner, F., Accelerating neuroevolutionary methods using a kalman ﬁlter, in Proc. 10th Genetic andEvolutionary Computation Conf. (GECCO-2008 ) (2008), pp. 1397–1404.
[30] Kassahun, Y., Wöhrle, H., Fabisch, A. and Tabie, M., Learning parameters of linear models in compressed parameter space, in Proc. Artificial Neural Networks andMachine Learning (ICANN2012 ) (2012), pp. 108–115.
[31] Kawarai, N. and Kobayashi, Y., Learning of whole arm manipulation with constraintof contact mode maintaining, J. Robot. Mechatron. 22 (2010) 542–550.
[32] Klopf, A. H., A neuronal model of classical conditioning, Psychobiology 16 (1988)
85–123.
[33] Kolter, J. and Ng, A., Policy search via the signed derivative, in Proc. Robotics:
Science and Systems (RSS ) (2009), pp. 27, Online.
[34] Kondo, T. and Ito, K., A reinforcement learning with adaptive state space recruitment strategy for real autonomous mobile robots, in Proc. IEEE/RSJ Int. Conf.
Intelligent Robots and Systems (2002), pp. 897–902.
[35] Konorski, J., Integrative Activity of the Brain (University of Chicago Press, Chicago,1967).
[36] Koppejan, R. and Whiteson, S., Neuroevolutionary reinforcement learning for generalized helicopter control, in GECCO 2009 : Proc. Genetic and Evolutionary Computation Conf. (2009), pp. 145–152.
[37] Kosco, B., Diﬀerential hebbian learning, in Proc. Neural Networks for Computing:
AIP, Vol. 151 (1986), pp. 277–282.
[38] Lee, H., Shen, Y., Yu, C., Singh, G. and Ng, A., Quadruped robot obstacle negotiation
via reinforcement learning, in Proc. IEEE Int. Conf. Robotics and Automation (2006),
pp. 3003–3010.
[39] Lovibond, P. F., Facilitation of instrumental behavior by a pavlovian appetitive conditioned stimulus, J. Exp. Psychol. Anim. B 9 (1983) 225–247.
[40] Manoonpong, P., Geng, T., Kulvicius, T., Porr, B. and Wörgötter, F., Adaptive, fast
walking in a biped robot under neuronal control and learning, PLoS Comput. Biol.
3 (2007) e134.
[41] Manoonpong, P., Parlitz, U. and Wörgötter, F., Neural control and adaptive neuralforward models for insect-like, energy-eﬃcient, and adaptable locomotion of walking
machines, Front. Neural Circuits 7 (2013). Doi: 10.3389/fncir.2013.00012.

1350015-36

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

Combining Correlation-Based and Reward-Based Learning[42] Manoonpong, P. and Wörgötter, F., Adaptive sensor-driven neural control for learning in walking machines, in Neural Information Processing, LNCS (2009), pp. 47–55.
[43] Manoonpong, P., Wörgötter, F. and Morimoto, J., Extraction of reward-related feature space using correlation-based and reward-based learning methods, in Neural
Information Processing, LNCS (2010), pp. 414–421.
[44] Morihiro, K., Matsui, N., and Nishimura, H., Eﬀects of chaotic exploration on reinforcement maze learning, in Knowledge Based Intelligent Information and Engineering Systems (2004), pp. 833–839.
[45] Morimoto, J. and Doya, K., Reinforcement learning of dynamic motor sequence:
Learning to stand up, in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems(1998), pp. 1721–1726.
[46] Morimoto, J. and Doya, K., Acquisition of stand-up behavior by a real robot using
hierarchical reinforcement learning, Robot. Auton. Syst. 36 (2001) 37–51.
[47] Mowrer, O., Learning Theory and Behavior (New York, Wiley, 1960).
[48] Ng, A. Y., Harada, D. and Russell, S. J., Policy invariance under reward transformations: Theory and application to reward shaping, in Proc. 16th Int. Conf. Machine
Learning (1999), pp. 278–287.
[49] Pasemann, F., Evolving neurocontrollers for balancing an inverted pendulum,
Network-Comp. Neural 9 (1998) 495–511.
[50] Pavlov, I., Conditioned Reflexes (Oxford University Press, Oxford, UK, 1927).
[51] Phon-Amnuaisuk, S., Learning cooperative behaviours in multiagent reinforcement
learning, in Neural Information Processing, LNCS (2009), pp. 570–579.
[52] Porr, B. and Wörgötter, F., Strongly improved stability and faster convergence oftemporal sequence learning by using input correlations only, Neural Comput. 18(2006) 1380–1412.
[53] Porr, B. and Wörgötter, F., Fast heterosynaptic learning in a robot food retrieval
task inspired by the limbic system, Biosystems 89 (2007) 294–299.
[54] Price, B. and Boutilier, C., Accelerating reinforcement learning through implicit imitation, J. Artif. Intell. Res. 19 (2003) 569–629.
[55] Rescorla, R. and Solomon, R., Two process learning theory: Relationship between
pavlovian conditioning and instrumental learning, Psychol. Rev. 88 (1967) 151–182.
[56] Rescorla, R. and Wagner, A., A theory of pavlovian conditioning: Variations in theeﬀectiveness of reinforcement and nonreinforcement, in Classical Conditioning II :Current Research and Theory (1972) 64–99.
[57] Sherstov, A. and Stone, P., Function approximation via tile coding: Automating
parameter choice, in Proc. Symp. Abstraction, Reformulation, and Approximation(SARA-05 ) (2005), pp. 194-205.
[58] Shibata, K., Nishino, T. and Okabe, Y., Active perception and recoginition learning
system based on Actor-Q architecture, Syst. Comput. Jpn. 33 (2002) 12–22.
[59] Skinner, B., The Behavior of Organisms: An Experimental Analysis (Appleton Century Croft, New York, 1938).
[60] Smith, S. C. and Herrmann, J. M., Homeokinetic reinforcement learning, in First
IAPR TC3 Workshop, PSL 2011, LNCS (2012), pp. 82–91.
[61] Suri, R. E., Bargas, J. and Arbib, M. A., Modeling functions of striatal dopamine
modulation in learning and planning, Neuroscience 103 (2001) 65–85.
[62] Sutton, R. and Barto, A., Reinforcement Learning: An Introduction (MIT Press,
Cambridge, 1998).
[63] Thorndike, E., Animal intelligence: An experimental study of the associative process
in animals, Psychol. Rev. Monogr. Suppl. 8 (1898) 68–72.

1350015-37

June 24, 2013 16:47 WSPC/S0219-5259 169-ACS 1350015

P. Manoonpong et al.

[64] Tokic, M., Adaptive -greedy exploration in reinforcement learning based on value
diﬀerences, in Proc. KI 2010 : Advances in Artificial Intelligence (2010), pp. 203–210.
[65] Touretzky, D. and Saksida, L., Operant conditioning in skinnerbots, Adapt. Behav.
5 (1997) 219–247.
[66] van Dijk, S. G. and Polani, D., Grounding subgoals in information transitions, in
Proc. IEEE Symp. Adaptive Dynamic Programming and Reinforcement Learning(Paris, France, 2011), pp. 105–111.
[67] Watkins, C. J. C. H., Learning from delayed rewards, Ph.D. thesis, University ofCambridge (1989).
[68] Williams, D. and Williams, H., Auto — maintenance in the pigeon: Sustained pecking
despite contingent non-reinforcement, J. Exp. Anal. Behav. 12 (1969) 511–520.
[69] Woolley, B. G. and Stanley, K. O., Evolving a single scalable controller for an octopus
arm with a variable number of segments, in Parallel Problem Solving from Nature,
Lecture Notes in Computer Science (2010), pp. 270–279.
[70] Wörgötter, F. and Porr, B., Temporal sequence learning, prediction and control —A review of diﬀerent models and their relation to biological mechanisms, NeuralComp. 17 (2005) 245–319.

1350015-38