Q-Learning Using SDNN for Noisy and Redundant Inputs

Paper:

Q-Learning in Continuous State-Action Spacewith Noisy and Redundant Inputs
by Using a Selective Desensitization Neural Network
Takaaki Kobayashi, Takeshi Shibuya, and Masahiko Morita
Faculty of Engineering, Information and Systems, University of Tsukuba
1-1-1 Tennodai, Tsukuba, Ibaraki 305-8573 Japan
E-mail: {takaaki@bcl.esys., shibuya@iit., mor@bcl.esys.}tsukuba.ac.jp[Received May 21, 2015; accepted August 18, 2015]

When applying reinforcement learning (RL) algorithms such as Q-learning to real-world applications,
we must consider the influence of sensor noise. The
simplest way to reduce such noise influence is to additionally use other types of sensors, but this may require more state space – and probably increase redundancy. Conventional value-function approximators used to RL in continuous state-action space do
not deal appropriately with such situations. The selective desensitization neural network (SDNN) has highgeneralization ability and robustness against noise andredundant input. We therefore propose an SDNNbased value-function approximator for Q-learning incontinuous state-action space, and evaluate its performance in terms of robustness against redundant input
and sensor noise. Results show that our proposal isstrongly robust against noise and redundant input andenables the agent to take better actions by using additional inputs without degrading learning efficiency.
These properties are eminently advantageous in realworld applications such as in robotic systems.

Keywords: reinforcement learning, function approximator, continuous state-action, sensor noise, redundant inputs

1. Introduction
Reinforcement learning (RL) [1] is a framework fortrial-and-error-based interactive learning inspired by behavioral learning in animals. An agent working in thisframework learns behavior for acquiring maximum rewards based on its experience in the environment. RL enables the agent to accomplish a task by simply rewarding
the agent, thus acting as a promising control for systems
such as robots in real-world environments.
In Q-learning [2], which is a widely used RL algorithm,
the agent decides an action based on two functions – action value and policy – in which updating the action-valuefunction improves the agent’s behavior. With its provenconvergence property [3] and its easy implementation, QVol.19 No.6, 2015

learning is used frequently in applied research. When
Q-learning is applied in an environment with continuous
state-action space, the action-value function is generally
approximated by using a function approximator [1, 4–9].
In real-world applications, Q-learning agents sufferfrom input uncertainty due to sensor noise. The simplest way to solve this problem would be to additionallyuse other types of sensors so that the noises from different sensors can be mutually cancelled. This, however,
could enlarge state space and probably increase redundancy. Agents with conventional value-function approximators thus require more samples for their learning.
For this reason, a Q-learning function approximatorshould have the following properties when used in realworld applications:• High robustness against input uncertainty• High learning ability and robustness against redun-dant input
A recent study has demonstrated the superiority of selective desensitization neural networks (SDNNs) in function approximation [10]. Using an SDNN, Horie etal. [11] estimated hand motion speed from surface electromyograms, which are noisy and highly redundant input. Their results suggest that the SDNN is very suitablefor function approximation involving noisy and redundant
input. This makes the SDNN a strong candidate for approximating the action-value function in continuous stateaction space with noisy and redundant state variables.
We propose an SDNN-based value-function approximator for Q-learning in continuous state-action space [12]and evaluate its performance in terms of robustness
against redundant input [12] and sensor noise through aseries of numerical experiments. These experiments usean acrobot swing-up task containing noisy and redundant
input.
This paper is organized as follows: Section 2 explains
the basic RL framework and the Q-learning algorithm,
together with widely used ways for approximating thevalue-function. Section 3 proposes SDNN-C, a novel
function approximation. Section 4 reviews the results ofsimulation experiments and Section 5 lists conclusions.

Journal of Advanced Computational Intelligenceand Intelligent Informatics

825

Kobayashi, T., Shibuya, T., and Morita, M.

2. Background2.1. Reinforcement Learning
RL is a trial-and-error-based interactive learning framework in which agents adjust their behavior to maximize
their rewards in the environment.
At discrete time step t, an agent observes its own state,
st , decides and takes action at , and obtains reward rt+1 .
The agent learns suitable actions through this experience.
Learning and deciding actions are implemented by using
two functions—action value and policy.
Action-value function Q(s, a) is defined as the expected
total rewards from taking action a in state s. For a particular state-action pair, an action value is the value of theaction-value function and indicates the desirability of theaction. In RL, the agent updates its action-value function
by using a learning algorithm such as Q-learning.
Policy provides probabilities for selecting possible actions in certain states. A simple, well-known policy is the
ε -greedy policy, which encourages both exploration andexploitation. This policy frequently leads to selecting theaction with the highest action value, but randomly withlow probability ε .

2.2. Q-Learning
Q-learning [2, 3], a widely used RL algorithm, updates
action-value function Q(st , at ) asQ(st , at ) ← (1 − α )Q(st , at )
+α rt+1 + γ maxa ∈A (st+1)Q(st+1 , a ) ,(1)where A (s) is a set of possible actions in state s. α ∈ [0, 1]
is a learning rate and γ ∈ [0, 1) is a discount rate.
Because Q(s, a) is stored as a look-up table, conventional Q-learning cannot operate in continuous stateaction space.

2.3. Function Approximators
Action-values in continuous state-action space are often implemented by using a function approximator, which
represents the action-value by fewer parameters than alook-up table.
The multilayer perceptron, although a well-known general function approximator, is not suitable for directly approximating an action-value function because it is an offline algorithm. This means that an additional mechanism such as extra memory is required to store histories
of states, actions, and rewards [4].
Function approximators based on a previous knowledge of action-value function landscapes have also been
proposed [5, 6], but are applicable only if there is previous knowledge available.
Linear methods such as tile coding [1], the radial basis
function network (RBFN) [1, 7] and fuzzy-based methods [8, 9] are often used to approximate a target function
by using a weighted linear sum of features. Although all
826

Fig. 1. SDNN-C structure.

of these methods perform well without previous knowledge, they involve increasing difficulty as the numbersof experience required grow exponentially with the state
space dimension [1]. This makes them unsuitable for theapproach of adding redundant sensors.
In short, available function approximators are not easily applied to Q-learning in continuous state-action spacewith noisy and redundant input.

3. Proposal
We propose introducing SDNN-C, an SDNN-based
function approximator, for action-value functions.

3.1. SDNN-C Structure
SDNN-C has the six layers shown in Fig. 1.
Layer 1, which is the input layer, consists of l (l ≥ 2)neural units, where l is the number of input signals. Each
unit corresponds to one input signal. The input signal vector is denoted by x .
Layer 2, which is a pattern coding layer for the state,consists of l neural groups, x̄x 1 , . . . , x̄x l . This group’s
units’ output are denoted x̄x i . Each group corresponds to
one unit in layer 1 and outputs an m-dimensional binary

Journal of Advanced Computational Intelligenceand Intelligent Informatics

Vol.19 No.6, 2015

Q-Learning Using SDNN for Noisy and Redundant Inputs

pattern on {−1, +1}m, where m is the number of units in
the group. The pattern is determined as follows:
1. Construct 9 random binary patterns ( p̄p1 , p̄p2 , . . . , p̄p9 ).
2. Construct 44 binary patterns by interpolating between p̄pi and p̄pi+1 (i = 1, 2, . . ., 8).
3. Divide the output range of the corresponding unit inlayer 1 into 360 equal subranges.
4. Assign 360 patterns to each subrange, omitting p̄p9 .
Layer 3, which is a state desensitization layer, consistsof l × (l − 1) neural groups, {x̄x μ (ν ) | μ , ν = 1, . . . , l; μ =ν }, where x̄x μ (ν ) denotes x̄x μ desensitized by x̄x ν . Desensitization neutralizes outputs of units in x̄x μ when the corresponding unit in x̄x ν outputs −1, regardless of the unitin x̄x μ . The output of the k-th unit is given by
1 + σμν x̄x ν k μμ (ν )
x̄k ,
x̄k
=
2
μ

x̄k is the output of the k-th unit of x̄x μ and σμν is an mdimensional permutation matrix that randomly permutes
units of x̄x ν . For simplicity, a vector containing all x̄x μ (ν )for μ and ν (μ = ν ), is denoted by x̄x SD .
Layer 4 consists of two different neural groups, i.e., ȳyand z̄z. ȳy is a pattern coding group for action y, and hasn units. It outputs an n-dimensional binary pattern representing the action determined by the following procedure:
1. Construct (n−n −1) patterns, in which n represents
the number of 1s. The i-th element of the k-th patternis given byq̄k,i =

+1 (if k ≤ i < k + n ).
−1 (otherwise)

2. Divide the range of the action variable into(n − n − 1) equal subranges.
3. Assign these patterns to each subrange.
z̄z is a hidden layer with n units. Every unit in z̄z is connected to every unit in layer 3. The output of the i-th unitof z̄z is given bySDxz̄i = φ w 
x̄
−
w
i0
i
ml(l−1)=φ∑ wi j x̄ jSD − wi0 ,

j=1

where


1
φ (u) =
0(u > 0),(otherwise)x̄ jSD is the j-th element of x̄x SD , wi j is a synaptic weight
between the j-th unit in layer 3 and the i-th unit inlayer 4 and wi0 is a threshold. The quantityz̄z ofSD − wxx̄
w
i0 is called the inner potential of the unit.
iVol.19 No.6, 2015

Parameters wi j and wi0 are updated as detailed in the next
section.
Layer 5 is a desensitization layer by action and has nneural units. The output of the i-th unit in this layer iscalculated by
1 + ȳiz̄i .
2Layer 6, which is the output layer, has single output
unit z, which is determined from the number of units outputting +1 in layer 5. The output of layer 6 is calculated
as follows:z̄i (ȳi ) =z = Q(xx, y)=g∑ z̄i (ȳi ) ,i∈I

where I is index set I = {1, 2, . . ., n} and g(u) is a scaling
function. z is also given byz=g∑ z̄ j ,j∈J(y)

where J(y) = { j : ∀ j ∈ I, ȳ j = 1}.

3.2. SDNN-C Learning
SDNN-C learns based on error correction training.
When an agent moves to successive state st+1 and receives reward rt+1 , it updates its action-value function by
updating synaptic weights wi j and threshold wi0 . This updating is done to minimize the distance between the current action value and a new action value given by Eq. (1).
Let z be the current action value and ẑ be the new action values. Let u be the number of units outputting +1
in layer 5. Let û be the estimated number of units satisfying ẑ = g(û). These variables determine the synaptic
weights and thresholds to be updated and their updated
values. The learning algorithm is detailed in Fig. 2.

4. Simulation Experiments
We conducted two simulation experiments to assess theeffectiveness of our proposed SDNN-C. In these experiments, we used an acrobot swing-up task that differs in
two ways from the original version [1]. Difference 1 isthe continuity of state-action space. In these experiments,
we used continuous state-action space. Difference 2 is thecondition for completing the task. We set limits on the
angular velocities for the two links and made it tighter as
episodes elapsed. This condition gradually requires more
precise action selection and contributes to evaluating theaccuracy of the action-value function against continuous
action. SDNN-C is used to approximate the action-valuefunction for Q-learning. We quantified SDNN-C performance for the following points:• The number of episodes required to learn a suitableaction-value function

Journal of Advanced Computational Intelligenceand Intelligent Informatics

827

Kobayashi, T., Shibuya, T., and Morita, M.

1.

Divide a set of units not neutralized by the desensitization process in layer 4 into |û − u| subsets
Z1 , Z2 , . . . , Z|û−u| , preserving index order. Subsets
should be of equal size, as far as possible.

2.

If possible, select one unit in each set Zi as follows:(a) if z is greater than ẑ, select the unit with the lowest magnitude of inner potential that outputs +1;(b) if z is less than ẑ, select the unit with the lowest
magnitude of inner potential that outputs 0.

3.

Update the synaptic weights and thresholds of selected units by Eqs. (2) and (3), respectively.
wi j ← wi j + c sgn(û − u) x̄ jSD . . . . . (2)wi0 ← wi0 − c sgn(û − u) , . . . . . . (3)where sgn(·) is the sign function, and c is a learning
coefficient.
Fig. 2. SDNN-C learning algorithm.

1/300, −10 + 2/300, . . ., 0, . . . , 10 − 1/300, 10} given byequal divisions of the torque range. The state of the acrobot is described by angles θ1 , θ2 ∈ [−π , π ) and angularvelocities θ˙1 , θ˙2 . We set physical parameters as follows:
Link masses m1 = m2 = 1 kg; Link lengths l1 = l2 = 1 m;
Distances to the center-of-mass of links lc1 = lc2 = 0.5 m;
Moments of inertia of links I1 = I2 = 1 kg · m2 ; Gravitational acceleration g = 9.8 m/s2 .
The agent’s goal in this task was to swing the tip of link
2 above the goal level in the minimum time and to momentarily stop it at the goal height. This task consists ofsubsequences called episodes. Each episode began withan initial state, (θ1 , θ2 , θ˙1 , θ˙2 ) = (0, 0, 0, 0). The agent
controlled the acrobot at 1-s intervals toward the goal. The
agent reached the goal if the current state of the acrobot
satisfied the following two conditions:
1. The tip reached 1.5 m above joint 1.
2. |θ˙1 | < 3πλ and |θ˙2 | < 5πλ ,where variable λ specifies the difficulty in reaching thegoal. λ is defined ask−1
10−(1+ 10 000−1 ) (k ≤ 10 000)
λ=
,
10−2
(k > 10 000)
where k is the number of episodes. Note that reaching
the goal becomes increasingly difficult as the number of
episodes increases. If the acrobot attains its goal or fails
to reach it within 50 s, it is restored to its initial state anda new episode is begun.
If the agent reaches its goal, it receives +10 as a reward.
If |θ1 | is below 5◦ or angular velocities θ˙1 ，θ˙2 are outsideof sensor range [−3π , 3π ], [−5π , 5π ], it receives −5. In
all other states, it receives no reward whatsoever.
Initial outputs of function approximators were set tozero in all states and actions because we lacked previous
knowledge of the acrobot completing this task.

Fig. 3. Acrobot.

• The number of episodes until the number of steps per

4.2. Experiment A

episode becomes larger again• The number of steps in the final episode if an agent

achieves a goal stably by the final episodeIn experiment A, we confirm SDNN-C efficiency incontinuous state-action space with redundant input. In experiment B, we confirm SDNN-C robustness in continuous state-action space with noisy and redundant input.

4.1. Acrobot Swing-Up Task
The acrobot [13] scheme is shown in Fig. 3. This underactuated two-link manipulator is commonly used as an
RL benchmark in continuous state spaces [1, 5]. Joint
1 of the acrobot is freely movable and affixed to a wall.
Joint 2 has an actuator that outputs the torque set by thecontroller. Torque τ is set from −10 Nm to +10 Nm. The
agent selects one action in action set A = {−10, −10 +828

In experiment A, we compare two cases: that with additional redundant inputs and that without.
4.2.1. Experimental Setup
The agent observed four normalized state variables
s1 , . . . , s4 given byθ1+ 0.5,2π
θ2
s2 =+ 0.5,2πθ˙1+ 0.5 ,
s3 = fsat
6π

 ˙
θ2
s4 = fsat+ 0.5 ,10π
s1 =Journal of Advanced Computational Intelligenceand Intelligent Informatics

Vol.19 No.6, 2015

Q-Learning Using SDNN for Noisy and Redundant Inputs

fsat (x) is a saturation function defined as⎧⎨ 1 (1 ≤ x)x (0 ≤ x < 1) .
fsat (x) =⎩ 0 (x < 0)
Two types of redundant inputs, s5 ∈ [0, 1] and s6 ∈ [0, 1],
were used. s5 is the simplified kinetic energy of the acrobot and s6 is the normalized height of the acrobot fromjoint 1. These were given by
s5 = 2 (s3 − 0.5)2 + (s4 − 0.5)2 ,1
cos (2π s1 ) − cos (2π (s1 + s2 )) + 0.5.
s6 =4
In this task, the agent was required to sequentially select correct actions. To prevent random actions from being selected during this evaluation, we specified two types
of episodes—learning episodes and test episodes. During a learning episode, the agent determined action basedon the ε -greedy policy and learned the action-value function. During the first 10 steps, probability ε was set to 0
and thereafter to 0.1. RL parameters were α = 0.5 andγ = 0.9. After each interval of 10 learning episodes, one
test episode was executed. During a test episode, the agent
used the greedy policy without learning the action-valuefunction. The test episode evaluated the agent’s performance throughout previous learning episodes.
We conducted 10 trials and evaluated the number ofsteps per test episode in each case. Each trial consisted of
15,000 learning episodes and 1500 test episodes.
SDNN-C parameters were m = 200, n = 900, n = 300,
and c = 0.1, enabling SDNN-C to express 601 action values. The output range of SDNN-C was [−20, 20], and thescaling function g(u) in layer 6 was defined by⎧⎪
20(280 < u)⎪⎨300
2g(u) =
u−(20 ≤ u ≤ 280) .
⎪
13
13
⎪
⎩−20
(u < 20)
All synaptic weights were assigned a small random value.
Even-numbered units were assigned large positive thresholds and odd-numbered large negative thresholds.
4.2.2. Experiments and Results
Experiment A results are shown in Fig. 4. The vertical axis indicates the number of steps in the test episodefor each learning episode and the median number of steps
among 10 trials is plotted.
These graphs show that the SDNN-C agent learned theaction-value function within about 150 learning episodes,
finally achieved the goal within 10 steps in both cases.
This shows the robust performance of our proposal in redundant state space dimensions.
The computational cost of SDNN-C increased quadratically as the number of inputs increased, whereas the costof a conventional method such as RBFN increased exponentially. This property is advantageous in the case of
Vol.19 No.6, 2015(a) The SDNN-C agent observing only 4 normalized state variables.

(b) The SDNN-C agent observing 4 normalized state variables plus 2
redundant variables.

Fig. 4. Experiment A results.

redundant input. In a case of 6-dimensional state space,for example, computation time for one step using SDNNC and RBFN with 4 basis functions for each dimension
was about 50 ms and 200 ms, respectively. Note that thehigher the dimension, the greater the difference between
RBFN and SDNN-C.

4.3. Experiment B
In experiment B, we conducted simulation experiments
using an acrobot swing-up task with noisy sensor input toconfirm SDNN-C robustness against noise and redundant
input. We also used SDNN-C as a value function approximator for Q-learning. Although RBFN is well known as
a function approximator in RL, it could not be used forthis experiment due to its high computational cost. In thisexperiment, we used fuzzy Q-learning comparatively because its computational cost is low enough below RBFN
on the parameters configured to make them perform at thesame level approximation accuracy.
4.3.1. Fuzzy Q-Learning
For comparison, we used Jouffe’s fuzzy Q-learning(FQL) algorithm [8]. The FQL agent has N rules activatedfor different states. Each rule Ri has parameter vector vi
representing the desirability of actions in discrete action
set U.
The agent selects action at based on activated rules andits parameter vectors as follows:at =∑ ε -Greedy(vi ) · αRi ,Ri ∈A(st )A(st ) is the activated rule set in state st , αRi is the output of rule Ri (called a truth value) and ε -Greedy(·) is afunction that selects a discrete action in U based on vi . In

Journal of Advanced Computational Intelligenceand Intelligent Informatics

829

Kobayashi, T., Shibuya, T., and Morita, M.

Table 1. Cases of redundant state space.
Name of case
Case I
Case II
Case III
Case IV

Components of state observed by theagent(s1 , s2 , s3 , s4 )(s˜1 , s˜2 , s˜3 , s˜4 )(s˜1 , s˜2 , s˜3 , s˜4 , s˜5 )(s˜1 , s˜2 , s˜3 , s˜4 , s˜5 , s˜6 )his original study, Jouffe proposed and used another typeof ε -greedy function, but in this experiment, we used thesame function applied to SDNN-C.
Parameter updating is based on Eq. (1)Fig. 5. Experiment B results for Case I.

∀Ri ∈ A(st ),v [ati ] ← vi [ati ] + c (rt+1 + γ Qt∗ (st+1 ) − Qt (st , at )) αRi ,ivi [ati ] is a parameter for the action selected by rule Ri attime t, c is a learning coefficient and Q∗ (s) represents the
optimal action value in state s. Qt∗ (st+1 ) and Qt (st , at ) arecalculated by∗i
Qt (st+1 ) = ∑max v [a] αRi ,Ri ∈A(st+1 )Qt (st , at ) =a∈U(a) Case II: containing no redundant variable∑ vi[ati ]αRi .

Ri ∈A(st )4.3.2. Experimental Setup
Experiment B was conducted for the four cases in Table 1. Case I contained no redundant input or sensor
noise.
In contrast, in case II, state variables observed by theagent included independent Gaussian noise, which were
calculated as follows:(b) Case III: containing 1 redundant variable s˜5s˜1 = fwrap (s1 + e1 ) ,s˜2 = fwrap (s2 + e2 ) ,s˜3 = fsat (s3 + e3 ) ,s˜4 = fsat (s4 + e4 ) ,ei (i = 1, . . . , 4) is Gaussian noise (mean μ = 0 and standard deviation σ = 0.02) and fwrap (x) is a function thatwraps x to the interval [0, 1].
In cases III and IV, the agent also observed noisy andredundant inputs s˜5 and s˜6 , given bys˜5 = fsat (s5 + e5 ) ,s˜6 = fsat (s6 + e6 ) .

Fig. 6. Experiment B results for Cases II–IV.

4.3.3. Experiments and Results

FQL parameters were determined by preliminary experiments. Specifically, each state dimension was partitioned at even intervals with nine triangular membershipfunctions. The number of fuzzy rules N was 9d , whered is the total number of state dimensions. Discrete action sets U of these rules were the same for all rules, andcontained 21 actions given by equally dividing the torquerange.
Trial settings and the parameter of SDNN-C and Qlearning were the same as those used in experiment A.
830(c) Case IV: containing 2 redundant variables s˜5 and s˜6Experiment B results are presented in Fig. 5, representing case I, and Fig. 6, representing cases II–IV, where themedian number of steps with SDNN-C and FQL amongthe 10 trials is plotted.
We compared the influence of sensor noise on bothmethods in Figs. 5 and 6(a). These graphs show thatthe performance of the FQL agent depended strongly on
the presence or absence of sensor noise. Although the
FQL agent could learn within approximately 100 episodes
in a noise-free case, it failed to learn stable upswing actions even in the early episodes in the noisy case. In case

Journal of Advanced Computational Intelligenceand Intelligent Informatics

Vol.19 No.6, 2015

Q-Learning Using SDNN for Noisy and Redundant Inputs

II, SDNN-C agent performance in early episodes was almost equal to both methods in case I. After that, it beganworsening at about episode 7,500, and the agent failed toachieve the goal stably due to the relative increase in thenoise influence.
Figure 6 shows that the FQL agent failed to learn stable upswing actions due to the effect of sensor noise in allcases. In contrast to FQL in early episodes, the SDNNC
agent learned the suitable action-value function within almost the same number of episodes regardless of additional
redundant input. The SDNN-C agent in case III kept asuitable action selection until a later episode than in case
II. In particular, in case IV, the SDNN-C agent finallyachieved the goal in about 4 steps, which is almost thesame as in case I. Although SDNN-C performance may
depend on the type of additional input, noise tolerance
tended to improve as the number of redundant inputs increased.
Based on these results, we confirmed that SDNN-C isstrongly robust against sensor noise and redundant input
and takes better actions by using information from additional inputs. We consider that these properties are realized thanks to powerful, nonlocal generalization and thehigh representation capability of SDNN, by which, designing state space may become easier as system designers are enabled to add any input that may be related tocontrol.

References:[1] R. S. Sutton and A. G. Barto, “Reinforcement Learning: An Introduction,” MIT Press, 1998.
[2] C. J. C. H. Watkins, “Learning from delayed rewards,” Ph.D. thesis,
University of Cambridge, 1989.
[3] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning,
Vol.8, pp. 279-292, 1992.
[4] C. Gaskett, D. Wettergreen, and A. Zelinsky, “Q-learning in continuous state and action spaces,” Proc. 12th Australian Joint Conf. on
Artificial Intell., Sydney, pp. 417-428, 1999.
[5] G. Konidaris, S. Osentoski, and P. Thomas, “Value function approximation in reinforcement learning using the Fourier basis,” Proc.
25th Conf. on Artificial Intell., San Francisco, pp. 380-385, 2011.
[6] A. Geramifard, M. Bowling, and R. S. Sutton, “Incremental leastsquare temporal difference learning,” Proc. 21th Conf. on Artificial
Intell., Boston, pp. 356-361, 2006.
[7] J. Park and I. W. Sandberg, “Universal approximation using
radial-basis-function networks,” Neural Computation, Vol.3, No.2,
pp. 246-257, 1991.
[8] L. Jouffe, “Fuzzy inference system learning by reinforcement methods,” IEEE Trans. on Systems, Man, and Cybernetics, Part C: Applications and Reviews, Vol.28, No.3, pp. 338-355, 1998.
[9] P. Y. Glorennec, “Reinforcement learning: An overview,” Proc. European Symposium on Intelligent Techniques (ESIT-00), Aachen,pp. 14-35, 2000.
[10] K. Nonaka, F. Tanaka, and M. Morita, “Empirical comparison of
feedforward neural networks on two-variable function approximation,” IEICE Trans. Inf. & Syst.(Japanese Edition), Vol.94, No.12,
pp. 2114-2125, 2011.
[11] K. Horie, A. Suemitsu, and M. Morita, “Direct estimation of handmotion speed from surface electromyograms using a selective desensitization neural network,” J. Signal Process., Vol.18, No.4,
pp. 225-228, 2014.
[12] T. Kobayashi, T. Shibuya, and M. Morita, “Q-learning in Continuous State-Action Space with Redundant Dimensions by Using a
Selective Desensitization Neural Network,” Proc. SCIS&ISIS 2014,
Kitakyushu, pp. 801-806, 2014.
[13] M.W. Spong, “The swing up control problem for the acrobot,” IEEE
Control Syst. Mag., Vol.15, No.1, pp. 49-55, 1995.

5. Conclusions
We have proposed SDNN-C, novel value-function approximation for Q-learning in continuous state-action
space based on SDNN. We evaluated this method in an
acrobot task extended to continuous state-action space including noisy and redundant input. In experiment A, we
confirmed that SDNN-C is applicable to continuous stateaction space with redundant input. Experiment B results
demonstrated that SDNN-C is strongly robust against sensor noise and redundant input, thus enabling an agent totake better action by using additional input without learning efficiency deteriorating. We consider that these properties have been realized by the powerful, nonlocal generalization and the high representation capability of SDNN.
These are advantageous in real-world applications such as
robotic systems.
In future studies, we plan to apply our proposal to extend methods to multidimensional action space. We alsoplan to experimentally compare the performance of our
proposed method to that of other RL algorithms such as
actor–critic methods.
Acknowledgements
This study was supported partly by JSPS KAKENHI Grant Numbers 22300079, 24760308, and 26590173.

Vol.19 No.6, 2015

Name:
Takaaki Kobayashi

Affiliation:
Faculty of Engineering, Information and
Systems, University of Tsukuba

Address:
1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8573 Japan

Brief Biographical History:
2006- College of Engineering Systems, University of Tsukuba
2010- Department of Intelligent Interaction Technologies, University ofTsukuba
2015- Faculty of Engineering, Information and Systems, University ofTsukuba

Main Works:• Neural networks and reinforcement learning• “Q-Learning in Continuous State-Action Space by Using a Selective
Desensitization Neural Network,” IEICE Trans. Inf. & Syst. (Japanese
Edition), Vol.98, No.2, pp. 287-299, 2015.

Journal of Advanced Computational Intelligenceand Intelligent Informatics

831

Kobayashi, T., Shibuya, T., and Morita, M.

Name:
Takeshi Shibuya

Affiliation:
Faculty of Engineering, Information and
Systems, University of Tsukuba

Address:
1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8573 Japan

Brief Biographical History:
2007-2010 Ph.D. Candidate, Graduate School of Engineering, Yokohama
National University
2007-2010 Research Fellow, the Japan Society for the Promotion of
Science2011- Assistant Professor, University of Tsukuba

Main Works:• “Complex-Valued Reinforcement Learning: a Context-Based Approachfor POMDPs,” Advances in Reinforcement Learning, pp. 255-274, InTech,
2011.

Membership in Academic Societies:• The Institute of Electrical and Electronics Engineers (IEEE)• Institute of Electronics, Information and Communication Engineers(IEICE)• The Institute of Electrical Engineers of Japan (IEEJ)Name:Masahiko Morita

Affiliation:
Faculty of Engineering, Information and
Systems, University of Tsukuba

Address:
1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8573 Japan

Brief Biographical History:
1991 Received Doctoral degree of Engineering from the University ofTokyo
1992.6-2000.3 Lecturer, University of Tsukuba
2000.3-2007.3 Assistant Professor, University of Tsukuba
2007.4-2007.5 Associate Professor, University of Tsukuba
2007.6- Professor, University of Tsukuba

Main Works:• Biological information processing and neural computation• “Associative memory with nonmonotone dynamics,” Neural networks,
Vol.6, No.1, pp. 115-126, 1993.

Membership in Academic Societies:• The Society of Instrument and Control Engineers (SICE)• The Institute of Electronics, Information and Communication Engineers(IEICE)• Japanese Neural Network Society (JNNS)

832

Journal of Advanced Computational Intelligenceand Intelligent Informatics

Vol.19 No.6, 2015