Reinforcement Learning for Continuous
Action using Stochastic Gradient AscentHajime KIMURA,

Shigenobu KOBAYASHITokyo Institute of Technology, 4259 Nagatsuda, Midori-ku Yokohama 226-8502
JAPANAbstract:

This paper considers a reinforcement learning (RL) where the set of possible

action is continuous and reward is considerably delayed. The proposed method is based on astochastic gradient ascent with respect to the policy parameter space; it does not require amodel of the environment to be given or learned, it does not need to approximate the valuefunction explicitly, and it is incremental, requiring only a constant amount of computation perstep. We demonstrate the behavior through a simple linear regulator problem and a cart-polecontrol problem.

1Introduction

This paper considers a reinforcement learning (RL) where the set of possible action
is continuous and reward is considerably delayed. RL is the on-line learning of an
input-output mapping through a process of trial and error to maximize some statistical
performance index.
[14] is a representative of the RL algorithms for Markov
decision processes in which the set of possible action is discrete. However, many applications in real-world need to handle the continuous action space, and often mixed itwith discrete action space. This paper describes a new approach to RL for continuousaction space. We de ne the agent's policy as a distribution of the action output, and wepresent a policy improvement algorithm. The proposed method is based on a stochastic gradient ascent with respect to the policy parameter space; it does not require amodel of the environment to be given or learned, it does not need to approximate thevalue function explicitly, and it is incremental, requiring only a constant amount ofcomputation per step. We demonstrate an application to linear regulator problems.
Q-learning

2

Related Works[3] and [1] have proposed DP-based RL methods for only LQR problems. RFALCON[10] uses Adaptive Heuristic Critic [2] combined with a fuzzy controller. It is a policy
improvement method which needs to estimate the value function explicitly.

AgentProbability of action aaX
(a; W; X )Observation
ActionW : internal variable--Figure 1: Stochastic policy; The agent can improve the policy  by modifying theparameter vector W .
3

Stochastic Gradient Ascent (SGA)

The objective of the agent is to form a[12], that assigns a probability
distribution over actions to each observation, so that maximize some reward function.
A policy (a; W; X ) denotes probability of selecting action a in the observation X (Figure 1). The policy (a; W; X ) is a probability density function when the set of possible
action values a is continuous. The policy is represented by a parametric function approximator using the internal variable vector W . The agent can improve the policy by modifying W . For example, W corresponds to synaptic weights where the action
selecting probability is represented by neural networks, or W means weight of rules inclassi er systems. The advantage of the parametric notation of  is that computational
restriction and mechanisms of the agent can be speci ed simply by a form of the function, and we can provide a sound theory of learning algorithms for arbitrary types ofagents.
stochastic policy

1. Observe X in the environment.
2. Execute action a with probability (a ; W; X ).
3. Receive the immediate reward r .
4. Calculate e (t) and D (t) astttttii

= @w@ ln (a ; W; X ) ,D (t) = e (t) + D (t 1) ,where (0  < 1) denotes the discount factor, and w does the ith component of W .
5. Calculate w (t) asw (t) = (r b) D (t) ,where b denotes the reinforcement baseline.
6. Policy Improvement: update W as
W (t) = (w1(t); w2(t)    w (t)   ) ,WW + (1 ) W (t) ,where is a nonnegative learning rate factor.
7. Move to the time step t + 1, and go to step 1.
ei (t)

tiiitiiiiti

iFigure 2: General form of the SGA algorithm.
Figure 2 shows a general form of the algorithm. The notation e (t) in the 4th
procedure is called[15], that speci es a correlation between the associatedpolicy parameter w and the executed action a . D (t) is a discounted running averageof eligibility. It accumulates the agent's history. When a positive reward is given, theagent updates W so that the probability of actions recorded in the history is increased.
ieligibility

itiSome theorems have shown in [7], [8] and [9] that the weight changes in the direction of
the expected discounted reward biased by the state occupancy probability. Although
any convergence theory of this algorithm have not shown, it has the following practicaladvantages.
 It is easy to implement multidimensional continuous action, that is often mixed
with discrete action.
 Memory-less stochastic policies can be considerably better than memory-less deterministic policies in the case of partially observable Markov decision processes(POMDPs) [12] or multi-player games [11].
 It is easy to incorporate an expert's knowledge into the policy function by applying
conventional supervised learning techniques.
Algorithms for Continuous Action
Remember that the policy (a; W; X ) is a probability density function when the setof possible action values a is continuous. The normal distribution is a simple andwell-known multiparameter distribution for a continuous random variable. It has twoparameters, the mean  and the standard deviation . When the policy function  isgiven by the equation 1, the eligibility of  and  are1(a ) )
 (a; ;  ) = p exp((1)
2 2
22e ==at 2(at )2
3(2)

2.
(3)
One useful feature of such a[15] is that the agent has a potential tocontrol it's degree of exploratory behavior. Because the parameter  is occupying
the denominators of equation 2 and 3, we must draw attention to the fact that theeligibility is to divergent when  goes close to 0. The divergence of the eligibility hasa bad in uence on the algorithm. One way to overcome this problem is to control thestep size of the update parameter vector using . Such an algorithm is obtained by
setting the learning rate parameter proportional to  , then the eligibility is given bye = a (4)(a )  .
(5)
e =
e

Gaussian unit

2

t

4

2t



2

Preliminary Experiment: A LQR Problem

The following linear quadratic regulator problem can serve as a benchmark. At a given
discrete-time t, the state of the environment is the real value x . The agent chooses acontrol action a which is also real value. The dynamics of the environment is:x = x + a + noise ,(6)
where noise is the normal distribution that follows the standard deviation  = 0:5.
The immediate reward is given byr = x a .
(7)
tt

t+1tt

noiset2t

2t

The goal is to maximize the total discounted reward,
1X
r ,
t(8)

t

t=0

where is some discount factor 0  < 1. Because the task is a linear quadratic
regulator (LQR) problem, it is possible to calculate the optimal control rule. From theRiccati equation, the optimal regulator is given by2pa = k x , where k = 1(9)1 + 2 + 4 + 1.
t

1t

1

2

The optimum value function is given by V (x ) = k x , where k is a some positive
constant. In this experiment, the possible state is constrained to lie in the range [ 4; 4].
When the state transition given by Equation 6 does not result in the range [ 4; 4], thevalue of x is truncated. When the agent chooses an action which is not lie in the range[ 4; 4], the action executed in the environment is also truncated.
t2

22tt

4.1

Implementation for SGA

The agent would rst compute values of  and  deterministically and then draw
its output from the normal distribution that follows mean equal to  and standard
deviation equal to . The agent has two internal variables, w and w , and it computes
the value of  and  according to1
=w x , =(10)
1 + exp( w ) :Then, w can be seen as a feedback gain parameter. The reason for the calculationof  is to guarantee the value to keep positive. We write e , e as the characteristic
eligibility of w and w respectively. From Equation 4 and 5, e and e are given by@ = (a ) x
(11)
e = e@w
1

1t

2

2

1

1

1

2

1

2

1

t

12t

@
 = ((at )2  2 )(1  ) .
(12)@w2The learning rate is xed to = 0:01, reinforcement baseline b = 0:0, discount rate
= 0:9. The value of w1 is initialized to 0:35  0:15, and w2 = 0, i.e.,  = 0:5.
e2 = e4.2

Implementation for Actor/Critic Algorithms

We compare the algorithm with an actor/critic algorithm. The critic quantizes thecontinuous state-space ( 4  x  4) into an array of boxes. We have tried two typesof the quantizing: one is discretizing x evenly into 3 boxes, the other is 10 boxes. The
critic attempts to store in each box a prediction of the value V^ by using TD(0) [13].
The critic provides TD-error = r + V^ (x ) V^ (x ) to the actor as a reinforcement.
The actor updates policy parameters with using Equation 11, 12 as:w = w +  (TD-error)  e
w = w +  (TD-error)  e
The learning rate for TD(0) is xed to 0:2, and all the other parameters are the sameas the SGA method.
tt+1

t

1

1

1

2

2

2

4.3

Results

0.6
beta = 0.9
0.4

Feedback gain

0.2

0-0.2

-0.4

-0.6

optimum-0.8

Optimum point

-1
0500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Learning steps
0Figure 3: The average performance of theproposed method over 100 trials.

-50-100-150-200-250-300
-350
00.6

0.4

-20.2

-1.5

0.5
-1-0.500.5

Feedback gain

Feedback gain

Deviation
11.5

2

10-0.2

Figure 5: Value function over the parameter space in the LQR problem, where= 0:9. It is fairly at around the optimum:  = 0:5884,  = 0.

gamma=0.9, Critic’s Grid=3-0.4
gamma=0.9, Critic’s Grid=10-0.6
optimum-0.8-10500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Learning steps

Figure 4: The average performance of theactor/critic algorithm over 100 trials. The
critic uses 3 or 10 boxes.

Figure 3 shows the performance of the SGA algorithm in the LQR problem. The
variable of the feedback gain has a tendency to drift around the optimum. The parameter of  decreased, but in most case, the growing stopped around 0:2. This result is not
so pessimistic. Figure 5 shows the value function which are de ned by Equation 7 and
8 over the parameter space ( and ). The value of performance is fairly at aroundthe optimal solution. For this reason, we can conclude that the proposed method would
obtain a good policy for the LQR without estimating value function.
Figure 4 shows the performance of the actor/critic algorithms. The actor/critic
algorithm using 3 boxes converged not close to the optimum feedback gain. The reason
for this is that the critic's ability of the function approximation (3 boxes) is insuÆcient
for learning policy, whereas the policy representation is the same.
5

Applying to a Cart-Pole Problem

The behavior of this algorithm is demonstrated through a computer simulation of acart-pole control task, that is a multi-dimensional nonlinear nonquadratic problem.
We modi ed the cart-pole problem described in [2] so that the action is taken to becontinuous.




F-x=0

xj j-Figure 6: The cart-pole problem.
The dynamics of the cart-pole system is modeled by
g sin  + cos 
,
 =`


F + m` _ sin   cos   sgn(x_ )
x =,
M +m
where M = 1:0(kg) denotes mass of the cart, m = 0:1(kg) is mass of the pole, 2` = 1(m)is a length of the pole, g = 9:8(m=sec ) is the acceleration of gravity, F (N) denotes
the force applied to cart's center of mass,  = 0:0005 is a coeÆcient of friction of cart,
 = 0:000002 is a coeÆcient of friction of pole. In this simulation, we use discrete-timesystem to approximate these equations, where t = 0:02sec. At each discrete time
step, the agent observes (x; x_ ; ; _), and controls the force F . The agent can executeaction in arbitrary range, but the possible action in the cart-pole system is constrained
to lie in the range [ 20; 20](N ). When the agent chooses an action which is not lie inthat range, the action executed in the system is truncated. The system begins with
(x; x_ ; ; _) = (0; 0; 0; 0). The system fails and receives a reward (penalty) signal of 1
when the pole falls over 12 degrees or the cart runs over the bounds of its track( 2:4  x  2:4), then the cart-pole system is reset to the initial state.
m`2 sin  +c sgn(x_ )M +mFm cos2 M +m432c2cp

p _m`In this experiment, the state space is normalized as (x; x_ ; ; _) = (2:4 m; 2 m/sec;  12=180 rad; 1:5 rad/sec) into (0:5; 0:5; 0:5; 0:5). The agent's policy function has ve internal variables w    w , and computes the  and  according toxx__ = w+w +w+w
2:4
2
12=180
1:5 ,1(13) = 0:1 +1 + exp( w ) :
1

15t

2tt

3

4t

5

The eligibilities e    e are given bye = (a ) x , e = (a ) x_e = (a )  , e = (a ) _e = ((a )  )(1 + 0:1  ) .
The critic discretizes the normalized state space evenly into 3  3  3  3 = 81 boxes,and attempts to store in each box V^ by using TD(0) algorithm [13]. Parameters are
set to = 0:95, = 0:01, and learning rate for TD(0) in the actor/critic is 0:5.
Figure 7 shows the performance of two learning algorithms in which the policy
representation is the same. The proposed algorithm achieved best results. In contrast,
the actor/critic algorithm couldn't learn the control policy because of the poor abilityof function approximation in the critic.
151t

3ttt

52ttt

4tt

22

4000

3500

Time steps until failure

3000

2500

2000

SGA

1500

1000

Actor/Critic
500

00

50

100

150

200

250

300

350

400

450

500

TrialsFigure 7: The average performance of the algorithms on 100 trials. The critic uses
3  3  3  3 boxes. A trial means an attempt from initial state to a failure.
6Conclusion

This paper has considered a reinforcement learning where the set of possible action iscontinuous and reward is considerably delayed. We have proposed an policy improvement method that is based on a stochastic gradient ascent with respect to the policy

parameter space; it does not require a model of the environment to be given or learned,
it does not need to approximate the value function explicitly, and it is incremental,
requiring only a constant amount of computation per step. To our knowledge, this is
the rst study of the stochastic gradient method on discounted reward applying to RL
tasks which have continuous action space. We have demonstrated the performance incomparison with an actor/critic algorithm. The proposed method enables to learn an
acceptable policy with less cost rather than increasing the critic's ability of function
approximation in our test cases.
References[1] Baird, L. C.: Reinforcement Learning in Continuous Time: Advantage Updating, Proceedings ofIEEE International Conference on Neural Networks , Vol. IV, pp. 2448-2453 (1994).
[2] Barto, A. G., Sutton, R. S. and Anderson, C. W.: Neuronlike Adaptive Elements That Can SolveDiÆcult Learning Control Problems,[3] Bradtke, S. J.: Reinforcement Learning Applied to Linear Quadratic Regulation, Advances in
Neural Information Processing Systems 5 , (1992).
[4] Clouse, J. A. & Utogo , P. E.: A Teaching Method for Reinforcement Learning, Proc. of the 9thInternational Conference on Machine Learning , pp. 93-101 (1992).
[5] Crites, R. H. and Barto, A. G.: An Actor/Critic Algorithm that is Equivalent to Q-Learning,Advances in Neural Information Processing Systems 7 , pp. 401-408 (1994).
[6] Doya, K. : EÆcient Nonlinear Control with Actor-Tutor Architecture, Advances in Neural Information Processing Systems 9 , pp. 1012{1018 (1996).
[7] Kimura, H., Yamamura, M., & Kobayashi, S.: Reinforcement Learning by Stochastic Hill Climbing
on Discounted Reward, Proceedings of the 12th International Conference on Machine Learning ,pp.295-303 (1995).
[8] Kimura, H. & Yamamura, M. & Kobayashi, S.: Reinforcement Learning in Partially Observable Markov Decision Processes: A Stochastic Gradient Method, Journal of Japanese Society forArti cial Intelligence , Vol.11, No.5, pp.761-768 (1996 in Japanese).
[9] Kimura, H., Miyazaki, K. and Kobayashi, S.: Reinforcement Learning in POMDPs with Function
Approximation, Proceedings of the 14th International Conference on Machine Learning , pp. 152{
160 (1997).
[10] Lin, C. J. and Lin, C. T.: Reinforcement Learning for An ART-Based Fuzzy Adaptive Learning
Control Network, IEEE Transactions on Neural Networks , Vol.7, No. 3, pp. 709-731 (1996).
[11] Littman, M. L.: Markov games as a framework for multi-agent reinforcement learning, Proc. of
11th International Conference on Machine Learning , pp. 157-163 (1994).
[12] Singh, S. P., Jaakkola, T. and Jordan, M. I.: Learning Without State-Estimation in PartiallyObservable Markovian Decision Processes, Proceedings of the 11th International Conference onMachine Learning , pp. 284-292 (1994).
[13] Sutton, R. S.: Learning to Predict by the Methods of Temporal Di erences, Machine Learning
3 , pp. 9-44 (1988).
[14] Watkins, C. J. C. H., & Dayan, P.: Technical Note: Q-Learning, Machine Learning 8 , pp. 55-68(1992).
[15] Williams, R. J.: Simple Statistical Gradient Following Algorithms for Connectionist Reinforcement Learning, Machine Learning 8 , pp. 229-256 (1992).