On Discontinuous Q-Functions in Reinforcement Learning ? Alexander Linden AI Research Division German National Research Center for Computer Science -LRB- GMD -RRB- P. O. Box 1316 W-5205 Sankt Augustin Germany email : Alexander.Linden@gmd.de Abstract . This paper considers the application of reinforcement learning to path finding tasks in continuous state space in the presence of obstacles . We show that cumulative evaluation functions -LRB- as Q-Functions -LSB- 28 -RSB- and V-Functions -LSB- 4 -RSB- -RRB- may be discontinuous if forbidden regions -LRB- as implied by obstacles -RRB- exist in state space . As the infinite number of states requires the use of function approximators such as backpropagation nets -LSB- 16 , 12 , 24 -RSB- , we argue that these discontinuities imply severe difficulties in learning cumulative evaluation functions . The discontinuities we detected might also explain why recent applications of reinforcement learning systems to complex tasks -LSB- 12 -RSB- failed to show desired performance . In our conclusion , we outline some ideas to circumvent the problem . 1 Introduction In robotics and artificial intelligence , path finding is generally included under the heading \ path planning '' . In path planning , one looks for ways of , among other things , generating collision-free and efficient paths for robot manipulators and autonomous vehicles . Its scope also includes issues of traditional control problems and problem solving tasks . Standard methods like heuristic search suffer from their combinatorial complexity and inflexibility in changing environments . To get rid of these deficiencies , heuristic evaluation functions can be used . The method consists in directly selecting actions on the basis of the evaluation , which ? Published in : Ohlbach , H. J. -LRB- Ed . -RRB- , Proceedings of the German Workshop on Artificial Intelligence held at Bonn 1992 , Lecture Notes in Computer Science , Springer , 1993 . estimates the success of an action w.r.t. accomplishing a given goal . If this estimation is computed without search in state space and only on the basis of current state information , then \ path planning '' appears to be an improper expression ; instead the term \ path finding '' has been used in the literature -LSB- 14 -RSB- . In the field of reinforcement learning one is concerned with systems that incrementally learn such heuristic evaluation functions through interaction with the environment . Indeed it can be shown that , under certain conditions , the learning algorithms converge to the optimal evaluation function -LSB- 21 , 27 , 28 , 7 -RSB- . With an appropriate action generation mechanism , they form the core to path finding . Furthermore , such evaluation functions may be dynamically adapted to changes of the environment -LSB- 23 -RSB- . Currently most work on reinforcement learning has been done in the domain of finite and discrete state and action spaces -LSB- 5 , 23 , 26 , 27 , 33 , 32 -RSB- . Tesauro -LSB- 25 -RSB- had great success in applying this learning method to the game of backgammon . Much less research has been done on applications requiring continuous state space representations -LSB- 1 , 2 , 12 -RSB- . Here only quite limited tasks have been examined . To the best of our knowledge , reinforcement learning has never been applied to the problem of optimal path finding with changing goals , obstacles and with continuous state and action space . In this context we have observed discontinuous cumulative evaluation functions , that have not yet been recognized elsewhere . The paper is structured as follows : In the next section , we review learning with delayed reinforcement . Section 3 provides a general formulation of the path finding task . We argue that the optimal cumulative evaluation function should reflect the costs of the optimal path to the goal . Section 4 gives some path finding examples , illustrating the existence of discontinuous cumulative evaluation functions . In section 5 , we discuss why the learning of evaluation functions in domains with continuous state spaces is quite problematic . We conclude with considerations for future research in order to circumvent these problems . The result of this paper may explain why recent applications of reinforcement learning to quite complex tasks -LSB- 12 -RSB- failed to show the desired performance . 2 Short Review of Reinforcement Learning The general computational approach to learning from reward and/or punishment is known as reinforcement learning -LSB- 15 -RSB- . Very recently strong connections between dynamic programming in operations research -LSB- 6 -RSB- , heuristic evaluation functions in artificial intelligence -LSB- 10 -RSB- , adaptive control -LSB- 11 -RSB- , and reinforcement learning have been discovered -LSB- 3 , 27 , 31 -RSB- . This has caused a rapidly growing interest in this field2 . Although a multitude of algorithms and architectures is available that implement reinforcement learning -LSB- 21 , 29 , 34 -RSB- , we only consider reinforcement learning systems that deal with delayed reinforcement -LSB- 5 , 21 , 28 -RSB- . 2 See also the May , 1992 special issue on Reinforcement Learning in the Machine Learning Journal In addition , all these systems share a central mechanism that allows a computational agent to evaluate situations with respect to a prediction of the accumulated future reinforcement . Minsky -LSB- 15 -RSB- called this evaluation function \ secondary reinforcement '' , we will call it cumulative evaluation function . Because our focus is the study of this function under certain conditions it is appropriate to review reinforcement learning by one exemplary architecture . For this purpose we have chosen the efficient and widely used Q-learning -LSB- 27 , 28 , 12 -RSB- architecture . Let us consider a computational agent . At each time step t , it observes the state of its environment st 2 S. Its current policy ? maps st onto an action at = ? -LRB- st -RRB- . This action is executed in the environment , which results in a transition st +1 = ffi -LRB- st ; at -RRB- . In addition , the agent receives some spontaneous evaluation3 et = e -LRB- st ; at -RRB- . Some actions might yield a much smaller spontaneous evaluation than others but in the long term be much more favourable . This is the phenomenon of delayed reinforcement . The mechanism of telling how good an action or state actually was in the long term is called temporal credit assignment -LSB- 15 -RSB- . In order to develop method for learning a policy that is optimal w.r.t. delayed reinforcement , we derive a cumulative evaluation function4 , which tells how good a given action at will be in some state st. If ? is fixed , the agent loops through a sequence of states st +1 ; st +2 ; : : : and receives a corresponding sequence of spontaneous evaluations et ; et +1 ; et +2 ; : : : . The evaluation function relative to ? has the form of the discounted sum of future spontaneous evaluations Q ? -LRB- st ; at -RRB- : = 1 X k = 0 ket + k -LRB- 1 -RRB- = et + 1 X k = 1 k ? et + k Note that the discount factor < = < 1 assures the convergence of the infinite sum . In this sense the optimal policy ?? yields the maximal evaluation for all states and action pairs : Q ? -LRB- st ; at -RRB- = et + max 1 X k = 1 k ? et + k -LRB- 2 -RRB- = et + ? max 1 X k = 0 k ? et +1 + k -LRB- 3 -RRB- This immediately leads to Bellman 's optimality equation in dynamic programming -LSB- 3 -RSB- Q ? -LRB- st ; at -RRB- = et + ? max at +1 Q ? -LRB- ffi -LRB- st ; at -RRB- ; at +1 -RRB- -LRB- 4 -RRB- 3 Often it is also called reinforcement , reward , immediate payoff , outcome 4 Also called quality -LRB- Q -RRB- function by Watkins -LSB- 27 -RSB- Watkins -LSB- 27 -RSB- has designed a learning rule for autonomous systems based on this equation5 . Watkins proved that this learning rule is convergent under specific assumptions : -LCB- The state space S is finite and discrete . -LCB- In each state s 2 S , only a finite number of actions is allowed . -LCB- The environment is ergodic , this means that each state can be reached from every other state . -LCB- The distribution of spontaneous evaluations e -LRB- s ; a -RRB- does not change over time . -LCB- Each action in each state will be tried infinitely often . This property can be encoded in the action selection mechanism by the boltzmann distribution . The algorithm uses a lookup table to store the estimated cumulated evaluations ^ Q . It works as follows : 1 . Initialization : a -RRB- 8s ; a ^ Q -LRB- s ; a -RRB- : = 0 . b -RRB- The agent starts randomly in some state st. 2 . Action selection by random exploration : a -RRB- The agent uses its Q-prediction to score each possible action ai in s -LRB- t -RRB- with mi . b -RRB- The boltzmann distribution chooses actions randomly with the following probabilities . Prob -LRB- ai -RRB- = emi = T = X k emk = T The probability that action ai is selected increases with its score mi , but as T ! 1 these differences vanish , and as T ! the action with the highest score is chosen with probability almost 1 . 3 . The execution of action at results in a new state st +1 and a new spontaneous evaluation et . 4 . The prediction ^ Q is updated with the following learning rule ? ^ Q -LRB- st ; at -RRB- : = ? ? < = et + ? max at +1 ^ Q -LRB- st +1 ; at +1 -RRB- ? ^ Q -LRB- st ; at -RRB- > = -LRB- 5 -RRB- If the difference on the right hand is zero for all s ; a then the equation 4 also holds true . The learning rule is a kind of temporal difference learning rule investigated first by Samuel -LSB- 17 , 18 -RSB- and reformulated and studied intensively by Sutton -LSB- 21 , 22 -RSB- . 5 . Increase t , decrease T and goto 2 . 5 Also Werbos -LSB- 30 , 31 -RSB- has outlined learning algorithms in the framework of dynamic programming . 3 The Path Finding Domain In this section , we transfer the ideas of reinforcement learning to the path finding domain with continuous state spaces . This is not to show any implementation and experiments but to have a framework for illustrating a remarkable property of cumulative evaluation functions in this domain . In general , a path finding task can be formulated as follows : There are -LCB- a set S of states , -LCB- a nonempty set G ae S of goal states , -LCB- a set A of actions , -LCB- a transition function ffi : S ? A ! S that maps a state and an action to a successor state , -LCB- a spontaneous evaluation function e : S ? A ! IR that assigns real-valued costs to the execution of some action in some state . The costs are a kind of -LRB- negative -RRB- reinforcement or punishment . In this case one wishes to choose those actions with lead to minimal cumulative evaluation . This is implemented by taking the minimum over equations 2 , 4 , and 5 instead of the maximum . ffi and e may be partial so that setups can be covered where not all actions are executable in every state , for example if obstacles are in the way . The task consists in finding a sequence of actions that leads the agent from some given start state to a goal state . In addition , this sequence should be kept as cheap as possible by minimizing the cumulative evaluation function . 4 Path finding in continuous state spaces The framework above may apply analogously to state spaces which are subsets of continuous spaces , however the convergence theorem of Watkins -LRB- see section 2 -RRB- does not work here . We can show by illustration that nearly all tasks with forbidden regions -LRB- in our case implied by the obstacles -RRB- in state space have a discontinuous evaluation function . We illustrate this first by the simplest scenario we could think of . Let the agent be a robot arm with one revolute joint -LRB- fig. 1 -RRB- . Its workspace contains a goal and an obstacle . The states are the positions of the arm and are represented by pairs of angles -LRB- ` ge ; ` oe -RRB- ; ` oe < > 0 , thus containing information about the location of the goal and the obstacle relative to the end effector position . The pairs with ` oe = form the forbidden region here . Goal states are all states with j' gej < = o ; o > 0 . Actions are limited joint transitions a 2 -LSB- ? amax ; ? '' -RSB- -LSB- -LSB- '' ; amax -RSB- with amax < ss and < '' < = o , and ffi -LRB- -LRB- ` ge ; ` oe -RRB- ; a -RRB- = -LRB- ` ge + a ; ` oe + a -RRB- . The actions are required to be at least of magnitude '' , so the arm is forced to move at each step so long the goal is not reached . '' < = o guarantees that the goal state interval can not be jumped with a smallest step . The costs of an action in a state depend only on the action : e -LRB- s ; a -RRB- = jaj . To prevent collisions with the obstacle , an action a is disallowed iff one of the following conditions holds : -LCB- ` oe > 0 , a < 0 , and ` oe + a < = -LCB- ` oe < 0 , a > 0 , and ` oe + a > = This means that ffi and e are only partial . j ge obstacle goal oe j end effector obstacle goal costs j gej jgo end effector Fig. 1 . LEFT : There is no obstacle between end effector and goal , therefore the best way is the direct way . RIGHT : As soon as the obstacle is in the way between end effector and goal the best and only way is the circumvention . To be able to find a -LRB- nearly -RRB- optimal path to a goal state from every initial state that it is put into , the agent should learn a policy . This can be achieved with Q-Learning using a slight modification of the algorithm in section 2 : When a goal state is reached in step 3 , the arm is set to a new non-goal state after step 4 . This has the additional effect that becomes superfluous since the minimal costs are achieved by getting to a goal state with a finite number of actions , without changing the direction of movement . With the optimal policy , the Q - function is as follows -LRB- from now on , its second argument is always assumed to represent an allowed action -RRB- : Q ? -LRB- -LRB- ` ge ; ` oe -RRB- ; a -RRB- = ae ; if j' gej < = o jaj + maxb2A Q ? -LRB- -LRB- ` ge + a ; ` oe + a -RRB- ; b -RRB- ; otherwise -LRB- 6 -RRB- The actual values then are as follows : Q ? -LRB- -LRB- ` ge ; ` oe -RRB- ; a -RRB- = 8 > > > > > > > > < > > > > > > > > : ; j' gej < = o ` ge ? o ; ` ge ? o > > ` oe ? ` ge ? o ; ` oe > > ` ge + o ` ge ? o ; ` oe > ` ge ? o > ? ` ge ? o ; > ` ge + o > ` oe 2ss ? ` ge ? o ; ` ge ? o > = ` oe > 2ss + ` ge ? o ; > ` oe > = ` ge + o -LRB- 7 -RRB- Obviously , the values of Q ? are independent of the action . Figure 2 shows them graphically -LRB- to keep the graphics clear , we took o = here -RRB- and demonstrates the existence of discontinuities . gej oej p p - p - p 2p - 2p + - - gej gej gej gej gej gej - pi pi - pi pi Fig. 2 . LEFT : Values of Q ? for the one-jointed robot arm . RIGHT : Density plot of the left part of the figure . -LRB- dark = low costs , light = high costs -RRB- If there were no obstacle -LRB- the state representation would then consist only in ` ge -RRB- , Q ? would be a continuous function with a maximum at ` ge = ss . The obstacle introduces two discontinuities , one with regard to ` oe alone -LRB- due to the differences of positioning the obstacle on the shorter or the longer path between goal and agent -RRB- , one with regard to both state components simultaneously -LRB- due to the , in general , different lengths of the optimal paths starting from states immediately left and right of the obstacle -RRB- . If the obstacle position were not integrated in state representation , only the second discontinuity would remain . Further examples for discontinuous evaluation functions are shown in Figure 3 . In the left part of the figure , the discontinuity occurs if the horizontal position of the obstacle is integrated in the state description ; the optimal path and its costs change dramatically when the corridor becomes too narrow . For a two-joint arm with two obstacles in its workspace -LRB- right part of the figure -RRB- , there are discontinuities even if the position of the -LRB- point form -RRB- obstacles are not included in the state description -LRB- large cost differences between the second segment of the arm being right or left of obstacle1 , and between any segment being left or right of obstacle2 -RRB- . But it gets a lot more complex if they are included ; for example , the optimal path changes if obstacle2 is moved to the left until the arm can not pass obstacle1 any more by turning the first joint a bit to the right and turning the second one to the left . Thus it has to take a different , much more costly path to the goal . goal physical agent optimal d-e goal physical agent optimal d path path a -RRB- b -RRB- obstacle goal 1j j2 endeffector obstacle1 2 Fig. 3 . LEFT : A robot has to decide whether to go to the corridor or to circumvent the big obstacle . Here the discontinuity is at the least d at which the robot can pass the way . RIGHT : A two-jointed robot arm with two obstacles has a very complex structure of discontinuities in its cumulative evaluation function . In all cases , the cumulative evaluation function is discontinuous in the states . Since all such functions are defined at least on the state space , the problem of discontinuities through forbidden regions applies to every method dealing with delayed reinforcement , not only to Q-Learning . 5 The Problem of Learning Evaluation Functions in Continuous Domains We have shown in the last section that optimal cumulative evaluation function may be discontinuous in the presence of forbidden regions in state space . Furthermore in the case of continuous state spaces , lookup tables are not very useful . They might store all evaluations to some degree of precision , but they can not generalize between them . It is obvious that practical applications can require a degree of discretization resulting in impractical sizes of the lookup table . These considerations have led many researchers to use compact representations -LSB- 12 , 25 -RSB- . They allow the handling of continuous-valued inputs quite naturally . Examples of such compact representations are parameterized structures -LSB- 24 , 25 -RSB- like a backpropagation net -LSB- 16 -RSB- , a polynomial data fitter or other neural networks . They are able to generalize or more technically interpolate among states , which means that the agent needs only to explore a fraction of states and infers the evaluation of unseen states from experience . There are some general problems known with the use of compact representations , which also apply to learning evaluation functions : 1 . The function approximator might be theoretically incapable of representing the optimal evaluation function to a sufficient degree . 2 . The gradient descent method usually applied to train the function approximator might get stuck in a local optimum , even if the above case does not apply . Additional problems arise in the context of the temporal difference updating method . There is currently no convergence proof available for continuous input spaces . Thus it is unknown wether training with temporal difference in the continuous case converges at least to a local optimum . We expect this situation to get worse a lot in the presence of discontinuous cumulative evaluation functions . Apart from these problems , function approximators such as backpropagation nets have severe difficulties to simply represent discontinuous functions . Backpropagation nets represent inherently continuous functions and an approximation of a discontinuous functions would require the weights to grow arbitrarily large -LSB- 13 -RSB- . In such a case , the weights would be subject to oscillations . The explanation is that large weights imply a steep and sharp error surface and therefore large error derivatives . With a fixed learning rate , an overshooting -LSB- 9 , 13 -RSB- of local minima in at least some dimensions happens . Anyway a limited number of hidden units allows the approximation of any function only to a certain precision . 6 Conclusion Currently we see two general kinds of approaches to circumvent the problem of discontinuity , problem dependent and independent ones . The two problem independent approaches are -LCB- to use discontinuous function approximators . Known discontinuous function approximators are competitive mechanisms , which assigns different single function approximators to different input regions -LSB- 8 , 20 , 19 -RSB- . Unclear is whether they could detect regions of discontinuities in input space . -LCB- to design relative cumulative evaluation functions , which has been already mentioned by Tesauro -LSB- 25 -RSB- for other purposes . Approaches using domain knowledge include the proper choice of coding schemes and representation issues . At least in simpler cases , it seems feasible to encode the inputs in a way that make discontinuities vanish . With a reformulation of the task representation in terms of a hierarchy and a structure in state and action spaces , the discontinuities might play a less significant role . Another point is that in the presence of sensor noise or actuator nondeterminism uncertainty arises . This uncertainty leads to estimates of evaluations that mediate between neighbouring states , which flaten the discontinuities . The more uncertainty exists the smoother the evaluation function surface will be . From the outlook presented above , we are quite optimistic . Nevertheless it is obvious that discontinuities of evaluation functions are a limitation to the applicability of reinforcement learning procedures to domains characterized above . The can prevent a solution from being optimal . In this context it is our belief that the experiment of Lin -LSB- 12 -RSB- has discontinuities and thus failed to show the desired performance . But it is beyond the scope of this paper to prove this . The message should be instead that one has to be aware of possible discontinuities and take them into account . Acknowledgements I gratefully acknowledge many discussions with Frank Weber and all other members of our research group . Discussions with Long-Ji Lin and Sebastian Thrun at CMU contributed to the conclusion of this work . References 1 . C. W. Anderson . Learning and problem solving with multilayer connectionist systems . Technical Report COINS TR 86-50 , Dept. of Computer and Information Science , University of Massachusetts , Amherst , MA , 1986 . 2 . A. G. Barto . Simulation Experiments with Goal-Seeking Adaptive Elements . COINS , Amherst , Massachusetts , 1984 . AFWAL-TR-84-1022 . 3 . A. G. Barto , S. J. Bradtke , and S. P. Singh . Real-time learning and control using asynchronous dynamic programming . Technical report , University of Massachusetts , Departement of Computer Science , Amherst MA 01003 , August 1991 . 4 . A. G. Barto , R. S. Sutton , and C. J. C. H. Watkins . Learning and sequential decision making . Technical Report COINS 89-95 , Department of Computer Science , University of Massachusetts , MA , September 1989 . 5 . A. G. Barto , R. S. Sutton , and C. J. C. H. Watkins . Learning and sequential decision making . In M. Gabriel and J. W. Moore , editors , Learning and Computational Neuroscience , pages 539 -LCB- 602 . MIT Press , Massachusetts , 1990 . 6 . R. E. Bellman . Dynamic Programming . Princeton University Press , Princeton , New Jersey , 1957 . 7 . P. Dayan . The convergence of TD -LRB->=-RRB- for general > = . Machine Learning Journal , 8 -LRB- 3/4 -RRB- , May 1992 . Special Issue on Reinforcement Learning . 8 . D. Fox , V. Heinze , K. Moller , and S. B. Thrun . Learning by error driven decomposition . In Proc . of NeuroNimes , France , 1991 . 9 . G. E. Hinton . Connectionist learning procedures . Artificial Intelligence , 40:185 -LCB- 234 , 1989 . 10 . R. E. Korf . Real-time heuristic search : New results . In AAAI-88 , pages 139 -LCB- 143 , 1988 . 11 . P. R. Kumar . A survery of some results in stochastic adaptive control . SIAM Journal of Control and Optimization , 23:329 -LCB- 380 , 1985 . 12 . L.-J . Lin . Self-improving reactive agents based on reinforcement learning , planning and teaching . Machine Learning Journal , 8 -LRB- 3/4 -RRB- , 1992 . Special Issue on Reinforcement Learning . 13 . A. Linden . Untersuchung von Backpropagation in konnektionistischen Systemen . Diplomarbeit , Universitat Bonn , Informatik-Institutsbericht Nr . 80 , 1990 . 14 . J. del R. Mill?an and C. Torras . A reinforcement connectionist approach to robot path finding in non-maze-like environments . Machine Learning Journal , 8 -LRB- 3/4 -RRB- , May 1992 . Special Issue on Reinforcement Learning . 15 . M. Minsky . Steps toward artificial intelligence . In E.A. Feigenbaum and J. Feldman , editors , Computers and Thought , pages 406 -LCB- 450 . McGraw-Hill , 1961 . 16 . D. E. Rumelhart , G. E. Hinton , and R. J. Williams . Learning internal representations by error propagation . In D. E. Rumelhart and J. L. McClelland , editors , Parallel Distributed Processing . Vol . I + II . MIT Press , 1986 . 17 . A. L. Samuel . Some studies in machine learning using the game of checkers . IBM Journal on Research and Development , pages 210 -LCB- 229 , 1959 . Reprinted in E. ~ A. Feigenbaum and J. Feldman -LRB- Eds . -RRB- 1963 , Computers and Thought , McGrawHill , New York . 18 . A. L. Samuel . Some studies in machine learning using the game of checkers . ii | recent progress . IBM Journal on Research and Development , pages 601 -LCB- 617 , 1967 . 19 . F. J. ? Smieja and H. Muhlenbein . Reflective modular neural network systems . Technical Report 633 , GMD , Sankt Augustin , Germany , February 1992 . 20 . F.J. ? Smieja . Multiple network systems -LRB- MINOS -RRB- modules : Task division and module discrimination . In Proceedings of the 8th AISB conference on Artificial Intelligence , Leeds , 16 -LCB- 19 April , 1991 , 1991 . 21 . R. S. Sutton . Temporal Credit Assignment in Reinforcement Learning . PhD thesis , University of Massachusetts , 1984 . 22 . R. S. Sutton . Learning to predict by the methods of temporal differences . Machine Learning , 3:9 -LCB- 44 , 1988 . 23 . R. S. Sutton . Integrated architectures for learning , planning , and reacting based on approximating dynamic programming . To appear in : Proceedings of the Seventh International Conference on Machine Learning , June 1990 , 1990 . 24 . R. S. Sutton , A. G. Barto , and R. J. Williams . Reinforcement Learning is Direct Adaptive Optimal Control . In Proceedings of the 1991 American Control Conference , 1991 . 25 . G. Tesauro . Practical issues in temporal difference learning . Machine Learning Journal , 8 -LRB- 3/4 -RRB- , 1992 . Special Issue on Reinforcement Learning . 26 . S. B. Thrun . Efficient exploration in reinforcement learning . Technical Report CMU-CS-92-102 , Carnegie Mellon University , Pittsburgh , Pennsylvania , January 1992 . 27 . C. J. C. H. Watkins . Learning from Delayed Rewards . PhD thesis , King 's College , 1989 . 28 . C. J. C. H. Watkins and P. Dayan . Q-learning . Machine Learning Journal , 8 -LRB- 3/4 -RRB- , May 1992 . Special Issue on Reinforcement Learning . 29 . P. Werbos . Consistency of HDP applied to a simple reinforcement learning problem . Neural Networks , 3:179 -LCB- 189 , 1990 . 30 . P. Werbos and J. Titus . An empirical test of new forecasting methods derived from a theory of intelligence : The predicition of conflict in latin america . IEEE Transactions on Systems , Man , and Cybernetics , SMC-8 :657 -LCB- 666 , 1978 . 31 . Paul J. Werbos . Backpropagation and neurocontrol : A review and prospectus . In Proceedings of IJCNN89 Washington , pages I 209 -LCB- 216 , 1989 . 32 . S. D. Whitehead . A study of cooperative mechanisms for faster reinforcement learning . Technical Report 365 , University of Rochester , Computer Science Department , Rochester , NY , March 1991 . 33 . S. D. Whitehead and D. H. Ballard . Active perception and reinforcement learning . Neural Computation , 2:409 -LCB- 419 , 1990 . 34 . R. J. Williams . Reinforcement-learning connectionist systems . Technical Report NU-CCS-87-3 , College of Computer Science , Northeastern University , Boston , 1987 . This article was processed using the LaTEX macro package with LLNCS style

O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O B I I O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I I O B I O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

backpropagation nets
continuous state space
evaluation functions
machine learning
obstacles
path finding
reinforcement learning
robotics
temporal difference learning