Above are papers related to the following paper: [Paper_ID] Paper #28287. 	[Title] Robot Shaping: Developing Situated Agents through Learning [Abstract] Learning plays a vital role in the development of situated agents. In this paper, we explore the use of reinforcement learning to "shape" a robot to perform a predefined target behavior. We connect both simulated and real robots to A LECSYS, a parallel implementation of a learning classifier system with an extended genetic algorithm. After classifying different kinds of Animat-like behaviors, we explore the effects on learning of different types of agent's architecture (monolithic, flat and hierarchical) and of training strategies. In particular, hierarchical architecture requires the agent to learn how to coordinate basic learned responses. We show that the best results are achieved when both the agent's architecture and the training strategy match the structure of the behavior pattern to be learned. We report the results of a number of experiments carried out both in simulated and in real environments, and show that the results of simulations carry smoothly to real robots. While most of our experiments deal with simple reactive behavior, in one of them we demonstrate the use of a simple and general memory mechanism. As a whole, our experimental activity demonstrates that classifier systems with genetic algorithms can be practically employed to develop autonomous agents. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6151. 	[Title] Strategy Learning with Multilayer Connectionist Representations 1 [Abstract] Results are presented that demonstrate the learning and fine-tuning of search strategies using connectionist mechanisms. Previous studies of strategy learning within the symbolic, production-rule formalism have not addressed fine-tuning behavior. Here a two-layer connectionist system is presented that develops its search from a weak to a task-specific strategy and fine-tunes its performance. The system is applied to a simulated, real-time, balance-control task. We compare the performance of one-layer and two-layer networks, showing that the ability of the two-layer network to discover new features and thus enhance the original representation is critical to solving the balancing task. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6155. 	[Title] On the Computational Economics of Reinforcement Learning [Abstract] Following terminology used in adaptive control, we distinguish between indirect learning methods, which learn explicit models of the dynamic structure of the system to be controlled, and direct learning methods, which do not. We compare an existing indirect method, which uses a conventional dynamic programming algorithm, with a closely related direct reinforcement learning method by applying both methods to an infinite horizon Markov decision problem with unknown state-transition probabilities. The simulations show that although the direct method requires much less space and dramatically less computation per control action, its learning ability in this task is superior to, or compares favorably with, that of the more complex indirect method. Although these results do not address how the methods' performances compare as problems become more difficult, they suggest that given a fixed amount of computational power available per control action, it may be better to use a direct reinforcement learning method augmented with indirect techniques than to devote all available resources to a computation-ally costly indirect method. Comprehensive answers to the questions raised by this study depend on many factors making up the eco nomic context of the computation. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1111052. 	[Title] Robust Value Function Approximation by Working Backwards Computing an accurate value function is the key [Abstract] In this paper, we examine the intuition that TD() is meant to operate by approximating asynchronous value iteration. We note that on the important class of discrete acyclic stochastic tasks, value iteration is inefficient compared with the DAG-SP algorithm, which essentially performs only one sweep instead of many by working backwards from the goal. The question we address in this paper is whether there is an analogous algorithm that can be used in large stochastic state spaces requiring function approximation. We present such an algorithm, analyze it, and give comparative results to TD on several domains. the state). Using VI to solve MDPs belonging to either of these special classes can be quite inefficient, since VI performs backups over the entire space, whereas the only backups useful for improving V fl are those on the "frontier" between already-correct and not-yet-correct V fl values. In fact, there are classical algorithms for both problem classes which compute V fl more efficiently by explicitly working backwards: for the deterministic class, Dijkstra's shortest-path algorithm; and for the acyclic class, Directed-Acyclic-Graph-Shortest-Paths (DAG-SP) [6]. 1 DAG-SP first topologically sorts the MDP, producing a linear ordering of the states in which every state x precedes all states reachable from x. Then, it runs through that list in reverse, performing one backup per state. Worst-case bounds for VI, Dijkstra, and DAG-SP in deterministic domains with X states and A actions/state are 1 Although [6] presents DAG-SP only for deterministic acyclic problems, it applies straightforwardly to the Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1114125. 	[Title] H-learning: A Reinforcement Learning Method to Optimize Undiscounted Average Reward [Abstract] In this paper, we introduce a model-based reinforcement learning method called H-learning, which optimizes undiscounted average reward. We compare it with three other reinforcement learning methods in the domain of scheduling Automatic Guided Vehicles, transportation robots used in modern manufacturing plants and facilities. The four methods differ along two dimensions. They are either model-based or model-free, and optimize discounted total reward or undiscounted average reward. Our experimental results indicate that H-learning is more robust with respect to changes in the domain parameters, and in many cases, converges in fewer steps to better average reward per time step than all the other methods. An added advantage is that unlike the other methods it does not have any parameters to tune. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #755217. 	[Title] Active Gesture Recognition using Partially Observable Markov Decision Processes [Abstract] M.I.T Media Laboratory Perceptual Computing Section Technical Report No. 367 Appeared 13th IEEE Intl. Conference on Pattern Recognition (ICPR '96), Vienna, Austria. Abstract We present a foveated gesture recognition system that guides an active camera to foveate salient features based on a reinforcement learning paradigm. Using vision routines previously implemented for an interactive environment, we determine the spatial location of salient body parts of a user and guide an active camera to obtain images of gestures or expressions. A hidden-state reinforcement learning paradigm based on the Partially Observable Markov Decision Process (POMDP) is used to implement this visual attention. The attention module selects targets to foveate based on the goal of successful recognition, and uses a new multiple-model Q-learning formulation. Given a set of target and distractor gestures, our system can learn where to foveate to maximally discriminate a particular gesture. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #8213. 	[Title] Robust Reinforcement Learning in Motion Planning [Abstract] While exploring to find better solutions, an agent performing online reinforcement learning (RL) can perform worse than is acceptable. In some cases, exploration might have unsafe, or even catastrophic, results, often modeled in terms of reaching `failure' states of the agent's environment. This paper presents a method that uses domain knowledge to reduce the number of failures during exploration. This method formulates the set of actions from which the RL agent composes a control policy to ensure that exploration is conducted in a policy space that excludes most of the unacceptable policies. The resulting action set has a more abstract relationship to the task being solved than is common in many applications of RL. Although the cost of this added safety is that learning may result in a suboptimal solution, we argue that this is an appropriate tradeoff in many problems. We illustrate this method in the domain of motion planning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6170. 	[Title] Improving Policies without Measuring Merits [Abstract] Performing policy iteration in dynamic programming should only require knowledge of relative rather than absolute measures of the utility of actions what Baird (1993) calls the advantages of actions at states. Nevertheless, existing methods in dynamic programming (including Baird's) compute some form of absolute utility function. For smooth problems, advantages satisfy two differential consistency conditions (including the requirement that they be free of curl), and we show that enforcing these can lead to appropriate policy improvement solely in terms of advantages. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #128540. 	[Title] Adapting Control Strategies for Situated Autonomous Agents [Abstract] This paper studies how to balance evolutionary design and human expertise in order to best design situated autonomous agents which can learn specific tasks. A genetic algorithm designs control circuits to learn simple behaviors, and given control strategies for simple behaviors, the genetic algorithm designs a combinational circuit that switches between these simple behaviors to perform a navigation task. Keywords: Genetic Algorithms, Computational Design, Autonomous Agents, Robotics. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #35. 	[Title] 4 Implementing Application Specific Routines  Genetic algorithms in search, optimization, and machine learning. Reading, MA: Addison-Wesley. [Abstract] To implement a specific application, you should only have to change the file app.c. Section 2 describes the routines in app.c in detail. If you use additional variables for your specific problem, the easiest method of making them available to other program units is to declare them in sga.h and external.h. However, take care that you do not redeclare existing variables. Two example applications files are included in the SGA-C distribution. The file app1.c performs the simple example problem included with the Pascal version; finding the maximum of x 10 , where x is an integer interpretation of a chromosome. A slightly more complex application is include in app2.c. This application illustrates two features that have been added to SGA-C. The first of these is the ithruj2int function, which converts bits i through j in a chromosome to an integer. The second new feature is the utility pointer that is associated with each population member. The example application interprets each chromosome as a set of concatenated integers in binary form. The lengths of these integer fields is determined by the user-specified value of field size, which is read in by the function app data(). The field size must be less than the smallest of the chromosome length and the length of an unsigned integer. An integer array for storing the interpreted form of each chromosome is dynamically allocated and assigned to the chromosome's utility pointer in app malloc(). The ithruj2int routine (see utility.c) is used to translate each chromosome into its associated vector. The fitness for each chromosome is simply the sum of the squares of these integers. This example application will function for any chromosome length. SGA-C is intended to be a simple program for first-time GA experimentation. It is not intended to be definitive in terms of its efficiency or the grace of its implementation. The authors are interested in the comments, criticisms, and bug reports from SGA-C users, so that the code can be refined for easier use in subsequent versions. Please email your comments to rob@galab2.mh.ua.edu, or write to TCGA: The authors gratefully acknowledge support provided by NASA under Grant NGT-50224 and support provided by the National Science Foundation under Grant CTS-8451610. We also thank Hillol Kargupta for donating his tournament selection implementation. Booker, L. B. (1982). Intelligent behavior as an adaptation to the task environment (Doctoral dissertation, Technical Report No. 243. Ann Arbor: University of Michigan, Logic of Computers Group). Dissertations Abstracts International, 43(2), 469B. (University Microfilms No. 8214966) Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1118245. 	[Title] Learning sorting and decision trees with POMDPs [Abstract] pomdps are general models of sequential decisions in which both actions and observations can be probabilistic. Many problems of interest can be formulated as pomdps, yet the use of pomdps has been limited by the lack of effective algorithms. Recently this has started to change and a number of problems such as robot navigation and planning are beginning to be formulated and solved as pomdps. The advantage of the pomdp approach is its clean semantics and its ability to produce principled solutions that integrate physical and information gathering actions. In this paper we pursue this approach in the context of two learning tasks: learning to sort a vector of numbers and learning decision trees from data. Both problems are formulated as pomdps and solved by a general pomdp algorithm. The main lessons and results are that 1) the use of suitable heuristics and representations allows for the solution of sorting and classification pomdps of non-trivial sizes, 2) the quality of the resulting solutions are competitive with the best algorithms, and 3) problematic aspects in decision tree learning such as test and mis-classification costs, noisy tests, and missing values are naturally accommodated. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6184. 	[Title] Forward models: Supervised learning with a distal teacher [Abstract] Internal models of the environment have an important role to play in adaptive systems in general and are of particular importance for the supervised learning paradigm. In this paper we demonstrate that certain classical problems associated with the notion of the "teacher" in supervised learning can be solved by judicious use of learned internal models as components of the adaptive system. In particular, we show how supervised learning algorithms can be utilized in cases in which an unknown dynamical system intervenes between actions and desired outcomes. Our approach applies to any supervised learning algorithm that is capable of learning in multi-layer networks. *This paper is a revised version of MIT Center for Cognitive Science Occasional Paper #40. We wish to thank Michael Mozer, Andrew Barto, Robert Jacobs, Eric Loeb, and James McClelland for helpful comments on the manuscript. This project was supported in part by BRSG 2 S07 RR07047-23 awarded by the Biomedical Research Support Grant Program, Division of Research Resources, National Institutes of Health, by a grant from ATR Auditory and Visual Perception Research Laboratories, by a grant from Siemens Corporation, by a grant from the Human Frontier Science Program, and by grant N00014-90-J-1942 awarded by the Office of Naval Research. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #648232. 	[Title] VECTOR ASSOCIATIVE MAPS: UNSUPERVISED REAL-TIME ERROR-BASED LEARNING AND CONTROL OF MOVEMENT TRAJECTORIES [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6196. 	[Title] The Parti-game Algorithm for Variable Resolution Reinforcement Learning in Multidimensional State-spaces [Abstract] Parti-game is a new algorithm for learning feasible trajectories to goal regions in high dimensional continuous state-spaces. In high dimensions it is essential that learning does not plan uniformly over a state-space. Parti-game maintains a decision-tree partitioning of state-space and applies techniques from game-theory and computational geometry to efficiently and adaptively concentrate high resolution only on critical areas. The current version of the algorithm is designed to find feasible paths or trajectories to goal regions in high dimensional spaces. Future versions will be designed to find a solution that optimizes a real-valued criterion. Many simulated problems have been tested, ranging from two-dimensional to nine-dimensional state-spaces, including mazes, path planning, non-linear dynamics, and planar snake robots in restricted spaces. In all cases, a good solution is found in less than ten trials and a few minutes. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #188471. 	[Title] Adaptive state space quantisation for reinforcement learning of collision-free navigation [Abstract] The paper describes a self-learning control system for a mobile robot. Based on sensor information the control system has to provide a steering signal in such a way that collisions are avoided. Since in our case no `examples' are available, the system learns on the basis of an external reinforcement signal which is negative in case of a collision and zero otherwise. Rules from Temporal Difference learning are used to find the correct mapping between the (discrete) sensor input space and the steering signal. We describe the algorithm for learning the correct mapping from the input (state) vector to the output (steering) signal, and the algorithm which is used for a discrete coding of the input state space. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #13885. 	[Title] Learning Controllers for Industrial Robots [Abstract] One of the most significant cost factors in robotics applications is the design and development of real-time robot control software. Control theory helps when linear controllers have to be developed, but it doesn't sufficiently support the generation of non-linear controllers, although in many cases (such as in compliance control), nonlinear control is essential for achieving high performance. This paper discusses how Machine Learning has been applied to the design of (non-)linear controllers. Several alternative function approximators, including Multilayer Perceptrons (MLP), Radial Basis Function Networks (RBFNs), and Fuzzy Controllers are analyzed and compared, leading to the definition of two major families: Open Field Function Function Approximators and Locally Receptive Field Function Approximators. It is shown that RBFNs and Fuzzy Controllers bear strong similarities, and that both have a symbolic interpretation. This characteristics allows for applying both symbolic and statistic learning algorithms to synthesize the network layout from a set of examples and, possibly, some background knowledge. Three integrated learning algorithms, two of which are original, are described and evaluated on experimental test cases. The first test case is provided by a robot KUKA IR-361 engaged into the "peg-into-hole" task, whereas the second is represented by a classical prediction task on the Mackey-Glass time series. From the experimental comparison, it appears that both Fuzzy Controllers and RBFNs synthesised from examples are excellent approximators, and that, in practice, they can be even more accurate than MLPs. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6210. 	[Title] Reinforcement Learning with Soft State Aggregation [Abstract] It is widely accepted that the use of more compact representations than lookup tables is crucial to scaling reinforcement learning (RL) algorithms to real-world problems. Unfortunately almost all of the theory of reinforcement learning assumes lookup table representations. In this paper we address the pressing issue of combining function approximation and RL, and present 1) a function approx-imator based on a simple extension to state aggregation (a commonly used form of compact representation), namely soft state aggregation, 2) a theory of convergence for RL with arbitrary, but fixed, soft state aggregation, 3) a novel intuitive understanding of the effect of state aggregation on online RL, and 4) a new heuristic adaptive state aggregation algorithm that finds improved compact representations by exploiting the non-discrete nature of soft state aggregation. Preliminary empirical results are also presented. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28227. 	[Title] Reinforcement Learning Methods for Continuous-Time Markov Decision Problems [Abstract] Semi-Markov Decision Problems are continuous time generalizations of discrete time Markov Decision Problems. A number of reinforcement learning algorithms have been developed recently for the solution of Markov Decision Problems, based on the ideas of asynchronous dynamic programming and stochastic approximation. Among these are TD(), Q-learning, and Real-time Dynamic Programming. After reviewing semi-Markov Decision Problems and Bellman's optimality equation in that context, we propose algorithms similar to those named above, adapted to the solution of semi-Markov Decision Problems. We demonstrate these algorithms by applying them to the problem of determining the optimal control for a simple queueing system. We conclude with a discussion of circumstances under which these algorithms may be usefully ap plied. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6213. 	[Title] Machine Learning Learning to Predict by the Methods of Temporal Differences Keywords: Incremental learning, prediction, [Abstract] This article introduces a class of incremental learning procedures specialized for prediction|that is, for using past experience with an incompletely known system to predict its future behavior. Whereas conventional prediction-learning methods assign credit by means of the difference between predicted and actual outcomes, the new methods assign credit by means of the difference between temporally successive predictions. Although such temporal-difference methods have been used in Samuel's checker player, Holland's bucket brigade, and the author's Adaptive Heuristic Critic, they have remained poorly understood. Here we prove their convergence and optimality for special cases and relate them to supervised-learning methods. For most real-world prediction problems, temporal-difference methods require less memory and less peak computation than conventional methods; and they produce more accurate predictions. We argue that most problems to which supervised learning is currently applied are really prediction problems of the sort to which temporal-difference methods can be applied to advantage. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6214. 	[Title] Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming [Abstract] This paper extends previous work with Dyna, a class of architectures for intelligent systems based on approximating dynamic programming methods. Dyna architectures integrate trial-and-error (reinforcement) learning and execution-time planning into a single process operating alternately on the world and on a learned model of the world. In this paper, I present and show results for two Dyna architectures. The Dyna-PI architecture is based on dynamic programming's policy iteration method and can be related to existing AI ideas such as evaluation functions and universal plans (reactive systems). Using a navigation task, results are shown for a simple Dyna-PI system that simultaneously learns by trial and error, learns a world model, and plans optimal routes using the evolving world model. The Dyna-Q architecture is based on Watkins's Q-learning, a new kind of reinforcement learning. Dyna-Q uses a less familiar set of data structures than does Dyna-PI, but is arguably simpler to implement and use. We show that Dyna-Q architectures are easy to adapt for use in changing environments. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1130567. 	[Title] NEUROCONTROL BY REINFORCEMENT LEARNING [Abstract] Reinforcement learning (RL) is a model-free tuning and adaptation method for control of dynamic systems. Contrary to supervised learning, based usually on gradient descent techniques, RL does not require any model or sensitivity function of the process. Hence, RL can be applied to systems that are poorly understood, uncertain, nonlinear or for other reasons untractable with conventional methods. In reinforcement learning, the overall controller performance is evaluated by a scalar measure, called reinforcement. Depending on the type of the control task, reinforcement may represent an evaluation of the most recent control action or, more often, of an entire sequence of past control moves. In the latter case, the RL system learns how to predict the outcome of each individual control action. This prediction is then used to adjust the parameters of the controller. The mathematical background of RL is closely related to optimal control and dynamic programming. This paper gives a comprehensive overview of the RL methods and presents an application to the attitude control of a satellite. Some well known applications from the literature are reviewed as well. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6220. 	[Title] Issues in Using Function Approximation for Reinforcement Learning [Abstract] Reinforcement learning techniques address the problem of learning to select actions in unknown, dynamic environments. It is widely acknowledged that to be of use in complex domains, reinforcement learning techniques must be combined with generalizing function approximation methods such as artificial neural networks. Little, however, is understood about the theoretical properties of such combinations, and many researchers have encountered failures in practice. In this paper we identify a prime source of such failuresnamely, a systematic overestimation of utility values. Using Watkins' Q-Learning [18] as an example, we give a theoretical account of the phenomenon, deriving conditions under which one may expected it to cause learning to fail. Employing some of the most popular function approximators, we present experimental results which support the theoretical findings. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #628815. 	[Title] Optimal Attitude Control of Satellites by Artificial Neural Networks: a Pilot Study [Abstract] A pilot study is described on the practical application of artificial neural networks. The limit cycle of the attitude control of a satellite is selected as the test case. One of the sources of the limit cycle is a position dependent error in the observed attitude. A Reinforcement Learning method is selected, which is able to adapt a controller such that a cost function is optimised. An estimate of the cost function is learned by a neural `critic'. In our approach, the estimated cost function is directly represented as a function of the parameters of a linear controller. The critic is implemented as a CMAC network. Results from simulations show that the method is able to find optimal parameters without unstable behaviour. In particular in the case of large discontinuities in the attitude measurements, the method shows a clear improvement compared to the conventional approach: the RMS attitude error decreases approximately 30%. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #38480. 	[Title] Learning to Achieve Goals [Abstract] Temporal difference methods solve the temporal credit assignment problem for reinforcement learning. An important subproblem of general reinforcement learning is learning to achieve dynamic goals. Although existing temporal difference methods, such as Q learning, can be applied to this problem, they do not take advantage of its special structure. This paper presents the DG-learning algorithm, which learns efficiently to achieve dynamically changing goals and exhibits good knowledge transfer between goals. In addition, this paper shows how traditional relaxation techniques can be applied to the problem. Finally, experimental results are given that demonstrate the superiority of DG learning over Q learning in a moderately large, synthetic, non-deterministic domain. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #13917. 	[Title] Designing Neural Networks for Adaptive Control [Abstract] This paper discusses the design of neural networks to solve specific problems of adaptive control. In particular, it investigates the influence of typical problems arising in real-world control tasks as well as techniques for their solution that exist in the framework of neurocontrol. Based on this investigation, a systematic design method is developed. The method is exemplified for the development of an adaptive force controller for a robot manipulator. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1104999. 	[Title] EVOLVING ROBOT BEHAVIORS [Abstract] This paper discusses the use of evolutionary computation to evolve behaviors that exhibit emergent intelligent behavior. Genetic algorithms are used to learn navigation and collision avoidance behaviors for robots. The learning is performed under simulation, and the resulting behaviors are then used to control the actual robot. Some of the emergent behavior is described in detail. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1130600. 	[Title] Adaptive state space quantisation: adding and removing neurons [Abstract] This paper describes a self-learning control system for a mobile robot. Based on local sensor data, a robot is taught to avoid collisions with obstacles. The only feedback to the control system is a binary-valued external reinforcement signal, which indicates whether or not a collision has occured. A reinforcement learning scheme is used to find a correct mapping from input (sensor) space to output (steering signal) space. An adaptive quantisation scheme is introduced, through which the discrete division of input space is built up from scratch by the system itself. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #114. 	[Title] Learning to Act using Real-Time Dynamic Programming [Abstract] fl The authors thank Rich Yee, Vijay Gullapalli, Brian Pinette, and Jonathan Bachrach for helping to clarify the relationships between heuristic search and control. We thank Rich Sutton, Chris Watkins, Paul Werbos, and Ron Williams for sharing their fundamental insights into this subject through numerous discussions, and we further thank Rich Sutton for first making us aware of Korf's research and for his very thoughtful comments on the manuscript. We are very grateful to Dimitri Bertsekas and Steven Sullivan for independently pointing out an error in an earlier version of this article. Finally, we thank Harry Klopf, whose insight and persistence encouraged our interest in this class of learning problems. This research was supported by grants to A.G. Barto from the National Science Foundation (ECS-8912623 and ECS-9214866) and the Air Force Office of Scientific Research, Bolling AFB (AFOSR-89-0526). Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1106547. 	[Title] Learning to Race: Experiments with a Simulated Race Car [Abstract] We have implemented a reinforcement learning architecture as the reactive component of a two layer control system for a simulated race car. We have found that separating the layers has expedited gradually improving competition and mult-agent interaction. We ran experiments to test the tuning, decomposition and coordination of the low level behaviors. We then extended our control system to allow passing of other cars and tested its ability to avoid collisions. The best design used reinforcement learning with separate networks for each behavior, coarse coded input and a simple rule based coordination mechanism. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28278. 	[Title] Chapter 1 Reinforcement Learning for Planning and Control [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1118332. 	[Title] June 1994 T o app ear in Neural Computation A Coun terexample to T emp [Abstract] Sutton's TD( ) metho d aims to provide a represen tation of the cost function in an absorbing Mark ov chain with transition costs. A simple example is given where the represen tation obtained dep ends on . For = 1 the represen tation is optimal with resp ect to a least squares error criterion, but as decreases towards 0 the represen tation becomes progressiv ely worse and, in some cases, very poor. The example suggests a need to understand better the circumstances under which TD(0) and Q-learning obtain satisfactory neural net work-based compact represen tations of the cost function. A variation of TD(0) is also prop osed, which performs b etter on the example. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #128. 	[Title] Reinforcement Learning Algorithms for Average-Payoff Markovian Decision Processes [Abstract] Reinforcement learning (RL) has become a central paradigm for solving learning-control problems in robotics and artificial intelligence. RL researchers have focussed almost exclusively on problems where the controller has to maximize the discounted sum of payoffs. However, as emphasized by Schwartz (1993), in many problems, e.g., those for which the optimal behavior is a limit cycle, it is more natural and com-putationally advantageous to formulate tasks so that the controller's objective is to maximize the average payoff received per time step. In this paper I derive new average-payoff RL algorithms as stochastic approximation methods for solving the system of equations associated with the policy evaluation and optimal control questions in average-payoff RL tasks. These algorithms are analogous to the popular TD and Q-learning algorithms already developed for the discounted-payoff case. One of the algorithms derived here is a significant variation of Schwartz's R-learning algorithm. Preliminary empirical results are presented to validate these new algorithms. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #130. 	[Title] Scaling Up Average Reward Reinforcement Learning by Approximating the Domain Models and the Value Function [Abstract] Almost all the work in Average-reward Re- inforcement Learning (ARL) so far has focused on table-based methods which do not scale to domains with large state spaces. In this paper, we propose two extensions to a model-based ARL method called H-learning to address the scale-up problem. We extend H-learning to learn action models and reward functions in the form of Bayesian networks, and approximate its value function using local linear regression. We test our algorithms on several scheduling tasks for a simulated Automatic Guided Vehicle (AGV) and show that they are effective in significantly reducing the space requirement of H-learning and making it converge faster. To the best of our knowledge, our results are the first in apply Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28290. 	[Title] Genetics-based Machine Learning and Behaviour Based Robotics: A New Synthesis complexity grows, the learning task [Abstract] difficult. We face this problem using an architecture based on learning classifier systems and on the description of the learning technique used and of the organizational structure proposed, we present experiments that show how behaviour acquisition can be achieved. Our simulated robot learns to structural properties of animal behavioural organization, as proposed by ethologists. After a Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1113739. 	[Title] Decision Tree Function Approximation in Reinforcement Learning [Abstract] We present a decision tree based approach to function approximation in reinforcement learning. We compare our approach with table lookup and a neural network function approximator on three problems: the well known mountain car and pole balance problems as well as a simulated automobile race car. We find that the decision tree can provide better learning performance than the neural network function approximation and can solve large problems that are infeasible using table lookup. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1126029. 	[Title] Exploration in Active Learning [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #782486. 	[Title] Living in a partially structured environment: How to bypass the limitations of classical reinforcement techniques [Abstract] In this paper, we propose an unsupervised neural network allowing a robot to learn sensori-motor associations with a delayed reward. The robot task is to learn the "meaning" of pictograms in order to "survive" in a maze. First, we introduce a new neural conditioning rule (PCR: Probabilistic Conditioning Rule) allowing to test hypotheses (associations between visual categories and movements) during a given time span. Second, we describe a real maze experiment with our mobile robot. We propose a neural architecture to solve this problem and we discuss the difficulty to build visual categories dynamically while associating them to movements. Third, we propose to use our algorithm on a simulation in order to test it exhaustively. We give the results for different kind of mazes and we compare our system to an adapted version of the Q-learning algorithm. Finally, we conclude by showing the limitations of approaches that do not take into account the intrinsic complexity of a reasonning based on image recognition. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #55968. 	[Title] Reinforcement Learning in the Multi-Robot Domain [Abstract] This paper describes a formulation of reinforcement learning that enables learning in noisy, dynamic environemnts such as in the complex concurrent multi-robot learning domain. The methodology involves minimizing the learning space through the use behaviors and conditions, and dealing with the credit assignment problem through shaped reinforcement in the form of heterogeneous reinforcement functions and progress estimators. We experimentally validate the ap proach on a group of four mobile robots learning a foraging task. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #124064. 	[Title] An Upper Bound on the Loss from Approximate Optimal-Value Functions [Abstract] Many reinforcement learning (RL) approaches can be formulated from the theory of Markov decision processes and the associated method of dynamic programming (DP). The value of this theoretical understanding, however, is tempered by many practical concerns. One important question is whether DP-based approaches that use function approximation rather than lookup tables, can avoid catastrophic effects on performance. This note presents a result in Bertsekas (1987) which guarantees that small errors in the approximation of a task's optimal value function cannot produce arbitrarily bad performance when actions are selected greedily. We derive an upper bound on performance loss which is slightly tighter than that in Bertsekas (1987), and we show the extension of the bound to Q-learning (Watkins, 1989). These results provide a theoretical justification for a practice that is common in reinforcement learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #58540. 	[Title] Machine Learning,  Explanation-Based Learning and Reinforcement Learning: A Unified View [Abstract] In speedup-learning problems, where full descriptions of operators are known, both explanation-based learning (EBL) and reinforcement learning (RL) methods can be applied. This paper shows that both methods involve fundamentally the same process of propagating information backward from the goal toward the starting state. Most RL methods perform this propagation on a state-by-state basis, while EBL methods compute the weakest preconditions of operators, and hence, perform this propagation on a region-by-region basis. Barto, Bradtke, and Singh (1995) have observed that many algorithms for reinforcement learning can be viewed as asynchronous dynamic programming. Based on this observation, this paper shows how to develop dynamic programming versions of EBL, which we call region-based dynamic programming or Explanation-Based Reinforcement Learning (EBRL). The paper compares batch and online versions of EBRL to batch and online versions of point-based dynamic programming and to standard EBL. The results show that region-based dynamic programming combines the strengths of EBL (fast learning and the ability to scale to large state spaces) with the strengths of reinforcement learning algorithms (learning of optimal policies). Results are shown in chess endgames and in synthetic maze tasks. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1116347. 	[Title] Fast Online Q() [Abstract] Q()-learning uses TD()-methods to accelerate Q-learning. The update complexity of previous online Q() implementations based on lookup-tables is bounded by the size of the state/action space. Our faster algorithm's update complexity is bounded by the number of actions. The method is based on the observation that Q-value updates may be postponed until they are needed. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28350. 	[Title] On the Convergence of Stochastic Iterative Dynamic Programming Algorithms [Abstract] Recent developments in the area of reinforcement learning have yielded a number of new algorithms for the prediction and control of Markovian environments. These algorithms, including the TD() algorithm of Sutton (1988) and the Q-learning algorithm of Watkins (1989), can be motivated heuristically as approximations to dynamic programming (DP). In this paper we provide a rigorous proof of convergence of these DP-based learning algorithms by relating them to the powerful techniques of stochastic approximation theory via a new convergence theorem. The theorem establishes a general class of convergent algorithms to which both TD() and Q-learning belong. This report describes research done at the Dept. of Brain and Cognitive Sciences, the Center for Biological and Computational Learning, and the Artificial Intelligence Laboratory of the Massachusetts Institute of Technology. Support for CBCL is provided in part by a grant from the NSF (ASC-9217041). Support for the laboratory's artificial intelligence research is provided in part by the Advanced Research Projects Agency of the Dept. of Defense. The authors were supported by a grant from the McDonnell-Pew Foundation, by a grant from ATR Human Information Processing Research Laboratories, by a grant from Siemens Corporation, by by grant IRI-9013991 from the National Science Foundation, by grant N00014-90-J-1942 from the Office of Naval Research, and by NSF grant ECS-9216531 to support an Initiative in Intelligent Control at MIT. Michael I. Jordan is a NSF Presidential Young Investigator. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1106112. 	[Title] Parameterized Heuristics for Intelligent Adaptive Network Routing in Large Communication Networks [Abstract] Parameterized heuristics offers an elegant and powerful theoretical framework for design and analysis of autonomous adaptive communication networks. Routing of messages in such networks presents a real-time instance of a multi-criterion optimization problem in a dynamic and uncertain environment. This paper describes a framework for heuristic routing in large networks. The effectiveness of the heuristic routing mechanism upon which Quo Vadis is based is described as part of a simulation study within a network with grid topology. A formal analysis of the underlying principles is presented through the incremental design of a set of heuristic decision functions that can be used to guide messages along a near-optimal (e.g., minimum delay) path in a large network. This paper carefully derives the properties of such heuristics under a set of simplifying assumptions about the network topology and load dynamics and identify the conditions under which they are guaranteed to route messages along an optimal path. The paper concludes with a discussion of the relevance of the theoretical results presented in the paper to the design of intelligent autonomous adaptive communication networks and an outline of some directions of future research. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #341188. 	[Title] A Unified Analysis of Value-Function-Based Reinforcement-Learning Algorithms [Abstract] Reinforcement learning is the problem of generating optimal behavior in a sequential decision-making environment given the opportunity of interacting with it. Many algorithms for solving reinforcement-learning problems work by computing improved estimates of the optimal value function. We extend prior analyses of reinforcement-learning algorithms and present a powerful new theorem that can provide a unified analysis of value-function-based reinforcement-learning algorithms. The usefulness of the theorem lies in how it allows the asynchronous convergence of a complex reinforcement-learning algorithm to be proven by verifying that a simpler synchronous algorithm converges. We illustrate the application of the theorem by analyzing the convergence of Q-learning, model-based reinforcement learning, Q-learning with multi-state updates, Q-learning for Markov games, and risk-sensitive reinforcement learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #132806. 	[Title] USING MARKER-BASED GENETIC ENCODING OF NEURAL NETWORKS TO EVOLVE FINITE-STATE BEHAVIOUR [Abstract] A new mechanism for genetic encoding of neural networks is proposed, which is loosely based on the marker structure of biological DNA. The mechanism allows all aspects of the network structure, including the number of nodes and their connectivity, to be evolved through genetic algorithms. The effectiveness of the encoding scheme is demonstrated in an object recognition task that requires artificial creatures (whose behaviour is driven by a neural network) to develop high-level finite-state exploration and discrimination strategies. The task requires solving the sensory-motor grounding problem, i.e. developing a functional understanding of the effects that a creature's movement has on its sensory input. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #416455. 	[Title] ROBO-SHEPHERD: LEARNING COMPLEX ROBOTIC BEHAVIORS [Abstract] This paper reports on recent results using genetic algorithms to learn decision rules for complex robot behaviors. The method involves evaluating hypothetical rule sets on a simulator and applying simulated evolution to evolve more effective rules. The main contributions of this paper are (1) the task learned is a complex behavior involving multiple mobile robots, and (2) the learned rules are verified through experiments on operational mobile robots. The case study involves a shepherding task in which one mobile robot attempts to guide another robot to a specified area. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #193742. 	[Title] Q-Learning with Hidden-Unit Restarting [Abstract] Platt's resource-allocation network (RAN) (Platt, 1991a, 1991b) is modified for a reinforcement-learning paradigm and to "restart" existing hidden units rather than adding new units. After restarting, units continue to learn via back-propagation. The resulting restart algorithm is tested in a Q-learning network that learns to solve an inverted pendulum problem. Solutions are found faster on average with the restart algorithm than without it. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #387795. 	[Title] University of Nevada Reno Design Strategies for Evolutionary Robotics [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1102550. 	[Title] Culling Teaching -1 Culling and Teaching in Neuro-evolution [Abstract] The evolving population of neural nets contains information not only in terms of genes, but also in the collection of behaviors of the population members. Such information can be thought of as a kind of culture of the population. Two ways of exploiting that culture are explored in this paper: (1) Culling overlarge litters: Generate a large number of offspring with different crossovers, quickly evaluate them by comparing their performance to the population, and throw away those that appear poor. (2) Teaching: Use backpropagation to train offspring toward the performance of the population. Both techniques result in faster, more effective neuro-evolution, and they can be effectively combined, as is demonstrated on the inverted pendulum problem. Additional methods of cultural exploitation are possible and will be studied in future work. These results suggest that cultural exploitation is a powerful idea that allows leveraging several aspects of the genetic algorithm. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1107674. 	[Title] EVOLUTIONARY ALGORITHMS IN ROBOTICS [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #23258. 	[Title] Exploration and Model Building in Mobile Robot Domains [Abstract] I present first results on COLUMBUS, an autonomous mobile robot. COLUMBUS operates in initially unknown, structured environments. Its task is to explore and model the environment efficiently while avoiding collisions with obstacles. COLUMBUS uses an instance-based learning technique for modeling its environment. Real-world experiences are generalized via two artificial neural networks that encode the characteristics of the robot's sensors, as well as the characteristics of typical environments the robot is assumed to face. Once trained, these networks allow for knowledge transfer across different environments the robot will face over its lifetime. COLUMBUS' models represent both the expected reward and the confidence in these expectations. Exploration is achieved by navigating to low confidence regions. An efficient dynamic programming method is employed in background to find minimal-cost paths that, executed by the robot, maximize exploration. COLUMBUS operates in real-time. It has been operating successfully in an office building environment for periods up to hours. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #567005. 	[Title] An unsupervised neural network for low-level control of a wheeled mobile robot: noise resistance, stability, [Abstract] We have recently introduced a neural network mobile robot controller (NETMORC) that autonomously learns the forward and inverse odometry of a differential drive robot through an unsupervised learning-by-doing cycle. After an initial learning phase, the controller can move the robot to an arbitrary stationary or moving target while compensating for noise and other forms of disturbance, such as wheel slippage or changes in the robot's plant. In addition, the forward odometric map allows the robot to reach targets in the absence of sensory feedback. The controller is also able to adapt in response to long-term changes in the robot's plant, such as a change in the radius of the wheels. In this article we review the NETMORC architecture and describe its simplified algorithmic implementation, we present new, quantitative results on NETMORC's performance and adaptability under noise-free and noisy conditions, we compare NETMORC's performance on a trajectory-following task with the performance of an alternative controller, and we describe preliminary results on the hardware implementation of NETMORC with the mobile robot ROBUTER. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28387. 	[Title] On the Complexity of Solving Markov Decision Problems [Abstract] Markov decision problems (MDPs) provide the foundations for a number of problems of interest to AI researchers studying automated planning and reinforcement learning. In this paper, we summarize results regarding the complexity of solving MDPs and the running time of MDP solution algorithms. We argue that, although MDPs can be solved efficiently in theory, more study is needed to reveal practical algorithms for solving large problems quickly. To encourage future research, we sketch some alternative methods of analysis that rely on the struc ture of MDPs. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1117942. 	[Title] Multi-criteria reinforcement learning [Abstract] We consider multi-criteria sequential decision making problems, where the criteria are ordered according to their importance. Structural properties of these problems are touched and reinforcement learning algorithms, which learn asymptotically optimal decisions, are derived. Computer experiments confirm the theoretical results and provide further insight in the learning processes. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1272. 	[Title] Learning a set of primitive actions with an Induction of decision trees. Machine Learning, 1(1):81-106, [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1106172. 	[Title] Category: Control, Navigation and Planning. Key words: Reinforcement learning, Exploration, Hidden state. Prefer oral presentation. [Abstract] This paper presents Fringe Exploration, a technique for efficient exploration in partially observable domains. The key idea, (applicable to many exploration techniques), is to keep statistics in the space of possible short-term memories, instead of in the agent's current state space. Experimental results in a partially observable maze and in a difficult driving task with visual routines show dramatic performance improvements. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #7432. 	[Title] How to Dynamically Merge Markov Decision Processes [Abstract] We are frequently called upon to perform multiple tasks that compete for our attention and resource. Often we know the optimal solution to each task in isolation; in this paper, we describe how this knowledge can be exploited to efficiently find good solutions for doing the tasks in parallel. We formulate this problem as that of dynamically merging multiple Markov decision processes (MDPs) into a composite MDP, and present a new theoretically-sound dynamic programming algorithm for finding an optimal policy for the composite MDP. We analyze various aspects of our algorithm and Every day, we are faced with the problem of doing multiple tasks in parallel, each of which competes for our attention and resource. If we are running a job shop, we must decide which machines to allocate to which jobs, and in what order, so that no jobs miss their deadlines. If we are a mail delivery robot, we must find the intended recipients of the mail while simultaneously avoiding fixed obstacles (such as walls) and mobile obstacles (such as people), and still manage to keep ourselves sufficiently charged up. Frequently we know how to perform each task in isolation; this paper considers how we can take the information we have about the individual tasks and combine it to efficiently find an optimal solution for doing the entire set of tasks in parallel. More importantly, we describe a theoretically-sound algorithm for doing this merging dynamically; new tasks (such as a new job arrival at a job shop) can be assimilated online into the solution being found for the ongoing set of simultaneous tasks. illustrate its use on a simple merging problem. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #628500. 	[Title] LEARNING TO AVOID COLLISIONS: A REINFORCEMENT LEARNING PARADIGM FOR MOBILE ROBOT NAVIGATION [Abstract] The paper describes a self-learning control system for a mobile robot. Based on sensor information the control system has to provide a steering signal in such a way that collisions are avoided. Since in our case no `examples' are available, the system learns on the basis of an external reinforcement signal which is negative in case of a collision and zero otherwise. We describe the adaptive algorithm which is used for a discrete coding of the state space, and the adaptive algorithm for learning the correct mapping from the input (state) vector to the output (steering) signal. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1120020. 	[Title] Learning from undiscounted delayed rewards [Abstract] The general framework of reinforcement learning has been proposed by several researchers for both the solution of optimization problems and the realization of adaptive control schemes. To allow for an efficient application of reinforcement learning in either of these areas, it is necessary to solve both the structural and the temporal credit assignment problem. In this paper, we concentrate on the latter which is usually tackled through the use of learning algorithms that employ discounted rewards. We argue that for realistic problems this kind of solution is not satisfactory, since it does not address the effect of noise originating from different experiences and does not allow for an easy explanation of the parameters involved in the learning process. As a possible solution, we propose to keep the delayed reward undiscounted, but to discount the actual adaptation rate. Empirical results show that dependent on the kind of discount used amore stable convergence and even an increase in performance can be obtained. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1031453. 	[Title] A Neural Network Pole Balancer that Learns and Operates on a Real Robot in Real Time [Abstract] A neural network approach to the classic inverted pendulum task is presented. This task is the task of keeping a rigid pole, hinged to a cart and free to fall in a plane, in a roughly vertical orientation by moving the cart horizontally in the plane while keeping the cart within some maximum distance of its starting position. This task constitutes a difficult control problem if the parameters of the cart-pole system are not known precisely or are variable. It also forms the basis of an even more complex control-learning problem if the controller must learn the proper actions for successfully balancing the pole given only the current state of the system and a failure signal when the pole angle from the vertical becomes too great or the cart exceeds one of the boundaries placed on its position. The approach presented is demonstrated to be effective for the real-time control of a small, self-contained mini-robot, specially outfitted for the task. Origins and details of the learning scheme, specifics of the mini-robot hardware, and results of actual learning trials are presented. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #12576. 	[Title] Learning Sequential Decision Rules Using Simulation Models and Competition [Abstract] The problem of learning decision rules for sequential tasks is addressed, focusing on the problem of learning tactical decision rules from a simple flight simulator. The learning method relies on the notion of competition and employs genetic algorithms to search the space of decision policies. Several experiments are presented that address issues arising from differences between the simulation model on which learning occurs and the target environment on which the decision rules are ultimately tested. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #308529. 	[Title] The Role of the Trainer in Reinforcement Learning [Abstract] In this paper we propose a threestage incremental approach to the development of autonomous agents. We discuss some issues about the characteristics which differentiate reinforcement programs (RPs), and define the trainer as a particular kind of RP. We present a set of results obtained running experiments with a trainer which provides guidance to the AutonoMouse, our mousesized autonomous robot. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28471. 	[Title] Markov Decision Processes in Large State Spaces [Abstract] In this paper we propose a new framework for studying Markov decision processes (MDPs), based on ideas from statistical mechanics. The goal of learning in MDPs is to find a policy that yields the maximum expected return over time. In choosing policies, agents must therefore weigh the prospects of short-term versus long-term gains. We study a simple MDP in which the agent must constantly decide between exploratory jumps and local reward mining in state space. The number of policies to choose from grows exponentially with the size of the state space, N . We view the expected returns as defining an energy landscape over policy space. Methods from statistical mechanics are used to analyze this landscape in the thermodynamic limit N ! 1. We calculate the overall distribution of expected returns, as well as the distribution of returns for policies at a fixed Hamming distance from the optimal one. We briefly discuss the problem of learning optimal policies from empirical estimates of the expected return. As a first step, we relate our findings for the entropy to the limit of high-temperature learning. Numerical simulations support the theoretical results. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28485. 	[Title] The Efficient Learning of Multiple Task Sequences [Abstract] I present a modular network architecture and a learning algorithm based on incremental dynamic programming that allows a single learning agent to learn to solve multiple Markovian decision tasks (MDTs) with significant transfer of learning across the tasks. I consider a class of MDTs, called composite tasks, formed by temporally concatenating a number of simpler, elemental MDTs. The architecture is trained on a set of composite and elemental MDTs. The temporal structure of a composite task is assumed to be unknown and the architecture learns to produce a temporal decomposition. It is shown that under certain conditions the solution of a composite MDT can be constructed by computationally inexpensive modifications of the solutions of its constituent elemental MDTs. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #91975. 	[Title] Learning Curve Bounds for Markov Decision Processes with Undiscounted Rewards [Abstract] Markov decision processes (MDPs) with undis-counted rewards represent an important class of problems in decision and control. The goal of learning in these MDPs is to find a policy that yields the maximum expected return per unit time. In large state spaces, computing these averages directly is not feasible; instead, the agent must estimate them by stochastic exploration of the state space. In this case, longer exploration times enable more accurate estimates and more informed decision-making. The learning curve for an MDP measures how the agent's performance depends on the allowed exploration time, T . In this paper we analyze these learning curves for a simple control problem with undiscounted rewards. In particular, methods from statistical mechanics are used to calculate lower bounds on the agent's performance in the thermodynamic limit T ! 1, N ! 1, ff = T =N (finite), where T is the number of time steps allotted per policy evaluation and N is the size of the state space. In this limit, we provide a lower bound on the return of policies that appear optimal based on imperfect statistics. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #36167. 	[Title] Category: Control, Navigation and Planning Preference: Oral presentation Exploiting Model Uncertainty Estimates for Safe Dynamic [Abstract] Model learning combined with dynamic programming has been shown to be effective for learning control of continuous state dynamic systems. The simplest method assumes the learned model is correct and applies dynamic programming to it, but many approximators provide uncertainty estimates on the fit. How can they be exploited? This paper addresses the case where the system must be prevented from having catastrophic failures during learning. We propose a new algorithm adapted from the dual control literature and use Bayesian locally weighted regression models with stochastic dynamic programming. A common reinforcement learning assumption is that aggressive exploration should be encouraged. This paper addresses the converse case in which the system has to reign in exploration. The algorithm is illustrated on a 4 dimensional simulated control problem. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #63832. 	[Title] An Evolutionary Approach to Learning in Robots [Abstract] Evolutionary learning methods have been found to be useful in several areas in the development of intelligent robots. In the approach described here, evolutionary algorithms are used to explore alternative robot behaviors within a simulation model as a way of reducing the overall knowledge engineering effort. This paper presents some initial results of applying the SAMUEL genetic learning system to a collision avoidance and navigation task for mobile robots. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1385. 	[Title] REINFORCEMENT LEARNING FOR COORDINATED REACTIVE CONTROL [Abstract] The demands of rapid response and the complexity of many environments make it difficult to decompose, tune and coordinate reactive behaviors while ensuring consistency. Reinforcement learning networks can address the tuning problem, but do not address the problem of decomposition and coordination. We hypothesize that interacting reactions can often be decomposed into separate control tasks resident in separate networks and that the interaction can be coordinated through the tuning mechanism and a higher level controller. To explore these issues, we have implemented a reinforcement learning architecture as the reactive component of a two layer control system for a simulated race car. By varying the architecture, we test whether decomposing reactivity into separate controllers leads to superior overall performance and learning convergence in our domain. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #13686. 	[Title] Framework for Combining Symbolic and Neural Learning rule extraction from neural networks the KBANN algorithm [Abstract] Technical Report 1123, Computer Sciences Department, University of Wisconsin - Madison, Nov. 1992 ABSTRACT This article describes an approach to combining symbolic and connectionist approaches to machine learning. A three-stage framework is presented and the research of several groups is reviewed with respect to this framework. The first stage involves the insertion of symbolic knowledge into neural networks, the second addresses the refinement of this prior knowledge in its neural representation, while the third concerns the extraction of the refined symbolic knowledge. Experimental results and open research issues are discussed. A shorter version of this paper will appear in Machine Learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1127810. 	[Title] Neural competitive maps for reactive and adaptive navigation [Abstract] We have recently introduced a neural network for reactive obstacle avoidance based on a model of classical and operant conditioning. In this article we describe the success of this model when implemented on two real autonomous robots. Our results show the promise of self-organizing neural networks in the domain of intelligent robotics. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #56708. 	[Title] ALECSYS and the AutonoMouse: Learning to Control a Real Robot by Distributed Classifier Systems [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #56709. 	[Title] GENETIC AND NON GENETIC OPERATORS IN ALECSYS [Abstract] It is well known that standard learning classifier systems, when applied to many different domains, exhibit a number of problems: payoff oscillation, difficult to regulate interplay between the reward system and the background genetic algorithm (GA), rule chains instability, default hierarchies instability, are only a few. ALECSYS is a parallel version of a standard learning classifier system (CS), and as such suffers of these same problems. In this paper we propose some innovative solutions to some of these problems. We introduce the following original features. Mutespec, a new genetic operator used to specialize potentially useful classifiers. Energy, a quantity introduced to measure global convergence in order to apply the genetic algorithm only when the system is close to a steady state. Dynamical adjustment of the classifiers set cardinality, in order to speed up the performance phase of the algorithm. We present simulation results of experiments run in a simulated two-dimensional world in which a simple agent learns to follow a light source. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1128839. 	[Title] Requirements and use of neural networks for industrial appli- [Abstract] Modern industry of today needs flexible, adaptive and fault-tolerant methods for information processing. Several applications have shown that neural networks fulfill these requirements. In this paper application areas, in which neural networks have been successfully used, are presented. Then a kind of check list is described, which mentioned the different steps, when applying neural networks. The paper finished with a discussion of some neural networks projects done in the research group Interactive Planning at the Research Center for Computer Science (FZI). Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #18834. 	[Title] A Study of the Generalization Capabilities of XCS [Abstract] We analyze the generalization behavior of the XCS classifier system in environments in which only a few generalizations can be done. Experimental results presented in the paper evidence that the generalization mechanism of XCS can prevent it from learning even simple tasks in such environments. We present a new operator, named Specify, which contributes to the solution of this problem. XCS with the Specify operator, named XCSS, is compared to XCS in terms of performance and generalization capabilities in different types of environments. Experimental results show that XCSS can deal with a greater variety of environments and that it is more robust than XCS with respect to population size. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1107355. 	[Title] Coordinating Reactive Behaviors  keywords: reactive systems, planning and learning [Abstract] Combinating reactivity with planning has been proposed as a means of compensating for potentially slow response times of planners while still making progress toward long term goals. The demands of rapid response and the complexity of many environments make it difficult to decompose, tune and coordinate reactive behaviors while ensuring consistency. Neural networks can address the tuning problem, but are less useful for decomposition and coordination. We hypothesize that interacting reactions can be decomposed into separate behaviors resident in separate networks and that the interaction can be coordinated through the tuning mechanism and a higher level controller. To explore these issues, we have implemented a neural network architecture as the reactive component of a two layer control system for a simulated race car. By varying the architecture, we test whether decomposing reactivity into separate behaviors leads to superior overall performance, coordination and learning convergence. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #293285. 	[Title] Why Experimentation can be better than "Perfect Guidance" [Abstract] Many problems correspond to the classical control task of determining the appropriate control action to take, given some (sequence of) observations. One standard approach to learning these control rules, called behavior cloning, involves watching a perfect operator operate a plant, and then trying to emulate its behavior. In the experimental learning approach, by contrast, the learner first guesses an initial operation-to-action policy and tries it out. If this policy performs sub-optimally, the learner can modify it to produce a new policy, and recur. This paper discusses the relative effectiveness of these two approaches, especially in the presence of perceptual aliasing, showing in particular that the experimental learner can often learn more effectively than the cloning one. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #44455. 	[Title] Evolving Visually Guided Robots [Abstract] A version of this paper appears in: Proceedings of SAB92, the Second International Conference on Simulation of Adaptive Behaviour J.-A. Meyer, H. Roitblat, and S. Wilson, editors, MIT Press Bradford Books, Cambridge, MA, 1993. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1120170. 	[Title] Learning Optimal Dialogue Strategies: A Case Study of a Spoken Dialogue Agent for Email [Abstract] This paper describes a novel method by which a dialogue agent can learn to choose an optimal dialogue strategy. While it is widely agreed that dialogue strategies should be formulated in terms of communicative intentions, there has been little work on automatically optimizing an agent's choices when there are multiple ways to realize a communicative intention. Our method is based on a combination of learning algorithms and empirical evaluation techniques. The learning component of our method is based on algorithms for reinforcement learning, such as dynamic programming and Q-learning. The empirical component uses the PARADISE evaluation framework (Walker et al., 1997) to identify the important performance factors and to provide the performance function needed by the learning algorithm. We illustrate our method with a dialogue agent named ELVIS (EmaiL Voice Interactive System), that supports access to email over the phone. We show how ELVIS can learn to choose among alternate strategies for agent initiative, for reading messages, and for summarizing email folders. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #191404. 	[Title] Exploiting Structure in Policy Construction [Abstract] Markov decision processes (MDPs) have recently been applied to the problem of modeling decision-theoretic planning. While traditional methods for solving MDPs are often practical for small states spaces, their effectiveness for large AI planning problems is questionable. We present an algorithm, called structured policy iteration (SPI), that constructs optimal policies without explicit enumeration of the state space. The algorithm retains the fundamental computational steps of the commonly used modified policy iteration algorithm, but exploits the variable and propositional independencies reflected in a temporal Bayesian network representation of MDPs. The principles behind SPI can be applied to any structured representation of stochastic actions, policies and value functions, and the algorithm itself can be used in conjunction with re cent approximation methods. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #434. 	[Title] Learning Analytically and Inductively [Abstract] Learning is a fundamental component of intelligence, and a key consideration in designing cognitive architectures such as Soar [ Laird et al., 1986 ] . This chapter considers the question of what constitutes an appropriate general-purpose learning mechanism. We are interested in mechanisms that might explain and reproduce the rich variety of learning capabilities of humans, ranging from learning perceptual-motor skills such as how to ride a bicycle, to learning highly cognitive tasks such as how to play chess. Research on learning in fields such as cognitive science, artificial intelligence, neurobiology, and statistics has led to the identification of two distinct classes of learning methods: inductive and analytic. Inductive methods, such as neural network Backpropagation, learn general laws by finding statistical correlations and regularities among a large set of training examples. In contrast, analytical methods, such as Explanation-Based Learning, acquire general laws from many fewer training examples. They rely instead on prior knowledge to analyze individual training examples in detail, then use this analysis to distinguish relevant example features from the irrelevant. The question considered in this chapter is how to best combine inductive and analytical learning in an architecture that seeks to cover the range of learning exhibited by intelligent systems such as humans. We present a specific learning mechanism, Explanation Based Neural Network learning (EBNN), that blends these two types of learning, and present experimental results demonstrating its ability to learn control strategies for a mobile robot using Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1128959. 	[Title] On the Relations Between Search and Evolutionary Algorithms [Abstract] Technical Report: CSRP-96-7 March 1996 Abstract Evolutionary algorithms are powerful techniques for optimisation whose operation principles are inspired by natural selection and genetics. In this paper we discuss the relation between evolutionary techniques, numerical and classical search methods and we show that all these methods are instances of a single more general search strategy, which we call the `evolutionary computation cookbook'. By combining the features of classical and evolutionary methods in different ways new instances of this general strategy can be generated, i.e. new evolutionary (or classical) algorithms can be designed. One such algorithm, GA fl , is described. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #976334. 	[Title] Applying Online Search Techniques to Continuous-State Reinforcement Learning key to the success of the local [Abstract] In this paper, we describe methods for efficiently computing better solutions to control problems in continuous state spaces. We provide algorithms that exploit online search to boost the power of very approximate value functions discovered by traditional reinforcement learning techniques. We examine local searches, where the agent performs a finite-depth lookahead search, and global searches, where the agent performs a search for a trajectory all the way from the current state to a goal state. The key to the success of the global methods lies in using aggressive state-space search techniques such as uniform-cost search and A fl , tamed into a tractable form by exploiting neighborhood relations and trajectory constraints that arise from continuous-space dynamic control. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1103315. 	[Title] Value Function Based Production Scheduling [Abstract] Production scheduling, the problem of sequentially configuring a factory to meet forecasted demands, is a critical problem throughout the manufacturing industry. The requirement of maintaining product inventories in the face of unpredictable demand and stochastic factory output makes standard scheduling models, such as job-shop, inadequate. Currently applied algorithms, such as simulated annealing and constraint propagation, must employ ad-hoc methods such as frequent replanning to cope with uncertainty. In this paper, we describe a Markov Decision Process (MDP) formulation of production scheduling which captures stochasticity in both production and demands. The solution to this MDP is a value function which can be used to generate optimal scheduling decisions online. A simple example illustrates the theoretical superiority of this approach over replanning-based methods. We then describe an industrial application and two reinforcement learning methods for generating an approximate value function on this domain. Our results demonstrate that in both deterministic and noisy scenarios, value function approx imation is an effective technique. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #136665. 	[Title] The Influence of Domain Properties on the Performance of Real-Time Search Algorithms [Abstract] This research is sponsored by the Wright Laboratory, Aeronautical Systems Center, Air Force Materiel Command, USAF, and the Advanced Research Projects Agency (ARPA) under grant number F33615-93-1-1330. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the sponsoring organizations. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #168410. 	[Title] A Modular Q-Learning Architecture for Manipulator Task Decomposition `Data storage in the cerebellar model ar [Abstract] Compositional Q-Learning (CQ-L) (Singh 1992) is a modular approach to learning to perform composite tasks made up of several elemental tasks by reinforcement learning. Skills acquired while performing elemental tasks are also applied to solve composite tasks. Individual skills compete for the right to act and only winning skills are included in the decomposition of the composite task. We extend the original CQ-L concept in two ways: (1) a more general reward function, and (2) the agent can have more than one actuator. We use the CQ-L architecture to acquire skills for performing composite tasks with a simulated two-linked manipulator having large state and action spaces. The manipulator is a non-linear dynamical system and we require its end-effector to be at specific positions in the workspace. Fast function approximation in each of the Q-modules is achieved through the use of an array of Cerebellar Model Articulation Controller (CMAC) (Albus Our research interests involve the scaling up of machine learning methods, especially reinforcement learning, for autonomous robot control. We are interested in function approximators suitable for reinforcement learning in problems with large state spaces, such as the Cerebellar Model Articulation Controller (CMAC) (Albus 1975) which permit fast, online learning and good local generalization. In addition, we are interested in task decomposition by reinforcement learning and the use of hierarchical and modular function approximator architectures. We are examining the effectiveness of a modified Hierarchical Mixtures of Experts (HME) (Jordan & Jacobs 1993) approach for reinforcement learning since the original HME was developed mainly for supervised learning and batch learning tasks. The incorporation of domain knowledge into reinforcement learning agents is an important way of extending their capabilities. Default policies can be specified, and domain knowledge can also be used to restrict the size of the state-action space, leading to faster learning. We are investigating the use of Q-Learning (Watkins 1989) in planning tasks, using a classifier system (Holland 1986) to encode the necessary condition-action rules. Jordan, M. & Jacobs, R. (1993), Hierarchical mixtures of experts and the EM algorithm, Technical Report 9301, MIT Computational Cognitive Science. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #64484. 	[Title] Learning from an Automated Training Agent [Abstract] A learning agent employing reinforcement learning is hindered because it only receives the critic's sparse and weakly informative training information. We present an approach in which an automated training agent may also provide occasional instruction to the learner in the form of actions for the learner to perform. The learner has access to both the critic's feedback and the trainer's instruction. In the experiments, we vary the level of the trainer's interaction with the learner, from allowing the trainer to instruct the learner at almost every time step, to not allowing the trainer to respond at all. We also vary a parameter that controls how the learner incorporates the trainer's actions. The results show significant reductions in the average number of training trials necessary to learn to perform the task. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1106406. 	[Title] Submitted to NIPS96, Section: Applications. Preference: Oral presentation Reinforcement Learning for Dynamic Channel Allocation in [Abstract] In cellular telephone systems, an important problem is to dynamically allocate the communication resource (channels) so as to maximize service in a stochastic caller environment. This problem is naturally formulated as a dynamic programming problem and we use a reinforcement learning (RL) method to find dynamic channel allocation policies that are better than previous heuristic solutions. The policies obtained perform well for a broad variety of call traffic patterns. We present results on a large cellular system In cellular communication systems, an important problem is to allocate the communication resource (bandwidth) so as to maximize the service provided to a set of mobile callers whose demand for service changes stochastically. A given geographical area is divided into mutually disjoint cells, and each cell serves the calls that are within its boundaries (see Figure 1a). The total system bandwidth is divided into channels, with each channel centered around a frequency. Each channel can be used simultaneously at different cells, provided these cells are sufficiently separated spatially, so that there is no interference between them. The minimum separation distance between simultaneous reuse of the same channel is called the channel reuse constraint . When a call requests service in a given cell either a free channel (one that does not violate the channel reuse constraint) may be assigned to the call, or else the call is blocked from the system; this will happen if no free channel can be found. Also, when a mobile caller crosses from one cell to another, the call is "handed off" to the cell of entry; that is, a new free channel is provided to the call at the new cell. If no such channel is available, the call must be dropped/disconnected from the system. One objective of a channel allocation policy is to allocate the available channels to calls so that the number of blocked calls is minimized. An additional objective is to minimize the number of calls that are dropped when they are handed off to a busy cell. These two objectives must be weighted appropriately to reflect their relative importance, since dropping existing calls is generally more undesirable than blocking new calls. with approximately 70 49 states. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #4584. 	[Title] References elements that can solve difficult learning control problems. on Simulation of Adaptive Behavior, pages [Abstract] Miller, G. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review, 63(2):81-97. Schmidhuber, J. (1990b). Towards compositional learning with dynamic neural networks. Technical Report FKI-129-90, Technische Universitat Munchen, Institut fu Informatik. Servan-Schreiber, D., Cleermans, A., and McClelland, J. (1988). Encoding sequential structure in simple recurrent networks. Technical Report CMU-CS-88-183, Carnegie Mellon University, Computer Science Department. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #39403. 	[Title] Auto-exploratory Average Reward Reinforcement Learning [Abstract] We introduce a model-based average reward Reinforcement Learning method called H-learning and compare it with its discounted counterpart, Adaptive Real-Time Dynamic Programming, in a simulated robot scheduling task. We also introduce an extension to H-learning, which automatically explores the unexplored parts of the state space, while always choosing greedy actions with respect to the current value function. We show that this "Auto-exploratory H-learning" performs better than the original H-learning under previously studied exploration methods such as random, recency-based, or counter-based exploration. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1105394. 	[Title] An Introspection Approach to Querying a Trainer [Abstract] Technical Report 96-13 January 22, 1996 Abstract This paper introduces the Introspection Approach, a method by which a learning agent employing reinforcement learning can decide when to ask a training agent for instruction. When using our approach, we find that the same number of trainer's responses produced significantly faster learners than by having the learner ask for aid randomly. Guidance received via our approach is more informative than random guidance. Thus, we can reduce the interaction that the training agent has with the learning agent without reducing the speed with which the learner develops its policy. In fact, by being intelligent about when the learner asks for help, we can even increase the learning speed for the same level of trainer interaction. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1107455. 	[Title] Solving Combinatorial Optimization Tasks by Reinforcement Learning: A General Methodology Applied to Resource-Constrained Scheduling [Abstract] This paper introduces a methodology for solving combinatorial optimization problems through the application of reinforcement learning methods. The approach can be applied in cases where several similar instances of a combinatorial optimization problem must be solved. The key idea is to analyze a set of "training" problem instances and learn a search control policy for solving new problem instances. The search control policy has the twin goals of finding high-quality solutions and finding them quickly. Results of applying this methodology to a NASA scheduling problem show that the learned search control policy is much more effective than the best known non-learning search procedure|a method based on simulated annealing. Please summarise all the given information.

Paper #28287 has the title `Robot Shaping: Developing Situated Agents through Learning  
` and the abstract `Learning plays a vital role in the development of situated agents. In this paper, we explore the use of reinforcement learning to "shape" a robot to perform a predefined target behavior. We connect both simulated and real robots to A LECSYS, a parallel implementation of a learning classifier system with an extended genetic algorithm. After classifying different kinds of Animat-like behaviors, we explore the effects on learning of different types of agent's architecture (monolithic, flat and hierarchical) and of training strategies. In particular, hierarchical architecture requires the agent to learn how to coordinate basic learned responses. We show that the best results are achieved when both the agent's architecture and the training strategy match the structure of the behavior pattern to be learned. We report the results of a number of experiments carried out both in simulated and in real environments, and show that the results of simulations carry smoothly to real robots. While most of our experiments deal with simple reactive behavior, in one of them we demonstrate the use of a simple and general memory mechanism. As a whole, our experimental activity demonstrates that classifier systems with genetic algorithms can be practically employed to develop autonomous agents. 
`.
Question: Which category should Paper #28287 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].


Paper #28287 has the title `Robot Shaping: Developing Situated Agents through Learning  
` and the abstract `Learning plays a vital role in the development of situated agents. In this paper, we explore the use of reinforcement learning to "shape" a robot to perform a predefined target behavior. We connect both simulated and real robots to A LECSYS, a parallel implementation of a learning classifier system with an extended genetic algorithm. After classifying different kinds of Animat-like behaviors, we explore the effects on learning of different types of agent's architecture (monolithic, flat and hierarchical) and of training strategies. In particular, hierarchical architecture requires the agent to learn how to coordinate basic learned responses. We show that the best results are achieved when both the agent's architecture and the training strategy match the structure of the behavior pattern to be learned. We report the results of a number of experiments carried out both in simulated and in real environments, and show that the results of simulations carry smoothly to real robots. While most of our experiments deal with simple reactive behavior, in one of them we demonstrate the use of a simple and general memory mechanism. As a whole, our experimental activity demonstrates that classifier systems with genetic algorithms can be practically employed to develop autonomous agents. 
`.
Question: Which category should Paper #28287 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].

This paper should be classified into: Reinforcement_Learning