Above are papers related to the following paper: [Paper_ID] Paper #6378. 	[Title] Learning to coordinate without sharing information [Abstract] Researchers in the field of Distributed Artificial Intelligence (DAI) have been developing efficient mechanisms to coordinate the activities of multiple autonomous agents. The need for coordination arises because agents have to share resources and expertise required to achieve their goals. Previous work in the area includes using sophisticated information exchange protocols, investigating heuristics for negotiation, and developing formal models of possibilities of conflict and cooperation among agent interests. In order to handle the changing requirements of continuous and dynamic environments, we propose learning as a means to provide additional possibilities for effective coordination. We use reinforcement learning techniques on a block pushing problem to show that agents can learn complimentary policies to follow a desired path without any knowledge about each other. We theoretically analyze and experimentally verify the effects of learning rate on system convergence, and demonstrate benefits of using learned coordination knowledge on similar problems. Reinforcement learning based coordination can be achieved in both cooperative and non-cooperative domains, and in domains with noisy communication channels and other stochastic characteristics that present a formidable challenge to using other coordination schemes. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1272. 	[Title] Learning a set of primitive actions with an Induction of decision trees. Machine Learning, 1(1):81-106, [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6151. 	[Title] Strategy Learning with Multilayer Connectionist Representations 1 [Abstract] Results are presented that demonstrate the learning and fine-tuning of search strategies using connectionist mechanisms. Previous studies of strategy learning within the symbolic, production-rule formalism have not addressed fine-tuning behavior. Here a two-layer connectionist system is presented that develops its search from a weak to a task-specific strategy and fine-tunes its performance. The system is applied to a simulated, real-time, balance-control task. We compare the performance of one-layer and two-layer networks, showing that the ability of the two-layer network to discover new features and thus enhance the original representation is critical to solving the balancing task. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #93320. 	[Title] Some Experiments with a Hybrid Model for Learning Sequential Decision Making [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #13960. 	[Title] Using a Case Base of Surfaces to Speed-Up Reinforcement Learning [Abstract] This paper demonstrates the exploitation of certain vision processing techniques to index into a case base of surfaces. The surfaces are the result of reinforcement learning and represent the optimum choice of actions to achieve some goal from anywhere in the state space. This paper shows how strong features that occur in the interaction of the system with its environment can be detected early in the learning process. Such features allow the system to identify when an identical, or very similar, task has been solved previously and to retrieve the relevant surface. This results in an orders of magnitude increase in learning rate. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6155. 	[Title] On the Computational Economics of Reinforcement Learning [Abstract] Following terminology used in adaptive control, we distinguish between indirect learning methods, which learn explicit models of the dynamic structure of the system to be controlled, and direct learning methods, which do not. We compare an existing indirect method, which uses a conventional dynamic programming algorithm, with a closely related direct reinforcement learning method by applying both methods to an infinite horizon Markov decision problem with unknown state-transition probabilities. The simulations show that although the direct method requires much less space and dramatically less computation per control action, its learning ability in this task is superior to, or compares favorably with, that of the more complex indirect method. Although these results do not address how the methods' performances compare as problems become more difficult, they suggest that given a fixed amount of computational power available per control action, it may be better to use a direct reinforcement learning method augmented with indirect techniques than to devote all available resources to a computation-ally costly indirect method. Comprehensive answers to the questions raised by this study depend on many factors making up the eco nomic context of the computation. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1126029. 	[Title] Exploration in Active Learning [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #18832. 	[Title] Reinforcement Learning: A Survey [Abstract] This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word "reinforcement." The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #628500. 	[Title] LEARNING TO AVOID COLLISIONS: A REINFORCEMENT LEARNING PARADIGM FOR MOBILE ROBOT NAVIGATION [Abstract] The paper describes a self-learning control system for a mobile robot. Based on sensor information the control system has to provide a steering signal in such a way that collisions are avoided. Since in our case no `examples' are available, the system learns on the basis of an external reinforcement signal which is negative in case of a collision and zero otherwise. We describe the adaptive algorithm which is used for a discrete coding of the state space, and the adaptive algorithm for learning the correct mapping from the input (state) vector to the output (steering) signal. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #54550. 	[Title] Analysis and Empirical Studies of Derivational Analogy [Abstract] Derivational analogy is a technique for reusing problem solving experience to improve problem solving performance. This research addresses an issue common to all problem solvers that use derivational analogy: overcoming the mismatches between past experiences and new problems that impede reuse. First, this research describes the variety of mismatches that can arise and proposes a new approach to derivational analogy that uses appropriate adaptation strategies for each. Second, it compares this approach with seven others in a common domain. This empirical study shows that derivational analogy is almost always more efficient than problem solving from scratch, but the amount it contributes depends on its ability to overcome mismatches Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1129243. 	[Title] Learning in Multi-Robot Systems [Abstract] This paper 1 discusses why traditional reinforcement learning methods, and algorithms applied to those models, result in poor performance in dynamic, situated multi-agent domains characterized by multiple goals, noisy perception and action, and inconsistent reinforcement. We propose a methodology for designing the representation and the forcement functions that take advantage of implicit domain knowledge in order to accelerate learning in such domains, and demonstrate it experimentally in two different mobile robot domains. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28447. 	[Title] MULTISTRATEGY LEARNING IN REACTIVE CONTROL SYSTEMS FOR AUTONOMOUS ROBOTIC NAVIGATION [Abstract] This paper presents a self-improving reactive control system for autonomous robotic navigation. The navigation module uses a schema-based reactive control system to perform the navigation task. The learning module combines case-based reasoning and reinforcement learning to continuously tune the navigation system through experience. The case-based reasoning component perceives and characterizes the system's environment, retrieves an appropriate case, and uses the recommendations of the case to tune the parameters of the reactive control system. The reinforcement learning component refines the content of the cases based on the current experience. Together, the learning components perform on-line adaptation, resulting in improved performance as the reactive control system tunes itself to the environment, as well as on-line case learning, resulting in an improved library of cases that capture environmental regularities necessary to perform on-line adaptation. The system is extensively evaluated through simulation studies using several performance metrics and system configurations. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #55968. 	[Title] Reinforcement Learning in the Multi-Robot Domain [Abstract] This paper describes a formulation of reinforcement learning that enables learning in noisy, dynamic environemnts such as in the complex concurrent multi-robot learning domain. The methodology involves minimizing the learning space through the use behaviors and conditions, and dealing with the credit assignment problem through shaped reinforcement in the form of heterogeneous reinforcement functions and progress estimators. We experimentally validate the ap proach on a group of four mobile robots learning a foraging task. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #124064. 	[Title] An Upper Bound on the Loss from Approximate Optimal-Value Functions [Abstract] Many reinforcement learning (RL) approaches can be formulated from the theory of Markov decision processes and the associated method of dynamic programming (DP). The value of this theoretical understanding, however, is tempered by many practical concerns. One important question is whether DP-based approaches that use function approximation rather than lookup tables, can avoid catastrophic effects on performance. This note presents a result in Bertsekas (1987) which guarantees that small errors in the approximation of a task's optimal value function cannot produce arbitrarily bad performance when actions are selected greedily. We derive an upper bound on performance loss which is slightly tighter than that in Bertsekas (1987), and we show the extension of the bound to Q-learning (Watkins, 1989). These results provide a theoretical justification for a practice that is common in reinforcement learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1110947. 	[Title] Lazy Induction Triggered by CBR [Abstract] In recent years, case-based reasoning has been demonstrated to be highly useful for problem solving in complex domains. Also, mixed paradigm approaches emerged for combining CBR and induction techniques aiming at verifying the knowledge and/or building an efficient case memory. However, in complex domains induction over the whole problem space is often not possible or too time consuming. In this paper, an approach is presented which (owing to a close interaction with the CBR part) attempts to induce rules only for a particular context, i.e. for a problem just being solved by a CBR-oriented system. These rules may then be used for indexing purposes or similarity assessment in order to support the CBR process in the future. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #149669. 	[Title] Reasoning about Time and Probability [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6184. 	[Title] Forward models: Supervised learning with a distal teacher [Abstract] Internal models of the environment have an important role to play in adaptive systems in general and are of particular importance for the supervised learning paradigm. In this paper we demonstrate that certain classical problems associated with the notion of the "teacher" in supervised learning can be solved by judicious use of learned internal models as components of the adaptive system. In particular, we show how supervised learning algorithms can be utilized in cases in which an unknown dynamical system intervenes between actions and desired outcomes. Our approach applies to any supervised learning algorithm that is capable of learning in multi-layer networks. *This paper is a revised version of MIT Center for Cognitive Science Occasional Paper #40. We wish to thank Michael Mozer, Andrew Barto, Robert Jacobs, Eric Loeb, and James McClelland for helpful comments on the manuscript. This project was supported in part by BRSG 2 S07 RR07047-23 awarded by the Biomedical Research Support Grant Program, Division of Research Resources, National Institutes of Health, by a grant from ATR Auditory and Visual Perception Research Laboratories, by a grant from Siemens Corporation, by a grant from the Human Frontier Science Program, and by grant N00014-90-J-1942 awarded by the Office of Naval Research. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #682666. 	[Title] Multiagent Reinforcement Learning: Theoretical Framework and an Algorithm [Abstract] In this paper, we adopt general-sum stochastic games as a framework for multiagent reinforcement learning. Our work extends previous work by Littman on zero-sum stochastic games to a broader framework. We design a multiagent Q-learning method under this framework, and prove that it converges to a Nash equilibrium under specified conditions. This algorithm is useful for finding the optimal strategy when there exists a unique Nash equilibrium in the game. When there exist multiple Nash equilibria in the game, this algorithm should be combined with other learning techniques to find optimal strategies. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1108267. 	[Title] Cooperative Bayesian and Case-Based Reasoning for Solving Multiagent Planning Tasks [Abstract] We describe an integrated problem solving architecture named INBANCA in which Bayesian networks and case-based reasoning (CBR) work cooperatively on multiagent planning tasks. This includes two-team dynamic tasks, and this paper concentrates on simulated soccer as an example. Bayesian networks are used to characterize action selection whereas a case-based approach is used to determine how to implement actions. This paper has two contributions. First, we survey integrations of case-based and Bayesian approaches from the perspective of a popular CBR task decomposition framework, thus explaining what types of integrations have been attempted. This allows us to explain the unique aspects of our proposed integration. Second, we demonstrate how Bayesian nets can be used to provide environmental context, and thus feature selection information, for the case-based reasoner. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #10798. 	[Title] Learning physical descriptions from functional definitions, examples, Learning from examples: The effect of different conceptual [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6196. 	[Title] The Parti-game Algorithm for Variable Resolution Reinforcement Learning in Multidimensional State-spaces [Abstract] Parti-game is a new algorithm for learning feasible trajectories to goal regions in high dimensional continuous state-spaces. In high dimensions it is essential that learning does not plan uniformly over a state-space. Parti-game maintains a decision-tree partitioning of state-space and applies techniques from game-theory and computational geometry to efficiently and adaptively concentrate high resolution only on critical areas. The current version of the algorithm is designed to find feasible paths or trajectories to goal regions in high dimensional spaces. Future versions will be designed to find a solution that optimizes a real-valued criterion. Many simulated problems have been tested, ranging from two-dimensional to nine-dimensional state-spaces, including mazes, path planning, non-linear dynamics, and planar snake robots in restricted spaces. In all cases, a good solution is found in less than ten trials and a few minutes. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #188471. 	[Title] Adaptive state space quantisation for reinforcement learning of collision-free navigation [Abstract] The paper describes a self-learning control system for a mobile robot. Based on sensor information the control system has to provide a steering signal in such a way that collisions are avoided. Since in our case no `examples' are available, the system learns on the basis of an external reinforcement signal which is negative in case of a collision and zero otherwise. Rules from Temporal Difference learning are used to find the correct mapping between the (discrete) sensor input space and the steering signal. We describe the algorithm for learning the correct mapping from the input (state) vector to the output (steering) signal, and the algorithm which is used for a discrete coding of the input state space. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28473. 	[Title] Adaptive Load Balancing: A Study in Multi-Agent Learning [Abstract] We study the process of multi-agent reinforcement learning in the context of load balancing in a distributed system, without use of either central coordination or explicit communication. We first define a precise framework in which to study adaptive load balancing, important features of which are its stochastic nature and the purely local information available to individual agents. Given this framework, we show illuminating results on the interplay between basic adaptive behavior parameters and their effect on system efficiency. We then investigate the properties of adaptive load balancing in heterogeneous populations, and address the issue of exploration vs. exploitation in that context. Finally, we show that naive use of communication may not improve, and might even harm system efficiency. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1103162. 	[Title] Team-Partitioned, Opaque-Transition Reinforcement Learning [Abstract] In this paper, we present a novel multi-agent learning paradigm called team-partitioned, opaque-transition reinforcement learning (TPOT-RL). TPOT-RL introduces the concept of using action-dependent features to generalize the state space. In our work, we use a learned action-dependent feature space. TPOT-RL is an effective technique to allow a team of agents to learn to cooperate towards the achievement of a specific goal. It is an adaptation of traditional RL methods that is applicable in complex, non-Markovian, multi-agent domains with large state spaces and limited training opportunities. Multi-agent scenarios are opaque-transition, as team members are not always in full communication with one another and adversaries may affect the environment. Hence, each learner cannot rely on having knowledge of future state transitions after acting in the world. TPOT-RL enables teams of agents to learn effective policies with very few training examples even in the face of a large state space with large amounts of hidden state. The main responsible features are: dividing the learning task among team members, using a very coarse, action-dependent feature space, and allowing agents to gather reinforcement directly from observation of the environment. TPOT-RL is fully implemented and has been tested in the robotic soccer domain, a complex, multi-agent framework. This paper presents the algorithmic details of TPOT-RL as well as empirical results demonstrating the effectiveness of the developed multi-agent learning approach with learned features. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #259772. 	[Title] Reasoning with Portions of Precedents [Abstract] This paper argues that the task of matching in case-based reasoning can often be improved by comparing new cases to portions of precedents. An example is presented that illustrates how combining portions of multiple precedents can permit new cases to be resolved that would be indeterminate if new cases could only be compared to entire precedents. A system that uses of portions of precedents for legal analysis in the domain of Texas worker's compensation law, GREBE, is described, and examples of GREBE's analysis that combine reasoning steps from multiple precedents are presented. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #10430. 	[Title] Rules and Precedents as Complementary Warrants Complementarity of Rules and Precedents for Classification In a [Abstract] This paper describes a model of the complementarity of rules and precedents in the classification task. Under this model, precedents assist rule-based reasoning by operationalizing abstract rule antecedents. Conversely, rules assist case-based reasoning through case elaboration, the process of inferring case facts in order to increase the similarity between cases, and term reformulation, the process of replacing a term whose precedents only weakly match a case with terms whose precedents strongly match the case. Fully exploiting this complementarity requires a control strategy characterized by impartiality, the absence of arbitrary ordering restrictions on the use of rules and precedents. An impartial control strategy was implemented in GREBE in the domain of Texas worker's compensation law. In a preliminary evaluation, GREBE's performance was found to be as good or slightly better than the performance of law students on the same task. A case is classified as belonging to a particular category by relating its description to the criteria for category membership. The justifications, or warrants [Toulmin, 1958], that can relate a case to a category, can vary widely in the generality of their antecedents. For example, consider warrants for classifying a case into the legal category "negligence." A rule, such as "An action is negligent if the actor fails to use reasonable care and the failure is the proximate cause of an injury," has very general antecedent terms (e.g., "breach of reasonable care"). Conversely, a precedent, such as "Dr. Jones was negligent because he failed to count sponges during surgery and as a result left a sponge in Smith," has very specific antecedent terms (e.g., "failure to count sponges"). Both types of warrants have been used by classification systems to relate cases to categories. Classification systems have used precedents to help match the antecedents of rules with cases. Completing this match is difficult when the terms in the antecedent are open-textured, i.e., when there is significant uncertainty whether they match specific facts [Gardner, 1984, McCarty and Sridharan, 1982]. This problem results from the "generality gap" separating abstract terms from specific facts [Porter et al., 1990]. Precedents of an open-textured term, i.e., past cases to which the term applied, can be used to bridge this gap. Unlike rule antecedents, the antecedents of precedents are at the same level of generality as cases, so no generality gap exists between precedents and new cases. Precedents therefore reduce the problem of matching specific case facts with open-textured terms to the problem of matching two sets of specific facts. For example, an injured employee's entitlement to worker's compensation depends on whether he was injured during an activity "in furtherance of employment." Determining whether any particular case should be classified as a compensable injury therefore requires matching the specific facts of the case (e.g., John was injured in an automobile accident while driving to his office) to the open-textured term "activity in furtherance of employment." The gap in generality between the case description and the abstract term makes this match problematical. However, completing this match may be much easier if there are precedents of the term "activity in furtherance of employment" (e.g., Mary's injury was not compensable because it occurred while she was driving to work, which is not an activity in furtherance of employment; Bill's injury was compensable because it occurred while he was driving to a house to deliver a pizza, an activity in furtherance of employment). In this case, John's driving to his office closely matches Mary's driving to work, so Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #17476. 	[Title] Markov games as a framework for multi-agent reinforcement learning [Abstract] In the Markov decision process (MDP) formalization of reinforcement learning, a single adaptive agent interacts with an environment defined by a probabilistic transition function. In this solipsistic view, secondary agents can only be part of the environment and are therefore fixed in their behavior. The framework of Markov games allows us to widen this view to include multiple adaptive agents with interacting or competing goals. This paper considers a step in this direction in which exactly two agents with diametrically opposed goals share an environment. It describes a Q-learning-like algorithm for finding optimal policies and demonstrates its application to a simple two-player game in which the optimal policy is probabilistic. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6213. 	[Title] Machine Learning Learning to Predict by the Methods of Temporal Differences Keywords: Incremental learning, prediction, [Abstract] This article introduces a class of incremental learning procedures specialized for prediction|that is, for using past experience with an incompletely known system to predict its future behavior. Whereas conventional prediction-learning methods assign credit by means of the difference between predicted and actual outcomes, the new methods assign credit by means of the difference between temporally successive predictions. Although such temporal-difference methods have been used in Samuel's checker player, Holland's bucket brigade, and the author's Adaptive Heuristic Critic, they have remained poorly understood. Here we prove their convergence and optimality for special cases and relate them to supervised-learning methods. For most real-world prediction problems, temporal-difference methods require less memory and less peak computation than conventional methods; and they produce more accurate predictions. We argue that most problems to which supervised learning is currently applied are really prediction problems of the sort to which temporal-difference methods can be applied to advantage. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6214. 	[Title] Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming [Abstract] This paper extends previous work with Dyna, a class of architectures for intelligent systems based on approximating dynamic programming methods. Dyna architectures integrate trial-and-error (reinforcement) learning and execution-time planning into a single process operating alternately on the world and on a learned model of the world. In this paper, I present and show results for two Dyna architectures. The Dyna-PI architecture is based on dynamic programming's policy iteration method and can be related to existing AI ideas such as evaluation functions and universal plans (reactive systems). Using a navigation task, results are shown for a simple Dyna-PI system that simultaneously learns by trial and error, learns a world model, and plans optimal routes using the evolving world model. The Dyna-Q architecture is based on Watkins's Q-learning, a new kind of reinforcement learning. Dyna-Q uses a less familiar set of data structures than does Dyna-PI, but is arguably simpler to implement and use. We show that Dyna-Q architectures are easy to adapt for use in changing environments. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #10183. 	[Title] Concept Learning and Heuristic Classification in Weak-Theory Domains 1 [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6215. 	[Title] Planning by Incremental Dynamic Programming [Abstract] This paper presents the basic results and ideas of dynamic programming as they relate most directly to the concerns of planning in AI. These form the theoretical basis for the incremental planning methods used in the integrated architecture Dyna. These incremental planning methods are based on continually updating an evaluation function and the situation-action mapping of a reactive system. Actions are generated by the reactive system and thus involve minimal delay, while the incremental planning process guarantees that the actions and evaluation function will eventually be optimal|no matter how extensive a search is required. These methods are well suited to stochastic tasks and to tasks in which a complete and accurate model is not available. For tasks too large to implement the situation-action mapping as a table, supervised-learning methods must be used, and their capabilities remain a significant limitation of the approach. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1120713. 	[Title] A Computational Model of Ratio Decidendi [Abstract] This paper proposes a model of ratio decidendi as a justification structure consisting of a series of reasoning steps, some of which relate abstract predicates to other abstract predicates and some of which relate abstract predicates to specific facts. This model satisfies an important set of characteristics of ratio decidendi identified from the jurisprudential literature. In particular, the model shows how the theory under which a case is decided controls its precedential effect. By contrast, a purely exemplar-based model of ratio decidendi fails to account for the dependency of prece-dential effect on the theory of decision. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #10186. 	[Title] Learning Default Concepts [Abstract] Classical concepts, based on necessary and sufficient defining conditions, cannot classify logically insufficient object descriptions. Many reasoning systems avoid this limitation by using "default concepts" to classify incompletely described objects. We address the task of learning such default concepts from observational data. We model the underlying performance task | classifying incomplete examples | by a probabilistic process where random test examples are passed through a "blocker" that can hide object attributes from the classifier. We then address the task of learning accurate default concepts from random training examples. We survey the learning techniques that have been proposed for this task in the machine learning and knowledge representation literatures, investigate the relative merits of each, and show that a superior learning technique can be developed from well known statistical principles. Finally, we extend Valiant's pac-learning framework to this context and obtain a number of useful learnability results. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1109199. 	[Title] Monitoring in Embedded Agents [Abstract] Finding good monitoring strategies is an important process in the design of any embedded agent. We describe the nature of the monitoring problem, point out what makes it difficult, and show that while periodic monitoring strategies are often the easiest to derive, they are not always the most appropriate. We demonstrate mathematically and empirically that for a wide class of problems, the so-called "cupcake problems", there exists a simple strategy, interval reduction, that outperforms periodic monitoring. We also show how features of the environment may influence the choice of the optimal strategy. The paper concludes with some thoughts about a monitoring strategy taxonomy, and what its defining features might be. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #38480. 	[Title] Learning to Achieve Goals [Abstract] Temporal difference methods solve the temporal credit assignment problem for reinforcement learning. An important subproblem of general reinforcement learning is learning to achieve dynamic goals. Although existing temporal difference methods, such as Q learning, can be applied to this problem, they do not take advantage of its special structure. This paper presents the DG-learning algorithm, which learns efficiently to achieve dynamically changing goals and exhibits good knowledge transfer between goals. In addition, this paper shows how traditional relaxation techniques can be applied to the problem. Finally, experimental results are given that demonstrate the superiority of DG learning over Q learning in a moderately large, synthetic, non-deterministic domain. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6224. 	[Title] Tight Performance Bounds on Greedy Policies Based on Imperfect Value Functions [Abstract] Consider a given value function on states of a Markov decision problem, as might result from applying a reinforcement learning algorithm. Unless this value function equals the corresponding optimal value function, at some states there will be a discrepancy, which is natural to call the Bellman residual, between what the value function specifies at that state and what is obtained by a one-step lookahead along the seemingly best action at that state using the given value function to evaluate all succeeding states. This paper derives a tight bound on how far from optimal the discounted return for a greedy policy based on the given value function will be as a function of the maximum norm magnitude of this Bellman residual. A corresponding result is also obtained for value functions defined on state-action pairs, as are used in Q-learning. One significant application of these results is to problems where a function approximator is used to learn a value function, with training of the approxi-mator based on trying to minimize the Bellman residual across states or state-action pairs. When control is based on the use of the resulting value function, this result provides a link between how well the objectives of function approximator training are met and the quality of the resulting control. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #47570. 	[Title] Justification Structures for Document Reuse [Abstract] Document drafting|an important problem-solving task of professionals in a wide variety of fields|typifies a design task requiring complex adaptation for case reuse. This paper proposes a framework for document reuse based on an explicit representation of the illocutionary and rhetorical structure underlying documents. Explicit representation of this structure facilitates (1) interpretation of previous documents by enabling them to "explain themselves," (2) construction of documents by enabling document drafters to issue goal-based specifications and rapidly retrieve documents with similar intentional structure, and (3) mainte nance of multi-generation documents. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #23258. 	[Title] Exploration and Model Building in Mobile Robot Domains [Abstract] I present first results on COLUMBUS, an autonomous mobile robot. COLUMBUS operates in initially unknown, structured environments. Its task is to explore and model the environment efficiently while avoiding collisions with obstacles. COLUMBUS uses an instance-based learning technique for modeling its environment. Real-world experiences are generalized via two artificial neural networks that encode the characteristics of the robot's sensors, as well as the characteristics of typical environments the robot is assumed to face. Once trained, these networks allow for knowledge transfer across different environments the robot will face over its lifetime. COLUMBUS' models represent both the expected reward and the confidence in these expectations. Exploration is achieved by navigating to low confidence regions. An efficient dynamic programming method is employed in background to find minimal-cost paths that, executed by the robot, maximize exploration. COLUMBUS operates in real-time. It has been operating successfully in an office building environment for periods up to hours. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1114336. 	[Title] Four Challenges for a Computational Model of Legal Precedent [Abstract] Identifying the open research issues in a field is a necessary step for progress in that field. This paper describes four open research problems in computational models of precedent-based legal reasoning: relating case representation to precedent use; modeling the selection and construction of both arguments based on pairwise case comparison and multiple-precedent arguments; modeling the process whereby purposes, policies, and principles are used in case similarity assessment; and extending the applicability of precedents to tasks other than classification. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1152740. 	[Title] Model of the Environment to Avoid Local Learning [Abstract] Pier Luca Lanzi Technical Report N. 97.46 December 20 th , 1997 Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1105764. 	[Title] Least-Squares Temporal Difference Learning [Abstract] Submitted to NIPS-98 TD() is a popular family of algorithms for approximate policy evaluation in large MDPs. TD() works by incrementally updating the value function after each observed transition. It has two major drawbacks: it makes inefficient use of data, and it requires the user to manually tune a stepsize schedule for good performance. For the case of linear value function approximations and = 0, the Least-Squares TD (LSTD) algorithm of Bradtke and Barto [5] eliminates all stepsize parameters and improves data efficiency. This paper extends Bradtke and Barto's work in three significant ways. First, it presents a simpler derivation of the LSTD algorithm. Second, it generalizes from = 0 to arbitrary values of ; at the extreme of = 1, the resulting algorithm is shown to be a practical formulation of supervised linear regression. Third, it presents a novel, intuitive interpretation of LSTD as a model-based reinforcement learning technique. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #95589. 	[Title] Planning with Closed-Loop Macro Actions [Abstract] Planning and learning at multiple levels of temporal abstraction is a key problem for artificial intelligence. In this paper we summarize an approach to this problem based on the mathematical framework of Markov decision processes and reinforcement learning. Conventional model-based reinforcement learning uses primitive actions that last one time step and that can be modeled independently of the learning agent. These can be generalized to macro actions, multi-step actions specified by an arbitrary policy and a way of completing. Macro actions generalize the classical notion of a macro operator in that they are closed loop, uncertain, and of variable duration. Macro actions are needed to represent common-sense higher-level actions such as going to lunch, grasping an object, or traveling to a distant city. This paper generalizes prior work on temporally abstract models (Sutton 1995) and extends it from the prediction setting to include actions, control, and planning. We define a semantics of models of macro actions that guarantees the validity of planning using such models. This paper present new results in the theory of planning with macro actions and illustrates its potential advantages in a gridworld task. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #97892. 	[Title] Correcting Imperfect Domain Theories: A Knowledge-Level Analysis [Abstract] Explanation-Based Learning [Mitchell et al., 1986; DeJong and Mooney, 1986] has shown promise as a powerful analytical learning technique. However, EBL is severely hampered by the requirement of a complete and correct domain theory for successful learning to occur. Clearly, in non-trivial domains, developing such a domain theory is a nearly impossible task. Therefore, much research has been devoted to understanding how an imperfect domain theory can be corrected and extended during system performance. In this paper, we present a characterization of this problem, and use it to analyze past research in the area. Past characterizations of the problem (e.g, [Mitchell et al., 1986; Rajamoney and DeJong, 1987]) have viewed the types of performance errors caused by a faulty domain theory as primary. In contrast, we focus primarily on the types of knowledge deficiencies present in the theory, and from these derive the types of performance errors that can result. Correcting the theory can be viewed as a search through the space of possible domain theories, with a variety of knowledge sources that can be used to guide the search. We examine the types of knowledge used by a variety of past systems for this purpose. The hope is that this analysis will indicate the need for a "universal weak method" of domain theory correction, in which different sources of knowledge for theory correction can be freely and flexibly combined. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #4584. 	[Title] References elements that can solve difficult learning control problems. on Simulation of Adaptive Behavior, pages [Abstract] Miller, G. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review, 63(2):81-97. Schmidhuber, J. (1990b). Towards compositional learning with dynamic neural networks. Technical Report FKI-129-90, Technische Universitat Munchen, Institut fu Informatik. Servan-Schreiber, D., Cleermans, A., and McClelland, J. (1988). Encoding sequential structure in simple recurrent networks. Technical Report CMU-CS-88-183, Carnegie Mellon University, Computer Science Department. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1130600. 	[Title] Adaptive state space quantisation: adding and removing neurons [Abstract] This paper describes a self-learning control system for a mobile robot. Based on local sensor data, a robot is taught to avoid collisions with obstacles. The only feedback to the control system is a binary-valued external reinforcement signal, which indicates whether or not a collision has occured. A reinforcement learning scheme is used to find a correct mapping from input (sensor) space to output (steering signal) space. An adaptive quantisation scheme is introduced, through which the discrete division of input space is built up from scratch by the system itself. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #892139. 	[Title] Using Communication to Reduce Locality in Distributed Multi-Agent Learning [Abstract] This paper attempts to bridge the fields of machine learning, robotics, and distributed AI. It discusses the use of communication in reducing the undesirable effects of locality in fully distributed multi-agent systems with multiple agents/robots learning in parallel while interacting with each other. Two key problems, hidden state and credit assignment, are addressed by applying local undirected broadcast communication in a dual role: as sensing and as reinforcement. The methodology is demonstrated on two multi-robot learning experiments. The first describes learning a tightly-coupled coordination task with two robots, the second a loosely-coupled task with four robots learning social rules. Communication is used to 1) share sensory data to overcome hidden state and 2) share reinforcement to overcome the credit assignment problem between the agents and bridge the gap between local/individual and global/group payoff. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #399339. 	[Title] A Comparison of Action Selection Learning Methods [Abstract] Our goal is to develop a hybrid cognitive model of how humans acquire skills on complex cognitive tasks. We are pursuing this goal by designing hybrid computational architectures for the NRL Navigation task, which requires competent senso-rimotor coordination. In this paper, we empirically compare two methods for control knowledge acquisition (reinforcement learning and a novel variant of action models), as well as a hybrid of these methods, with human learning on this task. Our results indicate that the performance of our action models approach more closely approximates the rate of human learning on the task than does reinforcement learning or the hybrid. We also experimentally explore the impact of background knowledge on system performance. By adding knowledge used by the action models system to the benchmark reinforcement learner, we elevate its performance above that of the action models system. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1115886. 	[Title] Evolutionary Differentiation of Learning Abilities a case study on optimizing parameter values in Q-learning by [Abstract] This paper describes the first stage of our study on evolution of learning abilities. We use a simple maze exploration problem designed by R. Sut-ton as the task of each individual, and encode the inherent learning parameters on the genome. The learning architecture we use is a one step Q-learning using look-up table, where the inherent parameters are initial Q-values, learning rate, discount rate of rewards, and exploration rate. Under the fitness measure proportioning to the number of times it achieves at the goal in the later half of life, learners evolve through a genetic algorithm. The results of computer simulation indicated that learning ability emerge when the environment changes every generation, and that the inherent map for the optimal path can be acquired when the environment doesn't change. These results suggest that emergence of learning ability needs environmental change faster than alternate generation. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #27631. 	[Title] Control Systems Magazine, 14, 1, pp.57-71. Robot Juggling: An Implementation of Memory-based Learning [Abstract] This paper explores issues involved in implementing robot learning for a challenging dynamic task, using a case study from robot juggling. We use a memory-based local model - ing approach (locally weighted regression) to represent a learned model of the task to be performed. Statistical tests are given to examine the uncertainty of a model, to optimize its pre diction quality, and to deal with noisy and corrupted data. We develop an exploration algorithm that explicitly deals with prediction accuracy requirements dur ing explo - ration. Using all these ingredients in combination with methods from optimal control, our robot achieves fast real - time learning of the task within 40 to 100 trials. * Address of both authors: Massachusetts Institute of Technology, The Artificial Intelligence Laboratory & The Department of Brain and Cognitive Sciences, 545 Technology Square, Cambride, MA 02139, USA. Email: ss-chaal@ai.mit.edu, cga@ai.mit.edu. Support was provided by the Air Force Office of Sci entific Research and by Siemens Cor pora tion. Support for the first author was provided by the Ger man Scholar ship Foundation and the Alexander von Hum boldt Founda tion. Support for the second author was provided by a Na tional Sci ence Foundation Pre sidential Young Investigator Award. We thank Gideon Stein for im ple ment ing the first version of LWR on the i860 microprocessor, and Gerrie van Zyl for build ing the devil stick robot and implementing the first version of devil stick learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6385. 	[Title] Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents [Abstract] Intelligent human agents exist in a cooperative social environment that facilitates learning. They learn not only by trial-and-error, but also through cooperation by sharing instantaneous information, episodic experience, and learned knowledge. The key investigations of this paper are, "Given the same number of reinforcement learning agents, will cooperative agents outperform independent agents who do not communicate during learning?" and "What is the price for such cooperation?" Using independent agents as a benchmark, cooperative agents are studied in following ways: (1) sharing sensation, (2) sharing episodes, and (3) sharing learned policies. This paper shows that (a) additional sensation from another agent is beneficial if it can be used efficiently, (b) sharing learned policies or episodes among agents speeds up learning at the cost of communication, and (c) for joint tasks, agents engaging in partnership can significantly outperform independent agents although they may learn slowly in the beginning. These tradeoffs are not just limited to multi-agent reinforcement learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1106172. 	[Title] Category: Control, Navigation and Planning. Key words: Reinforcement learning, Exploration, Hidden state. Prefer oral presentation. [Abstract] This paper presents Fringe Exploration, a technique for efficient exploration in partially observable domains. The key idea, (applicable to many exploration techniques), is to keep statistics in the space of possible short-term memories, instead of in the agent's current state space. Experimental results in a partially observable maze and in a difficult driving task with visual routines show dramatic performance improvements. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #117. 	[Title] Generalized Markov Decision Processes: Dynamic-programming and Reinforcement-learning Algorithms [Abstract] The problem of maximizing the expected total discounted reward in a completely observable Markovian environment, i.e., a Markov decision process (mdp), models a particular class of sequential decision problems. Algorithms have been developed for making optimal decisions in mdps given either an mdp specification or the opportunity to interact with the mdp over time. Recently, other sequential decision-making problems have been studied prompting the development of new algorithms and analyses. We describe a new generalized model that subsumes mdps as well as many of the recent variations. We prove some basic results concerning this model and develop generalizations of value iteration, policy iteration, model-based reinforcement-learning, and Q-learning that can be used to make optimal decisions in the generalized model under various assumptions. Applications of the theory to particular models are described, including risk-averse mdps, exploration-sensitive mdps, sarsa, Q-learning with spreading, two-player games, and approximate max picking via sampling. Central to the results are the contraction property of the value operator and a stochastic-approximation theorem that reduces asynchronous convergence to synchronous convergence. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28278. 	[Title] Chapter 1 Reinforcement Learning for Planning and Control [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1129208. 	[Title] Figure 3: Average model size accepted from a ran-dom prefix-closed samples of various size, and [Abstract] that is based on Angluin's L fl algorithm. The algorithm maintains a model consistent with its past examples. When a new counterexample arrives it tries to extend the model in a minimal fashion. We conducted a set of experiments where random automata that represent different strategies were generated, and the algorithm tried to learn them based on prefix-closed samples of their behavior. The algorithm managed to learn very compact models that agree with the samples. The size of the sample had a small effect on the size of the model. The experimental results suggest that for random prefix-closed samples the algorithm behaves well. However, following Angluin's result on the difficulty of learning almost uniform complete samples [ An-gluin, 1978 ] , it is obvious that our algorithm does not solve the complexity issue of inferring a DFA from a general prefix-closed sample. We are currently looking for classes of prefix-closed samples in which US-L* behaves well. [ Carmel and Markovitch, 1994 ] D. Carmel and S. Markovitch. The M* algorithm: Incorporating opponent models into adversary search. Technical Report CIS report 9402, Technion, March 1994. [ Carmel and Markovitch, 1995 ] D. Carmel and S. Markovitch. Unsupervised learning of finite automata: A practical approach. Technical Report CIS report 9504, Technion, March 1995. [ Shoham and Tennenholtz, 1994 ] Y. Shoham and M. Tennenholtz. Co-Learning and the evolution of social activity. Technical Report STAN-CS-TR-94-1511, Stanford Univrsity, Department of Computer Science, 1994. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1122425. 	[Title] Reinforcement Learning with Imitation in Heterogeneous Multi-Agent Systems [Abstract] The application of decision making and learning algorithms to multi-agent systems presents many interestingresearch challenges and opportunities. Among these is the ability for agents to learn how to act by observing or imitating other agents. We describe an algorithm, the IQ-algorithm, that integrates imitation with Q-learning. Roughly, a Q-learner uses the observations it has made of an expert agent to bias its exploration in promising directions. This algorithm goes beyond previous work in this direction by relaxing the oft-made assumptions that the learner (observer) and the expert (observed agent) share the same objectives and abilities. Our preliminary experiments demonstrate significant transfer between agents using the IQ-model and in many cases reductions in training time. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #345340. 	[Title] Generalized Prioritized Sweeping [Abstract] Prioritized sweeping is a model-based reinforcement learning method that attempts to focus an agent's limited computational resources to achieve a good estimate of the value of environment states. To choose effectively where to spend a costly planning step, classic prioritized sweeping uses a simple heuristic to focus computation on the states that are likely to have the largest errors. In this paper, we introduce generalized prioritized sweeping, a principled method for generating such estimates in a representation-specific manner. This allows us to extend prioritized sweeping beyond an explicit, state-based representation to deal with compact representations that are necessary for dealing with large state spaces. We apply this method for generalized model approximators (such as Bayesian networks), and describe preliminary experiments that compare our approach with classical prioritized sweeping. Please summarise all the given information.

Paper #6378 has the title `Learning to coordinate without sharing information  
` and the abstract `Researchers in the field of Distributed Artificial Intelligence (DAI) have been developing efficient mechanisms to coordinate the activities of multiple autonomous agents. The need for coordination arises because agents have to share resources and expertise required to achieve their goals. Previous work in the area includes using sophisticated information exchange protocols, investigating heuristics for negotiation, and developing formal models of possibilities of conflict and cooperation among agent interests. In order to handle the changing requirements of continuous and dynamic environments, we propose learning as a means to provide additional possibilities for effective coordination. We use reinforcement learning techniques on a block pushing problem to show that agents can learn complimentary policies to follow a desired path without any knowledge about each other. We theoretically analyze and experimentally verify the effects of learning rate on system convergence, and demonstrate benefits of using learned coordination knowledge on similar problems. Reinforcement learning based coordination can be achieved in both cooperative and non-cooperative domains, and in domains with noisy communication channels and other stochastic characteristics that present a formidable challenge to using other coordination schemes. 
`.
Question: Which category should Paper #6378 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].


Paper #6378 has the title `Learning to coordinate without sharing information  
` and the abstract `Researchers in the field of Distributed Artificial Intelligence (DAI) have been developing efficient mechanisms to coordinate the activities of multiple autonomous agents. The need for coordination arises because agents have to share resources and expertise required to achieve their goals. Previous work in the area includes using sophisticated information exchange protocols, investigating heuristics for negotiation, and developing formal models of possibilities of conflict and cooperation among agent interests. In order to handle the changing requirements of continuous and dynamic environments, we propose learning as a means to provide additional possibilities for effective coordination. We use reinforcement learning techniques on a block pushing problem to show that agents can learn complimentary policies to follow a desired path without any knowledge about each other. We theoretically analyze and experimentally verify the effects of learning rate on system convergence, and demonstrate benefits of using learned coordination knowledge on similar problems. Reinforcement learning based coordination can be achieved in both cooperative and non-cooperative domains, and in domains with noisy communication channels and other stochastic characteristics that present a formidable challenge to using other coordination schemes. 
`.
Question: Which category should Paper #6378 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].

This paper should be classified into: Reinforcement_Learning