Above are papers related to the following paper: [Paper_ID] Paper #118260. 	[Title] Learning Hierarchical Control Structures for Multiple Tasks and Changing Environments [Abstract] While the need for hierarchies within control systems is apparent, it is also clear to many researchers that such hierarchies should be learned. Learning both the structure and the component behaviors is a difficult task. The benefit of learning the hierarchical structures of behaviors is that the decomposition of the control structure into smaller transportable chunks allows previously learned knowledge to be applied to new but related tasks. Presented in this paper are improvements to Nested Q-learning (NQL) that allow more realistic learning of control hierarchies in reinforcement environments. Also presented is a simulation of a simple robot performing a series of related tasks that is used to compare both hierarchical and non-hierarchal learning techniques. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6209. 	[Title] Transfer of Learning by Composing Solutions of Elemental Sequential Tasks [Abstract] Although building sophisticated learning agents that operate in complex environments will require learning to perform multiple tasks, most applications of reinforcement learning have focussed on single tasks. In this paper I consider a class of sequential decision tasks (SDTs), called composite sequential decision tasks, formed by temporally concatenating a number of elemental sequential decision tasks. Elemental SDTs cannot be decomposed into simpler SDTs. I consider a learning agent that has to learn to solve a set of elemental and composite SDTs. I assume that the structure of the composite tasks is unknown to the learning agent. The straightforward application of reinforcement learning to multiple tasks requires learning the tasks separately, which can waste computational resources, both memory and time. I present a new learning algorithm and a modular architecture that learns the decomposition of composite SDTs, and achieves transfer of learning by sharing the solutions of elemental SDTs across multiple composite SDTs. The solution of a composite SDT is constructed by computationally inexpensive modifications of the solutions of its constituent elemental SDTs. I provide a proof of one aspect of the learning algorithm. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #212930. 	[Title] The Functional Transfer of Knowledge for Coronary Artery Disease Diagnosis [Abstract] A distinction between two forms of task knowledge transfer, representational and functional, is reviewed followed by a discussion of MTL, a modified version of the multiple task learning (MTL) neural network method of functional transfer. The MTL method employs a separate learning rate, k , for each task output node k. k varies as a function of a measure of relatedness, R k , between the kth task and the primary task of interest. An MTL network is applied to a diagnostic domain of four levels of coronary artery disease. Results of experiments demonstrate the ability of MTL to develop a predictive model for one level of disease which has superior diagnostic ability over models produced by either single task learning or standard multiple task learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #28485. 	[Title] The Efficient Learning of Multiple Task Sequences [Abstract] I present a modular network architecture and a learning algorithm based on incremental dynamic programming that allows a single learning agent to learn to solve multiple Markovian decision tasks (MDTs) with significant transfer of learning across the tasks. I consider a class of MDTs, called composite tasks, formed by temporally concatenating a number of simpler, elemental MDTs. The architecture is trained on a set of composite and elemental MDTs. The temporal structure of a composite task is assumed to be unknown and the architecture learns to produce a temporal decomposition. It is shown that under certain conditions the solution of a composite MDT can be constructed by computationally inexpensive modifications of the solutions of its constituent elemental MDTs. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6216. 	[Title] Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding [Abstract] On large problems, reinforcement learning systems must use parameterized function approximators such as neural networks in order to generalize between similar situations and actions. In these cases there are no strong theoretical results on the accuracy of convergence, and computational results have been mixed. In particular, Boyan and Moore reported at last year's meeting a series of negative results in attempting to apply dynamic programming together with function approximation to simple control problems with continuous state spaces. In this paper, we present positive results for all the control tasks they attempted, and for one that is significantly larger. The most important differences are that we used sparse-coarse-coded function approximators (CMACs) whereas they used mostly global function approximators, and that we learned online whereas they learned o*ine. Boyan and Moore and others have suggested that the problems they encountered could be solved by using actual outcomes ("rollouts"), as in classical Monte Carlo methods, and as in the TD() algorithm when = 1. However, in our experiments this always resulted in substantially poorer performance. We conclude that reinforcement learning can work robustly in conjunction with function approximators, and that there is little justification at present for avoiding the case of general . Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1108329. 	[Title] Using Case-Based Reasoning as a Reinforcement Learning Framework for Optimization with Changing Criteria [Abstract] Practical optimization problems such as job-shop scheduling often involve optimization criteria that change over time. Repair-based frameworks have been identified as flexible computational paradigms for difficult combinatorial optimization problems. Since the control problem of repair-based optimization is severe, Reinforcement Learning (RL) techniques can be potentially helpful. However, some of the fundamental assumptions made by traditional RL algorithms are not valid for repair-based optimization. Case-Based Reasoning (CBR) compensates for some of the limitations of traditional RL approaches. In this paper, we present a Case-Based Reasoning RL approach, implemented in the C A B I N S system, for repair-based optimization. We chose job-shop scheduling as the testbed for our approach. Our experimental results show that C A B I N S is able to effectively solve problems with changing optimization criteria which are not known to the system and only exist implicitly in a extensional manner in the case base. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #8875. 	[Title] A Coevolutionary Approach to Learning Sequential Decision Rules [Abstract] We present a coevolutionary approach to learning sequential decision rules which appears to have a number of advantages over non-coevolutionary approaches. The coevolutionary approach encourages the formation of stable niches representing simpler sub-behaviors. The evolutionary direction of each subbehavior can be controlled independently, providing an alternative to evolving complex behavior using intermediate training steps. Results are presented showing a significant learning rate speedup over a non-coevolutionary approach in a simulated robot domain. In addition, the results suggest the coevolutionary approach may lead to emer gent problem decompositions. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #38480. 	[Title] Learning to Achieve Goals [Abstract] Temporal difference methods solve the temporal credit assignment problem for reinforcement learning. An important subproblem of general reinforcement learning is learning to achieve dynamic goals. Although existing temporal difference methods, such as Q learning, can be applied to this problem, they do not take advantage of its special structure. This paper presents the DG-learning algorithm, which learns efficiently to achieve dynamically changing goals and exhibits good knowledge transfer between goals. In addition, this paper shows how traditional relaxation techniques can be applied to the problem. Finally, experimental results are given that demonstrate the superiority of DG learning over Q learning in a moderately large, synthetic, non-deterministic domain. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #118259. 	[Title] Emergent Hierarchical Control Structures: Learning Reactive/Hierarchical Relationships in Reinforcement Environments [Abstract] The use of externally imposed hierarchical structures to reduce the complexity of learning control is common. However, it is acknowledged that learning the hierarchical structure itself is an important step towards more general (learning of many things as required) and less bounded (learning of a single thing as specified) learning. Presented in this paper is a reinforcement learning algorithm called Nested Q-learning that generates a hierarchical control structure in reinforcement learning domains. The emergent structure combined with learned bottom-up reactive reactions results in a reactive hierarchical control system. Effectively, the learned hierarchy decomposes what would otherwise be a monolithic evaluation function into many smaller evaluation functions that can be recombined without the loss of previously learned information. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #8213. 	[Title] Robust Reinforcement Learning in Motion Planning [Abstract] While exploring to find better solutions, an agent performing online reinforcement learning (RL) can perform worse than is acceptable. In some cases, exploration might have unsafe, or even catastrophic, results, often modeled in terms of reaching `failure' states of the agent's environment. This paper presents a method that uses domain knowledge to reduce the number of failures during exploration. This method formulates the set of actions from which the RL agent composes a control policy to ensure that exploration is conducted in a policy space that excludes most of the unacceptable policies. The resulting action set has a more abstract relationship to the task being solved than is common in many applications of RL. Although the cost of this added safety is that learning may result in a suboptimal solution, we argue that this is an appropriate tradeoff in many problems. We illustrate this method in the domain of motion planning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #181782. 	[Title] Hierarchical Explanation-Based Reinforcement Learning [Abstract] Explanation-Based Reinforcement Learning (EBRL) was introduced by Dietterich and Flann as a way of combining the ability of Reinforcement Learning (RL) to learn optimal plans with the generalization ability of Explanation-Based Learning (EBL) (Di-etterich & Flann, 1995). We extend this work to domains where the agent must order and achieve a sequence of subgoals in an optimal fashion. Hierarchical EBRL can effectively learn optimal policies in some of these sequential task domains even when the subgoals weakly interact with each other. We also show that when a planner that can achieve the individual subgoals is available, our method converges even faster. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #168410. 	[Title] A Modular Q-Learning Architecture for Manipulator Task Decomposition `Data storage in the cerebellar model ar [Abstract] Compositional Q-Learning (CQ-L) (Singh 1992) is a modular approach to learning to perform composite tasks made up of several elemental tasks by reinforcement learning. Skills acquired while performing elemental tasks are also applied to solve composite tasks. Individual skills compete for the right to act and only winning skills are included in the decomposition of the composite task. We extend the original CQ-L concept in two ways: (1) a more general reward function, and (2) the agent can have more than one actuator. We use the CQ-L architecture to acquire skills for performing composite tasks with a simulated two-linked manipulator having large state and action spaces. The manipulator is a non-linear dynamical system and we require its end-effector to be at specific positions in the workspace. Fast function approximation in each of the Q-modules is achieved through the use of an array of Cerebellar Model Articulation Controller (CMAC) (Albus Our research interests involve the scaling up of machine learning methods, especially reinforcement learning, for autonomous robot control. We are interested in function approximators suitable for reinforcement learning in problems with large state spaces, such as the Cerebellar Model Articulation Controller (CMAC) (Albus 1975) which permit fast, online learning and good local generalization. In addition, we are interested in task decomposition by reinforcement learning and the use of hierarchical and modular function approximator architectures. We are examining the effectiveness of a modified Hierarchical Mixtures of Experts (HME) (Jordan & Jacobs 1993) approach for reinforcement learning since the original HME was developed mainly for supervised learning and batch learning tasks. The incorporation of domain knowledge into reinforcement learning agents is an important way of extending their capabilities. Default policies can be specified, and domain knowledge can also be used to restrict the size of the state-action space, leading to faster learning. We are investigating the use of Q-Learning (Watkins 1989) in planning tasks, using a classifier system (Holland 1986) to encode the necessary condition-action rules. Jordan, M. & Jacobs, R. (1993), Hierarchical mixtures of experts and the EM algorithm, Technical Report 9301, MIT Computational Cognitive Science. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #95579. 	[Title] Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition [Abstract] This paper describes the MAXQ method for hierarchical reinforcement learning based on a hierarchical decomposition of the value function and derives conditions under which the MAXQ decomposition can represent the optimal value function. We show that for certain execution models, the MAXQ decomposition will produce better policies than Feudal Q learning. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #230300. 	[Title] Learning in Continuous Domains with Delayed Rewards [Abstract] Much has been done to develop learning techniques for delayed reward problems in worlds where the actions and/or states are approximated by discrete representations. Although this is acceptable in some applications there are many more situations where such an approximation is difficult and unnatural. For instance, in applications such as robotic,s where real machines interact with the real world, learning techniques that use real valued continuous quantities are required. Presented in this paper is an extension to Q-learning that uses both real valued states and actions. This is achieved by introducing activation strengths to each actuator system of the robot. This allow all actuators to be active to some continuous amount simultaneously. Learning occurs by incrementally adapting both the expected future reward to goal evaluation function and the gradients of that function with respect to each actuator system. Please summarise all the given information.

Paper #118260 has the title `Learning Hierarchical Control Structures for Multiple Tasks and Changing Environments  
` and the abstract `While the need for hierarchies within control systems is apparent, it is also clear to many researchers that such hierarchies should be learned. Learning both the structure and the component behaviors is a difficult task. The benefit of learning the hierarchical structures of behaviors is that the decomposition of the control structure into smaller transportable chunks allows previously learned knowledge to be applied to new but related tasks. Presented in this paper are improvements to Nested Q-learning (NQL) that allow more realistic learning of control hierarchies in reinforcement environments. Also presented is a simulation of a simple robot performing a series of related tasks that is used to compare both hierarchical and non-hierarchal learning techniques. 
`.
Question: Which category should Paper #118260 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].


Paper #118260 has the title `Learning Hierarchical Control Structures for Multiple Tasks and Changing Environments  
` and the abstract `While the need for hierarchies within control systems is apparent, it is also clear to many researchers that such hierarchies should be learned. Learning both the structure and the component behaviors is a difficult task. The benefit of learning the hierarchical structures of behaviors is that the decomposition of the control structure into smaller transportable chunks allows previously learned knowledge to be applied to new but related tasks. Presented in this paper are improvements to Nested Q-learning (NQL) that allow more realistic learning of control hierarchies in reinforcement environments. Also presented is a simulation of a simple robot performing a series of related tasks that is used to compare both hierarchical and non-hierarchal learning techniques. 
`.
Question: Which category should Paper #118260 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].

This paper should be classified into: Reinforcement_Learning