nils-herrmann/clean_scirepeval huggingface dataset:
Accelerating Reinforcement Learning through the Discovery of Useful Subgoals

An ability to adjust to changing environments and unforeseen circumstances is likely to be an important component of a successful autonomous space robot. This paper shows how to augment reinforcement learning algorithms with a method for automatically discovering certain types of subgoals online. By creating useful new subgoals while learning, the agent is able to accelerate learning on a current task and to transfer its expertise to related tasks through the reuse of its ability to attain subgoals. Subgoals are created based on commonalities across multiple paths to a solution. We cast the task of finding these commonalities as a multiple-instance learning problem and use the concept of diverse density to find solutions. We introduced this approach in  and here we present additional results for a simulated mobile robot task.
Lyapunov-Constrained Action Sets for Reinforcement Learning

Lyapunov analysis is a standard approach to studying the stability of dynamical systems and to designing controllers. We propose to design the actions of a reinforcement learning (RL) agent to be descending on a Lyapunov function. For minimum cost-to-target problems, this has the theoretical benefit of guaranteeing that the agent will reach a goal state on every trial, regardless of the RL algorithm it uses. In practice, Lyapunov-descent constraints can significantly shorten learning trials, improve initial and worst-case performance, and accelerate learning. Although this method of constraining actions may limit the extent to which an RL agent can minimize cost, it allows one to construct robust RL systems for problems in which Lyapunov domain knowledge is available. This includes many important individual problems as well as general classes of problems, such as the control of feedback linearizable systems (e.g., industrial robots) and continuous-state path-planning problems. We demonstrate the general approach on two simulated control problems: pendulum swing-up and robot arm control.
Reinforcement Learning with Stability Guarantees TITLE2

We present a new approach to improving the reliability and overall performance of reinforcement learning (RL) control for minimum cost-to-target problems. In particular, we propose combining RL with Lyapunov-based control, an analytical approach from control theory for designing controllers that make dynamical systems stable. By building Lyapunov-based constraints into a control architecture, we can prove that the controlled system will be brought to a desired target state. At the same time, RL is used to minimize the cost incurred by the controller. The resulting controllers combine theoretical guarantees on their behavior with improved practical performance compared to either standard RL or Lyapunov-based methods alone. Because the controllers ensure stability by design, the guarantees we establish hold during learning as well as after. In fact, the guarantees hold independently of many characteristics of the RL method being used, including the method of function approximation. We illustrate our techniques on a pendulum swing-up task.
Stochastic Scheduling and Planning Using Reinforcement Learning

This project investigated the extension of reinforcement learning (RL) methods to large-scale optimization problems relevant to Air Force operations planning, scheduling, and maintenance. The objectives of this project were to: (1) investigate the utility of RL on large-scale logistics problems; (2) extend existing RL theory and practice to enhance the ease of application and the performance of RL on these problems; and (3) explore new problem formulations in order to take maximal advantage of RL methods. A method using RL to modify local search cost functions was developed and shown to yield significant improvement over a traditional local search method on a core vehicle routing problem. A new method for stochastic dynamic optimization was studied, a theoretical result proven, and utility demonstrated using a simulated aerial mission planning task. A learning-based method for optimizing subproblem selection in divide-and-conquer approaches was developed and demonstrated on graph-coloring and on a multiple-vehicle mission planning task.
Combining Reinforcement Learning with a Local Control Algorithm

We explore combining reinforcement learning with a hand-crafted local controller in a manner suggested by the chaotic control algorithm of Vincent, Schmitt and Vincent (1994). A closedloop controller is designed using conventional means that creates a domain of attraction about a target state. Chaotic behavior is used or induced to bring the system into this region, at which time the local controller is turned on to bring the system to the target state and stabilize it there. We describe experiments in which we use reinforcement learning instead of, and in addition to, chaotic behavior to learn an efficient policy for driving the system into the local controller’s domain of attraction. Using a simulated double pendulum, we illustrate how this method allows reinforcement learning to be effective in a problem that cannot be easily solved by reinforcement learning alone, and we show how reinforcement learning can improve upon the chaotic control algorithm when the domain of attraction can only be approximately determined. Similar results are shown using the Henon map. This is a simple and effective way of extending reinforcement learning to more difficult problems.
Automated State Abstraction for Options using the U-Tree Algorithm

Learning a complex task can be significantly facilitated by defining a hierarchy of subtasks. An agent can learn to choose between various temporally abstract actions, each solving an assigned subtask, to accomplish the overall task. In this paper, we study hierarchical learning using the framework of options. We argue that to take full advantage of hierarchical structure, one should perform option-specific state abstraction, and that if this is to scale to larger tasks, state abstraction should be automated. We adapt McCallum's U-Tree algorithm to automatically build option-specific representations of the state feature space, and we illustrate the resulting algorithm using a simple hierarchical task. Results suggest that automated option-specific state abstraction is an attractive approach to making hierarchical learning systems more effective.
Machine Learning for Subproblem Selection

Subproblem generation, solution, and recombination is a standard approach to combinatorial optimization problems. In many settings identifying suitable subproblems is itself a significant component of the technique. Such subproblems are often identified using a heuristic rule. Here we show how to use machine learning to make this identification. In particular we use a learned objective function to direct search in an appropriate space of subproblem decompositions. We demonstrate the efficacy of our technique for problem decomposition on a particular wellknown combinatorial optimization problem, graph coloring for geometric graphs.
Learning and control in a chaotic system

Most real world problems can be solved at least partially by simple, nonadaptive means. When we want to use an adaptive learning technique like reinforcement learning for practical purposes, we need to be able to cooperate with one or several of these partial solutions. In this paper we consider the problem of making a reinforcement learning agent cooperate with a hand-crafted local controller and a global chaotic controller, and designing a regime that improves upon these controller.
Learned Subproblem Selection Techniques for Combinatorial Optimization

Subproblem generation, solution, and recombination is a standard solution method for combinatorial optimization problems. In many settings identifying suitable subproblems is itself a significant component of the method. Such subproblems are often identified using a heuristic rule. Here we show how to use machine learning to make this identification. In particular we use a learned objective function to direct search in an appropriate space of subproblem decompositions. We demonstrate the efficacy of our technique for problem decomposition on two examples: graph coloring for geometric graphs, a deterministic optimization problem, and the Multiple Uninhabited Air Vehicle Surveillance Control Problem, or MUAV, a stochastic optimization problem that is related to vehicle routing problems in operations research.
A Cerebellar Model of Timing and Prediction in the Control of Reaching

A simplified model of the cerebellum was developed to explore its potential for adaptive, predictive control based on delayed feedback information. An abstract representation of a single Purkinje cell with multistable properties was interfaced, using a formalized premotor network, with a simulated single degree-of-freedom limb. The limb actuator was a nonlinear spring-mass system based on the nonlinear velocity dependence of the stretch reflex. By including realistic mossy fiber signals, as well as realistic conduction delays in afferent and efferent pathways, the model allowed the investigation of timing and predictive processes relevant to cerebellar involvement in the control of movement. The model regulates movement by learning to react in an anticipatory fashion to sensory feedback. Learning depends on training information generated from corrective movements and uses a temporally asymmetric form of plasticity for the parallel fiber synapses on Purkinje cells.
Learning to reach via corrective movements

When infants begin to perform goal-directed reaching, the kinematics of their reaches show multiple accelerations and decelerations of the hand, which appear to reflect a correcting series of submovements. Although each submovement is an inaccurate correction, the sequence of submovements is often successful in reaching the target. Under some circumstances, such as when learning a novel task or when high accuracy is required, adult behavior also consists of a series of submovements that appear to be corrective in nature. We investigate the hypothesis that this reflects the action of two primary processes: one that regulates the metric properties of the submovements, and one that initiates them at the proper times and in roughly in the correct directions. We hypothesize that the cerebellum is the key structure underlying the first of these processes. The cerebellum is modeled as an adaptive, predictive controller that regulates movement by learning to react in an anticipatory fashion to sensory feedback.
Reinforcement learning in motor control

How do we learn motor skills such as reaching, walking, swimming, or riding a bicycle? There is a large literature on motor skill acquisition which is full of controversies (for an introduction to human motor control, see Schmidt and Lee, 1999), but there is general agreement that motor learning requires the learner, human or not, to receive response-produced feedback through various senses providing information about performance. Careful consideration of the nature of the feedback used in learning is important for understanding the role of reinforcement learning in motor control (see REINFORCEMENT LEARNING). One function of feedback is to guide the performance of movements. This is the kind of feedback with which we are familiar from control theory, where it is the basis of servo control, although its role in guiding animal movement is more complex. Another function of feedback is to provide information useful for improving subsequent movement. Feedback having this function has been called learning feedback . Note that this functional distinction between feedback for control and for learning does not mean that the signals or channels serving these functions need to be different.
Fagg, Barto & Houk: Learning to Reach via Corrective Movements; Tenth Yale Workshop on Adaptive and Learning Systems Learning to Reach via Corrective Movements

When infants begin to perform goal-directed reaching between approximately 4 and 5 months-of-age, the kinematics of their reaches show multiple accelerations and decelerations of the hand, which appear to reeect a correcting series of submovements. Although each sub-movement is an inaccurate correction, the sequence of submovements is often successful in reaching the target. By what process does the infant's reach controller come to eventually produce adult-like reaching behavior, characterized by bell-shaped velocity prooles and only occasional corrections? We suggest that motor command parameters are tuned in a supervised learning fashion using training information derived from proprioceptive feedback received during corrective movements. Although this is not high-quality training information, it is suucient to tune motor command parameters in order to bring a dynamic arm quickly and accurately to the target, as we show in a series of simulations. We also suggest that additional parameters can be tuned to satisfy constraints other than end-point constraints through a reinforcement learning process.
Reinforcement Learning and Local Search: a Case Study Reinforcement Learning and Local Search: a Case Study

We describe a reinforcement learning-based variation to the combinatorial optimization technique known as local search. The hillclimbing aspect of local search uses the problem's primary cost function to guide search via local neighborhoods to high quality solutions. In complicated optimization problems, however, other problem characteristics can also help guide the search process. In this report we present an approach to constructing more general, derived, cost functions for combinatorial optimization problems using reinforcement learning. Such derived cost functions integrate a variety of problem characteristics into a single hillclimbing function. We illustrate our technique by developing several such functions for the Dial-A-Ride Problem, a variant of the well-known Traveling Salesman Problem.
A model of cerebellar learning for control of arm movements using muscle synergies

Biological control systems have long been studied as possible inspiration for the construction of robotic controllers. The cerebellum is known to be involved in the production and learning of smooth, coordinated movements. In this paper, we present a model of cerebellar control of a muscle-actuated, two-link, planar arm. The model learns in a trial-and-error fashion to generate the appropriate sequence of motor signals that accurately bring the arm to a specified target. The motor signals produced by the cerebellum are specified in muscle synergy space. When the cerebellum fails to bring the arm to the target, an extra-cerebellar module performs low-quality corrective movements, from which the cerebellum updates its program. In learning to perform the task, the cerebellum constructs an implicit inverse model of the plant. This model uses a combination of delayed sensory signals and recently-generated motor commands to compute the new output motor signal.
Cerebellar learning for control of a two-link arm in muscle space

Biological control systems have long been studied as possible inspiration for the construction of robotic controllers. The cerebellum is known to be involved in the production and learning of smooth, coordinated movements. In this paper, we present a model of cerebellar control of a muscle-actuated, two-link, planar arm. The model learns in a trial-and-error fashion to produce bursts of muscle activity that accurately bring the arm to a specified target. When the cerebellum fails to bring the arm to the target, an extra-cerebellar module performs four-quality corrective movements, from which the cerebellum may update its program. In learning to perform the task, the cerebellum constructs an implicit inverse model of the plant. This model uses a combination of delayed sensory signals and recently-generated motor commands to compute the new output motor signal.
Automated Aircraft Recovery via Reinforcement Learning: Initial Experiments

Initial experiments described here were directed toward using reinforcement learning (RL) to develop an automated recovery system (ARS) for high-agility aircraft. An ARS is an outer-loop flight-control system designed to bring an aircraft from a range of out-of-control states to straight-and-level flight in minimum time while satisfying physical and physiological constraints. Here we report on results for a simple version of the problem involving only single-axis (pitch) simulated recoveries. Through simulated control experience using a medium-fidelity aircraft simulation, the RL system approximates an optimal policy for pitch-stick inputs to produce minimum-time transitions to straight-and-level flight in unconstrained cases while avoiding ground-strike. The RL system was also able to adhere to a pilot-station acceleration constraint while executing simulated recoveries.
Prediction of complex two-dimensional trajectories by a cerebellar model of smooth pursuit eye movement.

A neural network model based on the anatomy and physiology of the cerebellum is presented that can generate both simple and complex predictive pursuit, while also responding in a feedback mode to visual perturbations from an ongoing trajectory. The model allows the prediction of complex movements by adding two features that are not present in other pursuit models: an array of inputs distributed over a range of physiologically justified delays, and a novel, biologically plausible learning rule that generated changes in synaptic strengths in response to retinal slip errors that arrive after long delays. To directly test the model, its output was compared with the behavior of monkeys tracking the same trajectories. There was a close correspondence between model and monkey performance. Complex target trajectories were created by summing two or three sinusoidal components of different frequencies along horizontal and/or vertical axes. Both the model and the monkeys were able to track these complex sum-of-sines trajectories with small phase delays that averaged 8 and 20 ms in magnitude, respectively. Both the model and the monkeys showed a consistent relationship between the high- and low-frequency components of pursuit: high-frequency components were tracked with small phase lags, whereas low-frequency components were tracked with phase leads. The model was also trained to track targets moving along a circular trajectory with infrequent right-angle perturbations that moved the target along a circle meridian. Before the perturbation, the model tracked the target with very small phase differences that averaged 5 ms. After the perturbation, the model overshot the target while continuing along the expected nonperturbed circular trajectory for 80 ms, before it moved toward the new perturbed trajectory. Monkeys showed similar behaviors with an average phase difference of 3 ms during circular pursuit, followed by a perturbation response after 90 ms. In both cases, the delays required to process visual information were much longer than delays associated with nonperturbed circular and sum-of-sines pursuit. This suggests that both the model and the eye make short-term predictions about future events to compensate for visual feedback delays in receiving information about the direction of a target moving along a changing trajectory. In addition, both the eye and the model can adjust to abrupt changes in target direction on the basis of visual feedback, but do so after significant processing delays.
Reinforcement Learning for Mixed Open-loop and Closed-loop Control

Closed-loop control relies on sensory feedback that is usually assumed to be free. But if sensing incurs a cost, it may be cost-effective to take sequences of actions in open-loop mode. We describe a reinforcement learning algorithm that learns to combine open-loop and closed-loop control when sensing incurs a cost. Although we assume reliable sensors, use of open-loop control means that actions must sometimes be taken when the current state of the controlled system is uncertain. This is a special case of the hidden-state problem in reinforcement learning, and to cope, our algorithm relies on short-term memory. The main result of the paper is a rule that significantly limits exploration of possible memory states by pruning memory states for which the estimated value of information is greater than its cost. We prove that this rule allows convergence to an optimal policy.
Text-Based Information Retrieval Using Exponentiated Gradient Descent

The following investigates the use of single-neuron learning algorithms to improve the performance of text-retrieval systems that accept natural-language queries. A retrieval process is explained that transforms the natural-language query into the query syntax of a real retrieval system: the initial query is expanded using statistical and learning techniques and is then used for document ranking and binary classification. The results of experiments suggest that Kivinen and Warmuth's Exponentiated Gradient Descent learning algorithm works significantly better than previous approaches.
More models of the cerebellum

Neural models are needed at many levels, ranging from the biophysically based to connectionist abstractions. The adjustable pattern generator (APG) model of the cerebellum discussed in the target article by Houk, Buckingham, and Barto operates at an intermediate level and is constrained by a large body of anatomical and physiological data. The associative memory model originally proposed by Marr (1969) and Albus (1971) on the basis of the microcircuitry of the cerebellar cortex is still relevant today, although it needs to be coupled to the problem of motor pattern generation. In considering the problem of cerebellar learning, one needs some source of training information to adjust network performance so that it performs fruitful interactions with the environment; however, it is not true that this “teacher” needs to be particularly smart. We address these and other important issues raised by the commentators.
Local Bandit Approximation for Optimal Learning Problems

In general, procedures for determining Bayes-optimal adaptive controls for Markov decision processes (MDP's) require a prohibitive amount of computation-the optimal learning problem is intractable. This paper proposes an approximate approach in which bandit processes are used to model, in a certain "local" sense, a given MDP. Bandit processes constitute an important subclass of MDP's, and have optimal learning strategies (defined in terms of Gittins indices) that can be computed relatively efficiently. Thus, one scheme for achieving approximately-optimal learning for general MDP's proceeds by taking actions suggested by strategies that are optimal with respect to local bandit models.
Multi-Agent Reinforcement Learning and Adaptive Neural Networks.

This project investigated learning systems consisting of multiple interacting controllers, or agents: each of which employed a modern reinforcement learning method. The objective was to study the utility of reinforcement learning as an approach to complex decentralized control problems. The major accomplishment was a detailed study of multi-agent reinforcement learning applied to a large-scale decentralized stochastic control problem. This study included a very successful demonstration that a multi-agent reinforcement learning system using neural networks could learn high-performance dispatching of multiple elevator cars in a simulated multi-story building. This problem is representative of very large-scale dynamic optimization problems of practical importance that are intractable for standard methods. The performance achieved by the distributed elevator controller surpassed that of the best of the elevator control algorithms accessible in the literature, showing that reinforcement learning can be a useful approach to difficult decentralized control problems. Additional empirical results demonstrated the performance of reinforcement learning-systems in the setting of nonzero-sum games, with mixed results. Some progress was also made in improving theoretical understanding of multi-agent reinforcement learning.