nils-herrmann/clean_scirepeval huggingface dataset:
Local Graph Partitioning as a Basis for Generating Temporally-Extended Actions in Reinforcement Learning

We present a new method for automatically creating useful temporally-extended actions in reinforcement learning. Our method identifies states that lie between two densely-connected regions of the state space and generates temporally-extended actions (e.g., options) that take the agent efficiently to these states. We search for these states using graph partitioning methods on local views of the transition graph. This local perspective is a key property of our algorithms that differentiates it from most of the earlier work in this area, and one that allows it to scale to problems with large
Building a Basic Block Instruction Scheduler with Reinforcement Learning and Rollouts

The execution order of a block of computer instructions on a pipelined machine can make a difference in running time by a factor of two or more. Compilers use heuristic schedulers appropriate to each specific architecture implementation to achieve the best possible program speed. However, these heuristic schedulers are time-consuming and expensive to build. We present empirical results using both rollouts and reinforcement learning to construct heuristics for scheduling basic blocks. In simulation, the rollout scheduler outperformed a commercial scheduler on all benchmarks tested, and the reinforcement learning scheduler outperformed the commercial scheduler on several benchmarks and performed well on the others. The combined reinforcement learning and rollout approach was also very successful. We present results of running the schedules on Compaq Alpha machines and show that the results from the simulator correspond well to the actual run-time results.
Linear Least-Squares algorithms for temporal difference learning

We introduce two new temporal diffence (TD) algorithms based on the theory of linear least-squares function approximation. We define an algorithm we call Least-Squares TD (LS TD) for which we prove probability-one convergence when it is used with a function approximator linear in the adjustable parameters. We then define a recursive version of this algorithm, Recursive Least-Squares TD (RLS TD). Although these new TD algorithms require more computation per time-step than do Sutton's TD(λ) algorithms, they are more efficient in a statistical sense because they extract more information from training experiences. We describe a simulation experiment showing the substantial improvement in learning rate achieved by RLS TD in an example Markov prediction problem. To quantify this improvement, we introduce theTD error variance of a Markov chain, ωTD, and experimentally conclude that the convergence rate of a TD algorithm depends linearly on ωTD. In addition to converging more rapidly, LS TD and RLS TD do not have control parameters, such as a learning rate parameter, thus eliminating the possibility of achieving poor performance by an unlucky choice of parameters.
Reinforcement learning in the real world

Reinforcement learning refers to improving performance through trial-and-error experience. Although, modern computational approaches to reinforcement learning were inspired by animal learning, they have now branched out in a very several different directions. Some researchers are interested in finding high-quality approximate solutions to large-scale stochastic planning problems that are important for industry and government. Others are pursuing the goal of building intelligent, resourceful autonomous agents that, like animals, can succeed while acting in real-time in complex environments. While these goals have much in common - and they both involve the real world - they represent two very different perspectives on reinforcement learning and related methods. After first reviewing the major elements of modern reinforcement learning and its relationship to optimal control, other types of machine learning, and to neuroscience, I present several striking example applications and describe some of the latest research directed toward scaling up to ever more complex problems. I conclude by laying out a view of the future of reinforcement learning research, emphasizing that the issues that need to be addressed depend strongly on which type of reinforcement learning one has in mind.
Supervised ActorCritic Reinforcement Learning

Chapter 7 introduced policy gradients as a way to improve on stochastic search of the policy space when learning. This chapter presents supervised actor-critic reinforcement learning as another method for improving the effectiveness of learning. With this approach, a supervisor adds structure to a learning problem and supervised learning makes that structure part of an actor-critic framework for reinforcement learning. Theoretical background and a detailed algorithm description are provided, along with several examples that contain enough detail to make them easy to understand and possible to duplicate. These examples also illustrate the use of two kinds of supervisors: a feedback controller that is easily designed yet suboptimal, and a human operator providing intermittent control of a simulated robotic arm.
Control, Optimization, Security, and Selfhealing of Benchmark Power SystemsThe views expressed here are those of the authors, Momoh (on leave from Howard University), and Zivi and not the official views of NSF and U.S. Naval Academy, USA

This chapter presents several challenging and benchmark problems from the field of power systems. The first benchmark is the IEEE 118 Bus commercial terrestrial Electrical Power System (EPS). The second benchmark represents a finite inertia hybrid ac/dc shipboard Integrated Power System (IPS). The analytic utility and Navy benchmark models and their respective simulations have been experimentally validated and have been used to determine system reliability, reconfigurability, stability, and security. The challenge is to provide novel control and optimization methods and tools to improve the quality of service despite natural and hostile disruptions under uncertain operating conditions. Along with these problems several smaller problems are also presented which demonstrate different aspects of the challenges of power system control. The purpose of this chapter is formulation of problems that ADP methods could be applied to, therefore the emphasis is more on detailed problem description and simulation, not on any particular solution.
Adaptive Critic Based Neural Network for ControlConstrained Agile Missile

This chapter uses the adaptive critic approach, which was introduced in Chapters 3 and 4, to steer an agile missile with bounds on the angle of attack (control variable) from various initial Mach numbers to a given final Mach number in minimum time while completely reversing its flight path angle. While a typical adaptive critic consists of a critic and controller, the agile missile problem needs chunking in terms of the independent control variable and, therefore, cascades of critics and controllers. Detailed derivations of equations and conditions on the constraint boundary are provided. For numerical experiments, the authors consider vertical plane scenarios. Numerical results demonstrate some attractive features of the adaptive critic approach and show that this formulation works very well in guiding the missile to its final conditions for this state constrained optimization problem from an envelope of initial conditions.
Reinforcement Learning in Large, High‐Dimensional State Spaces

The previous chapter addresses the ?>curse of dimensionality ?> by treating ADP as the dual of the linear programming problem and introduces the method known as approximate linear programming. This chapter presents another method for dealing with the "curse of dimensionality," the policy gradient reinforcement learning framework. The Action Transition Policy Gradient (ATPG) algorithm presented here estimates a gradient in the policy space that increases reward. Following a brief motivation the authors present their algorithm in detail and discuss its properties. Finally, detailed experimental results are presented to show the types of problems that the algorithm can be applied to and what type of performance can be expected. Another algorithm, Boundary Localized Reinforcement Learning, is also discussed in this chapter. This is a mode switching controller that can be used to increase the rate of convergence.
Hierarchical Approaches to Concurrency, Multiagency, and Partial Observability

In this chapter the authors summarize their research in hierarchical probabilistic models for decision making involving concurrent action, multiagent coordination, and hidden state estimation in stochastic environments. A hierarchical model for learning concurrent plans is first described for observable single agent domains, which combines compact state representations with temporal process abstractions to determine how to parallelize multiple threads of activity. A hierarchical model for multiagent coordination is then presented, where primitive joint actions and joint states are hidden. Here, high-level coordination is learned by exploiting overall task structure, which greatly speeds up convergence by abstracting from low-level steps that do not need to be synchronized. Finally, a hierarchical frame-work for hidden state estimation and action is presented, based on multi-resolution statistical modeling of the past history of observations and actions.
An algebraic approach to abstraction in reinforcement learning

To operate effectively in complex environments learning agents ignore irrelevant details. Stated in general terms this is a very difficult problem. Much of the work in this field is specialized to specific modeling framework for Markov decision processes (MDPs) based on homomorphisms relating MDPs. We build on classical finite-state automata literature and develop a minimization framework for MDPs that can exploit structure and symmetries to derive smaller equivalent models of the problem. Since employing homomorphisms approximate and partial homomorphisms and develop bounds for the loss that Our MDP minimization results can be readily employed by reinforcement approach to hierarchical RL, specifically using the options framework. We introduce relativized options, a generalization of Markov sub-goal options, that allow us to define options without an absolute frame of reference. We introduce an extension to the options framework, based on relativized options, that allows us to learn simultaneously at multiple levels of the hierarchy guarantees regarding the performance of hierarchical systems that employ approximate in several test-beds. 
Relativized options can also be interpreted as behavioral schemas. We demonstrate that such schemas can be profitably employed in a hierarchical RL setting. We also develop algorithms that learn the appropriate parameter binding to a given schema. We empirically demonstrate the validity and utility of these algorithms. Relativized options allow us to model certain aspects of deictic or indexical representations. We develop a modification of our parameter binding algorithm suited to hierarchical RL architectures that employ deictic representations.
Reinforcement Learning and Its Relationship to Supervised Learning

The modern study of approximate dynamic programming (DP) combines ideas from several research traditions. Among these is the field of Artificial Intelligence, whose earliest period focussed on creating artificial learning systems. Today, Machine Learning is an active branch of Artificial Intelligence (although it includes researchers from many other disciplines as well) devoted to continuing the development of artificial learning systems. Some of the problems studied in Machine Learning concern stochastic sequential decision processes, and some approaches to solving them are based on DP. These problems and algorithms fall under the general heading of reinforcement learning. In this chapter, we discuss stochastic sequential decision processes from the perspective of Machine Learning, focussing on reinforcement learning and its relationship to the more commmonly studied supervised learning problems. Machine Learning is the study ofmethods for constructing and improving software systems by analyzing examples of their behavior rather than by directly programming them. Machine Learning methods are appropriate in application settings where people are unable to provide precise specifications for desired program behavior, but where examples of desired behavior are available, or where it is possible to assign a measure of goodness to examples of behavior. Such situations include optical character recognition, handwriting recognition, speech recognition, automated steering of automobiles, and robot control and navigation. A key property of tasks in which examples of desired behavior are available is that people can perform them quite easily, but people cannot articulate exactly how they perform them. Hence, people
Cortical involvement in the recruitment of wrist muscles.

In executing a voluntary movement, one is faced with the problem of translating a specification of the movement in task space (e.g., a visual goal) into a muscle-recruitment pattern. Among many brain regions, the primary motor cortex (MI) plays a prominent role in the specification of movements. In what coordinate frame MI represents movement has been a topic of considerable debate. In a two-dimensional wrist step-tracking experiment, Kakei et al. described some MI cells as encoding movement in a muscle-coordinate frame and other cells as encoding movement in an extrinsic-coordinate frame. This result was interpreted as evidence for a cascade of transformations within MI from an extrinsic representation of movement to a muscle-like representation. However, we present a model that demonstrates that, given a realistic extrinsic-like representation of movement, a simple linear network is capable of representing the transformation from an extrinsic space to the muscle-recruitment patterns implementing the movements on which Kakei et al. focused. This suggests that cells exhibiting extrinsic-like qualities can be involved in the direct recruitment of spinal motor neurons. These results call into question models that presume a serial cascade of transformations terminating with MI pyramidal tract neurons that vary their activation exclusively with muscle activity. Further analysis of the model shows that the correlation between the activity of an MI neuron and a muscle does not predict the strength of the connection between the MI neuron and muscle. This result cautions against the use of correlation methods as a measure of cellular connectivity.
Hierarchical Decision Making

As the field of reinforcement learning has advanced, interest in solving realistic control problems has increased. However, Markov Decision Process (MDP) models have not proven sufficient to the task. This has led to increased use of Semi-Markov Decision Process models and the development of Hierarchical Reinforcement Learning (HRL). This chapter is an overview of HRL beginning with a discussion of the problems with the standard MDP models, then presenting the theory behind HRL, and finishing with some actual HRL algorithms that have been proposed. To see some examples of how hierarchical methods perform, see Chapter 11.
Elevator Group Control Using Multiple Reinforcement Learning Agents

Recent algorithmic and theoretical advances in reinforcement learning (RL) have attracted widespread interest. RL algorithms have appeared that approximate dynamic programming on an incremental basis. They can be trained on the basis of real or simulated experiences, focusing their computation on areas of state space that are actually visited during control, making them computationally tractable on very large problems. If each member of a team of agents employs one of these algorithms, a new collective learning algorithm emerges for the team as a whole. In this paper we demonstrate that such collective RL algorithms can be powerful heuristic methods for addressing large-scale control problems.Elevator group control serves as our testbed. It is a difficult domain posing a combination of challenges not seen in most multi-agent learning research to date. We use a team of RL agents, each of which is responsible for controlling one elevator car. The team receives a global reward signal which appears noisy to each agent due to the effects of the actions of the other agents, the random nature of the arrivals and the incomplete observation of the state. In spite of these complications, we show results that in simulation surpass the best of the heuristic elevator control algorithms of which we are aware. These results demonstrate the power of multi-agent RL on a very large scale stochastic dynamic optimization problem of practical utility.
Improved Temporal Difference Methods with Linear Function Approximation

We consider temporal difference algorithms within the context of infinite-horizon finite-state dynamic programming problems with discounted cost, and linear cost function approximation. We show, under standard assumptions, that a least squares-based temporal difference method, proposed by Nedic and Bertsekas (NeB03), converges with a stepsize equal to 1. To our knowl- edge, this is the first iterative temporal difference method that converges without requiring a diminishing stepsize. We discuss the connections of the method with Sutton's TD(λ) and with various versions of least squares-based value iteration, and we show via analysis and experiment that the method is substantially and often dramatically faster than TD(λ), as well as simpler and more reliable. We also discuss the relation of our method with the LSTD method of Boyan (Boy02), and Bradtke and Barto (BrB96).
Reinforcement learning with supervision by a stable controller

Reinforcement learning (RL) methods provide a means for solving optimal control problems when accurate models are unavailable. For many such problems, however, RL alone is impractical and the associated learning problem must be structured somehow to take advantage of prior knowledge. In this paper we examine the use of such knowledge in the form of a stable controller that generates control inputs in parallel with an RL system. The controller acts as a supervisor that not only teaches the RL system about favorable control actions but also protects the learning system from risky behavior. We demonstrate the approach with a simulated robotic arm and a real seven-DOF manipulator.
A Agent Environment StatesActions Rewards Critic B Agent Internal Environment Rewards Critic External Environment Sensations StatesDecisions Actions " Organism " Figure 1

Humans and other animals often engage in activities for their own sakes rather than as steps toward solving practical problems. Psychologists call these intrinsically motivated behaviors. What we learn during intrinsically motivated behavior is essential for our development as competent autonomous entities able to efficiently solve a wide range of practical problems as they arise. In this paper we present initial results from a computational study of intrinsically motivated learning aimed at allowing artificial agents to construct and extend hierarchies of reusable skills that are needed for competent autonomy. At the core of the model are recent theoretical and algorithmic advances in computational reinforcement learning, specifically, new concepts related to skills and new learning algorithms for learning with skill hierarchies.
Toward Dynamic Stochastic Optimal Power Flow

This chapter deals with optimal power flow tools, which must be capable of handling multiple objectives while simultaneously satisfying operational constraints. This chapter contains an overview of the generalized optimal power flow problem, descriptions of the traditional techniques used for optimal power flow and their limitations, and discussion of the current state-of-the-art in robust optimal power flow. Finally, the author presents a generic framework for fitting ADP techniques into the optimal power flow framework. This is presented as an open problem for ADP, and the framework is given so that researchers can test and compare different techniques.
NearOptimal Control Through Reinforcement Learning and Hybridization

This chapter focuses on learning to act in a near-optimal manner through reinforcement learning for problems that either have no model or whose model is very complex. The emphasis here is on continuous action space (CAS) methods. Monte-Carlo approaches are employed to estimate function values in an iterative, incremental procedure. Derivative-free line search methods are used to find a near-optimal action in the continuous action space for a discrete subset of the state space. This near-optimal policy is then extended to the entire continuous state space using a fuzzy additive model. To compensate for approximation errors, a modified procedure for perturbing the generated control policy is developed. Convergence results, under moderate assumptions and stopping criteria, are established. References to sucessful applications of the controller are provided.
Backpropagation Through Time and Derivative Adaptive CriticsA Common Framework for ComparisonPortions of this chapter were previously published in [4, 7,9, 1214,23].

This chapter compares and contrasts derivative adaptive critics (DAC) such as dual heuristic programming (DHP), which was first introduced in Chapter 1 and also discussed in Chapter 3 with back-propagation through time (BPTT). A common framework is built and it is shown that both are techniques for determining the derivatives for training parameters in recurrent neural networks. This chapter goes into sufficient mathematical detail that the reader can understand the theoretical relationship between the two techniques. The author presents a hybrid technique that combines elements of both BPTT and DAC and provides detailed pseudocode. Computational issues and classes of challenging problems are discussed.
Using relative novelty to identify useful temporal abstractions in reinforcement learning

We present a new method for automatically creating useful temporal abstractions in reinforcement learning. We argue that states that allow the agent to transition to a different region of the state space are useful subgoals, and propose a method for identifying them using the concept of relative novelty. When such a state is identified, a temporally-extended activity (e.g., an option) is generated that takes the agent efficiently to this state. We illustrate the utility of the method in a number of tasks.
Recent Advances in Hierarchical Reinforcement Learning

Reinforcement learning is bedeviled by the curse of dimensionality: the number of parameters to be learned grows exponentially with the size of any compact encoding of a state. Recent attempts to combat the curse of dimensionality have turned to principled ways of exploiting temporal abstraction, where decisions are not required at each step, but rather invoke the execution of temporally-extended activities which follow their own policies until termination. This leads naturally to hierarchical control architectures and associated learning algorithms. We review several approaches to temporal abstraction and hierarchical organization that machine learning researchers have recently developed. Common to these approaches is a reliance on the theory of semi-Markov decision processes, which we emphasize in our review. We then discuss extensions of these ideas to concurrent activities, multiagent coordination, and hierarchical memory for addressing partial observability. Concluding remarks address open challenges facing the further development of reinforcement learning in a hierarchical setting.