nils-herrmann/clean_scirepeval huggingface dataset:
Reinforcement learning on explicitly specified time scales

In recent years hierarchical concepts of temporal abstraction have been integrated in the reinforcement learning framework to improve scalability. However, existing approaches are limited to domains where a decomposition into subtasks is known a priori. In this article we propose the concept of explicitly selecting time scale related abstract actions if no subgoal related abstract actions are available. This concept is realised with multi-step actions on different time scales that are combined in one single action set. We exploit the special structure of the action set in the MSA-Q-learning algorithm. This approach is suited for learning optimal policies in “unstructured” domains where a decomposition into subtasks is not known in advance or does not exist at all. By learning different explicitly specified time scales simultaneously, we achieve a considerable improvement of learning speed, which we demonstrate on several benchmark problems.
Learning to Control at Multiple Time Scales

In reinforcement learning the interaction between the agent and the environment generally takes place on a fixed time scale, which means that the control interval is set to a fixed time step. In order to determine a suitable fixed time scale one has to trade off accuracy in control against learning complexity. In this paper, we present an alternative approach that enables the agent to learn a control policy by using multiple time scales simultaneously. Instead of preselecting a fixed time scale, there are several time scales available during learning and the agent can select the appropriate time scale depending on the system state. The different time scales are multiples of a finest time scale which is denoted as the primitive time scale. Actions on a coarser time scale consist of several identical actions on the primitive time scale and are called multistep actions (MSAs). The special structure of these actions is efficiently exploited in our recently proposed MSA-Q-learning algorithm. In this paper, we use the MSAs to learn a control policy for a thermostat control problem. Our algorithm yields a fast and highly accurate control policy; in contrast, the standard Q-learning algorithms without MSAs fails to learn any useful control policy for this problem.
Learning by Experience from Others — Social Learning and Imitation in Animals and Robots

A challenging current research direction is the design of intelligent software systems — ‘agents’ — that are able to autonomously solve certain tasks within their environment. Application areas of software agents can be found in robotics, as for example agents that control robots to rescue people in dangerous environments, and also in virtual worlds as electronic markets, where intelligent agents have to compete against other market participants, that pursue their own goals.
Brainstormers 2003 - Team Description

The main interest behind the Brainstormers' efiort in the robocup soccer domain is to develop and to apply machine learning tech- niques in complex domains. Especially, we are interested in reinforcement learning methods, where the training signal is only given in terms of suc- cess or failure. Our flnal goal is a learning system, where we only plug in 'win the match' - and our agents learn to generate the appropriate be- haviour. Unfortunately, even from very optimistic complexity estimations it becomes obvious, that in the soccer domain, both conventional solution techniques and also advanced today's reinforcement learning techniques come to their limit - there are more than (108£50) 23 difierent states and more than (1000) 300 difierent policies per agent per half time. This paper describes the new architecture of the Brainstormers team, the improved self-localization using particle fllters and the extensions of the learning algorithm to simultaniously learn with and without ball behaviors.
Speeding-up Reinforcement Learning with Multi-step Actions

In recent years hierarchical concepts of temporal abstraction have been integrated in the reinforcement learning framework to improve scalability. However, existing approaches are limited to domains where a decomposition into subtasks is known a priori. In this paper we propose the concept of explicitly selecting time scale related actions if no subgoalrelated abstract actions are available. This is realised with multistep actions on different time scales that are combined in one single action set. The special structure of the action set is exploited in the MSAQ-learning algorithm. By learning on different explicitly specified time scales simultaneously, a considerable improvement of learning speed can be achieved. This is demonstrated on two benchmark problems.
Design of autonomously learning controllers using FYNESSE

With the growing number and difficulty of control problems there is an increasing demand for design methods that are easy to use. Fynesse fulfils this requirement: without knowledge of a process model the system learns a control policy. Optimization goals like time-optimal or energy-optimal control as well as restrictions of allowed manipulated variables or system states can be defined in a simple and flexible way. Fynesse only learns on basis of success and failure of former control interactions and, thus, learning can be carried out directly at the real process. A priori knowledge about the control policy as, for example, a fuzzy or linear control law considerably improves the learning process. Moreover, the learned policy can be interpreted as fuzzy control law which allows for easily checking the plausibility.
Reinforcement Learning for Cooperating and Communicating Reactive Agents in Electrical Power Grids

Social behaviour in intelligent agent systems is often considered to be achieved by deliberative, in-depth reasoning techniques. This paper shows, how a purely reactive multi-agent system can learn to evolve cooperative behaviour, by means of learning from previous experiences. In particular, we describe a learning multi agent approach to the problem of controlling power flow in an electrical power-grid. The problem is formulated within the framework of dynamic programming. Via a global optimization goal, a set of individual agents is forced to autonomously learn to cooperate and communicate. The ability of the purely reactive distributed systems to solve the global problem by means of establishing a communication mechanism is shown on two prototypical network configurations.
Learning Situation Dependent Success Rates of Actions in a RoboCup Scenario

A quickly changing, not predictable environment complicates autonomous decision making in a system of mobile robots. To simplify action selection we suggest a suitable reduction of decision space by restricting the number of executable actions the agent can choose from. We use supervised neural learning to automaticly learn success rates of actions to facilitate decision making. To determine probabilities of success each agent relies on its sensory data. We show that using our approach it is possible to compute probabilities of success close to the real success rates of actions and further we give a few results of games of a RoboCup simulation team based on this approach. 
 
The RoboCup soccer server offers a couple of low level commands for soccer agents to choose from each 100 ms. Mainly they have the following options: turn (angle), dash (power), kick (power) (angle). If we treat the task of playing soccer as an optimization problem the aim is to control our agents in the given environment such that they score more goals than the opponent team does. We can estimate the number of possible policies by discretising the angle and the power value of the low level commands: Assuming 72 possible angles (5 degree steps) to turn to or to kick to and 10 power levels to dash with or to kick with we get 802 different commands to choose from for a player possessing the ball at one time step. This means we have up to 8023000 different policies over a period of five minutes for only one agent. This forces us to reduce the number of possible choices per time step. To do this we introduce a number of actions such as pass, shoot2goal or go2ball from which the agent can choose. We compute explicit situation dependent success rates for these actions using neural networks (one for each action). Prom all promising actions (estimated success rate exceeds threshold) the one ranked highest in a priority list (shoot2goal is ranked higher than pass...) is chosen to be executed. In order to evaluate our concept we compared estimated success rates with real success rates and played simulation games against different teams. In addition to our statistics our concept was quite successful in official games of our team Karlsruhe Brainstormers against some simulator league teams of 1999. For further information please contact Sebastian Buck.
Karlsruhe Brainstormers 2000 Team Description

The main motivation behind the Karlsruhe Brainstormer’s effort in the robotic soccer simulation league of the RoboCup project is to develop and to apply Reinforcement Learning (RL) techniques in complex domains. Our long term goal is to have a learning system which is able to learn by itself the best winning behaviour. The soccer simulation domain allows more than (108x50)23 different positionings of the 22 players and the ball - the complete state space considering object velocities and player’s stamina is magnitudes larger. In every cycle, an agent can choose between more than 300 basic commands (parametrized turns and dashes), which makes a choice of 30011 joint actions for the team per cycle! A problem of such complexity is a big challenge for today’s RL methods; in the Brainstormers project we are investigating methods to practically handle learning problems of such size.
Karlsruhe Brainstormers - A Reinforcement Learning Approach to Robotic Soccer

Our long-term goal is to build a robot soccer team where the decision making part is based completely on Reinforcement Learning (RL) methods. The paper describes the overall approach pursued by the Karlsruhe Brainstormers simulator league team. Main parts of basic decision making are meanwhile solved using RL techniques. On the tactical level, first empirical results are presented for 2 against 2 attack situations.
Fynesse: a hybrid architecture for self-learning control

|A system is presented that autonomously learns to control a priori unknown nonlinear dynamical systems and enables the user to interpret, examine and correct the control strategy in every stage of learning. Based on ve requirements that are essential to us for a widely applicable learning controller the hybrid control architecture Fynesse is derived: The control strategy is represented by a fuzzy relation that can be interpreted and contain a priori knowledge, whereas the more complex part of learning is solved by a neural network that is trained by dynamic programming methods. The advantages of both paradigms | learning capability of neural networks and interpretability of fuzzy systems | are preserved since the two modules are strictly separated. The application of the Fynesse controller to a chemical plant demonstrates the high quality of autonomously learned control. Furthermore it shows the beneets of the integration of a priori knowledge and the interpretation of the controller in terms of fuzzy rules.
Karlsruhe Brainstormers - Design Principles

The following paper describes the design principles of decision making in the Karlruhe Brainstormers team that participated in the RoboCup Simulator League in Stockholm 1999. It is based on two basic ingredigents: the priority - probability - quality (PPQ) concept is a hybrid rule-based/ learning approach for tactical decisons, whereas the definition of goal-orientented moves allows to apply neural network based reinforcement learning techniques on the lower level.
Using reinforcement learning for engine control

The experiments described are directed towards using reinforcement learning to solve control problems for a combustion engine. The control task presented is to follow an arbitrary sequence of target values for the number of revolutions under the additional condition of keeping the air-to-fuel-ratio close to the optimum by manipulating the system inputs throttle valve angle and fuel injection duration. For this challenging problem of controlling a nonlinear multiple-input-multiple-output system an autonomously learning multi-controller architecture is developed. We also present a comparison to conventional approaches using PI-controllers developed according to the frequently used Ziegler-Nichols parameter adaptation rules.
Approaches for the integration of a priori knowledge into anautonomously learning control

The control architecture Fynesse learns control strategies only on basis of success and failure of former control interactions. Thus, learning can be carried out directly at the real process. For reasons of safety and design costs, the number of interactions with the real process must be kept as small as possible. Therefore Fynesse allows the integration of a priori knowledge about the control strategy. In this paper, diierent techniques for the integration are proposed. In experiments it is shown that the integration of prior control knowledge reduces the number of interactions with the process dramatically. Furthermore the learning procedure is stabilized, which is indispensable for the application of Fynesse as an adaptive system.
Fuzzy-neuro-controlled verified instruction scheduler

We present a fuzzy-neuro approach for instruction scheduling in compilers for modern high performance processors. Instruction scheduling is an optimization problem in NP and is usually addressed with processor dependent heuristics or processor simplifying cost models. The costs for executing a given instruction sequence on the processor can not be determined exactly in practice, because the exact execution model is too complex or simply not available from the manufacturer. Our approach enables the compiler to adapt the cost measure dynamically by learning the processor behavior and typical optimization situations on the basis of reinforcement learning. Additionally, we are able to include fuzzy a priori scheduling knowledge and derive verified implementations by the technique of program checking.
Distributed Value Functions

Many interesting problems, such as power grids, network switches, and tra c ow, that are candidates for solving with reinforcement learning (RL), also have properties that make distributed solutions desirable. We propose an algorithm for distributed reinforcement learning based on distributing the representation of the value function across nodes. Each node in the system only has the ability to sense state locally, choose actions locally, and receive reward locally (the goal of the system is to maximize the sum of the rewards over all nodes and over all time). However each node is allowed to give its neighbors the current estimate of its value function for the states it passes through. We present a value function learning rule, using that information, that allows each node to learn a value function that is an estimate of a weighted sum of future rewards for all the nodes in the network. With this representation, each node can choose actions to improve the performance of the overall system. We demonstrate our algorithm on the distributed control of a simulated power grid. We compare it against other methods including: use of a global reward signal, nodes that act locally with no communication, and nodes that share rewards (but not value function) information with each other. Our results show that the distributed value function algorithm outperforms the others, and we conclude with an analysis of what problems are best suited for distributed value functions and the new research directions opened up by this work.
Concepts and Facilities of a Neural Reinforcement Learning Control Architecture for Technical Process Control

AbstractThe paper presents the concepts of a neural control architecture that is able to learn high quality control behaviour in technical process control from scratch. As the input to the learning system, only the control target must be specified. In the first part of the article, the underlying theoretical principles of dynamic programming methods are explained, and their adaptation to the context of technical process
control is described. The second part discusses the basic capabilities of the learning system on a typical benchmark problem, where a special focus lies on the quality of the acquired control law. The application to a highly nonlinear chemical reactor and to an instable multi-output system shows the ability of the proposed neural control architecture to learn even difficult control strategies from scratch.
Search space reduction for strategy learning in sequential decision processes

Sequential decision making in large domains requires high computational expense. With the classical dynamic programming approach, a rising problem size soon leads to intractability because of time and memory constraints. This situation can be significantly remedied by using more advanced reinforcement learning techniques in combination with generalizing function approximators. However, this may lead to unstable learning behaviour as the strict convergence results are no longer valid. The paper presents an approach to stabilize learning by gradually reducing the search space for the optimal decision policy. This is done by iteratively adapting the action set according to the progress of learning. Experiments are described within the FYNESSE control architecture that is a framework for autonomously learning adaptive control strategies.
A Neural Reinforcement Learning Approach to Learn Local Dispatching Policies in Production Scheduling

Finding optimal solutions for job shop scheduling problems requires high computational effort, especially under consideration of uncertainty and frequent replanning. In contrast to computational solutions, domain experts are often able to derive good local dispatching heuristics by looking at typical problem instances. They can be efficiently applied by looking at few relevant features. However, these rules are usually not optimal, especially in complex decision situations. Here we describe an approach that tries to combine both worlds. A neural network based agent autonomously optimizes its local dispatching policy with respect to a global optimization goal, defined for the overall plant. On two benchmark scheduling problems, we show both learning and generalization abilities of the proposed approach.
Reinforcement learning without an explicit terminal state

Introduces a reinforcement learning framework based on dynamic programming for a class of control problems, where no explicit terminal state exists. This situation especially occurs in the context of technical process control: the control task is not terminated once a predefined target value is reached, but instead the controller has to continue to control the system in order to avoid the system's output drifting away from its target value again. We propose a set of assumptions and give a proof for the convergence of the value iteration method. From this a new algorithm, which we call the fixed horizon algorithm, is derived. The performance of the proposed algorithm is compared to an approach that assumes the existence of an explicit terminal state. The application to a cart/double pole-system finally shows the application to a difficult practical control task.
High Quality Thermostat Control by Reinforcement Learning - A Case Study

High Quality Thermostat Control by Reinforcement Learning A Case Study Martin Riedmiller Institut f ur Logik, Komplexit at und Deduktionssysteme Universit at Karlsruhe, D-76128 Karlsruhe, Germany e-mail: riedml@ira.uka.de Abstract| Temperature control is an important issue in many manufacturing processes. The requirement for high precision, fast reaction to disturbances, time delays of varying length due to considerably changing characteristics of the respective production process make it a challenging application eld for the improvement and development of reinforcement learning techniques. The article shows some rst results on the application of a neural reinforcement learning controller to a thermostat control problem. Open problems are discussed and some ideas for further research directions are presented.  In many manufacturing applications it is important to keep a liquid (water, oil, chemical substance) at a certain temperature. Reasons for this may be that a chemical reaction only has the desired outcome, if the temperature is kept within (very) tight bounds. This is the case for example in wafer production processes, but many more industrial applications exist. They considerably vary with respect to the quality and the amount of the liquids used, resulting in a broad range of di erent process characteristics. This variety makes it very di cult and costly to design a controller that shows good control characteristics in every application situation. Reinforcement learning seems to be a promising approach to overcome this problem by learning to adapt the control law to varying scenarios. A. System description The following hardware structure is a common apparatus for liquid temperature control with a very broad application range ( gure 1): There is a heating device which is used to directly heat a liquid within a smaller internal tank (about 1 liter). This liquid is then pumped through a tube which is going through a larger external tank, thereby emitting energy and thus heating the liquid in the external tank (typically 10 60 liters). The temperature of the liquid in the external tank thus can be controlled by rst heating the internal liquid. The Theat T ext Pump T int Power Fig. 1. Typical hardware structure to control the liquid temperature in the external tank (right) temperature of the external liquid now depends on many parameters: the quality of the internal and the external liquid, the amount of internal liquid that is pumped through the tube per minute, the size of the interval and the external tank, the environment temperature, external disturbances, the quality of the tube, and so on.
A Neural Approach for the Control of Piezoelectric Micromanipulation Robots

Micromanipulation has become an issue of primary importance in industry and biomedicine, since human manual capabilities are restricted to certain tolerances. The manipulation of biological cells or the assembly of a complete microsystem composed of different microcomponents are examples of the application of piezoelectric-driven microrobots. An automated microrobot-based micromanipulation desktop-station is developed by an interdisciplinary group at the University of Karlsruhe. The process of assembly takes place in the field of view of a light optical microscope. This paper focuses on motion control problems of the microrobots. The ability of an intelligent microsystem to adapt itself to the process requirements is of great importance, especially for assembly robots. The microrobots must be able to operate in a partially defined environment and to ensure reasonable behaviour in unpredicted situations. A neural control concept based on a reference model approach is proposed as a solution. It is shown, that the neural controller is able to learn the desired behaviour. It considerably outperforms an analytically designed linear controller. This is demonstrated both in simulation and in the real environment.