nils-herrmann/clean_scirepeval huggingface dataset:
Sensitive Discount Optimality: Unifying Discounted and Average Reward Reinforcement Learning

Research in reinforcement learning (RL) has thus far concentrated on two optimality criteria: the discounted framework, which has been very well-studied, and the average-reward framework, in which interest is rapidly increasing. In this paper, we present a framework called sensitive discount optimality which ooers an elegant way of linking these two paradigms. Although sensitive discount optimality has been well studied in dynamic programming, with several provably convergent algorithms, it has not received any attention in RL. This framework is based on studying the properties of the expected cumulative discounted reward, as discounting tends to 1. Under these conditions, the cumulative discounted reward can be expanded using a Laurent series expansion to yields a sequence of terms, the rst of which is the average reward, the second involves the average adjusted sum of rewards (or bias), etc. We use the sensitive discount optimality framework to derive a new model-free average reward technique, which is related to Q-learning type methods proposed by Bertsekas, Schwartz, and Singh, but which unlike these previous methods, optimizes both the rst and second terms in the Laurent series (average reward and bias values).  This paper has not been submitted to any other conference.
Topic-based document segmentation with probabilistic latent semantic analysis

This paper presents a new method for topic-based document segmentation, i.e., the identification of boundaries between parts of a document that bear on different topics. The method combines the use of the Probabilistic Latent Semantic Analysis (PLSA) model with the method of selecting segmentation points based on the similarity values between pairs of adjacent blocks. The use of PLSA allows for a better representation of sparse information in a text block, such as a sentence or a sequence of sentences. Furthermore, segmentation performance is improved by combining different instantiations of the same model, either using different random initializations or different numbers of latent classes. Results on commonly available data sets are significantly better than those of other state-of-the-art systems.
Efficient methods for computing investment strategies for multi-market commodity trading

The focus of this work is the computation of efficient strategies for commodity trading in a multimarket environment. In today's "global economy" commodities are often bought in one location and then sold (right away, or after some storage period) in different markets. Thus, a trading decision in one location must be based on expectations about future price curves in all other relevant markets, and on current and future storage and transportation costs. Investors try to compute a strategy that maximizes expected return, usually with some limitations on assumed risk. With standard stochastic assumptions on commodity price fluctuations, computing an optimal strategy can be modeled as a Markov decision process (MDP). However, in general, such a formulation does not lead to efficient algorithms. In this work a model for representing the multimarket trading problem is proposed and how to obtain efficient structured algorithms for computing optimal strategies is shown for a number of commonly used trading objective functions (expected net present value, mean-variance, and value at risk).
Computing Global Strategies for Multi-Market Commodity Trading

The focus of this work is the computation of efficient strategies for commodity trading in a multi-market environment. In today's "global economy" commodities are often bought in one location and then sold (right away, or after some storage period) in different markets. Thus, a trading decision in one location must be based on expectations about future price curves in all other relevant markets, and on current and future storage and transportation costs. Investors try to compute a strategy that maximizes expected return, usually with some limitations on assumed risk. 
 
With standard stochastic assumptions on commodity price fluctuations, computing an optimal strategy can be modeled as a Markov decision process (MDP). However, in general such a formulation does not lead to efficient algorithms. In this work we propose a model for representing the multi-market trading problem and show how to obtain efficient structured algorithms for computing optimal strategies for a number of commonly used trading objective functions (Expected NPV, Mean-Variance, and Value at Risk).
Kevin Maréchal* Centre for Economic and Social Studies on the Environment (CESSE)

In my research I effectively use computers to solve challenging decision-making and control problems. In particular, I focus on developing general tools that enable a machine to learn a solution adaptively. This is a far superior method to executing a hand-coded solution, because most of the problems we face today contain not only complex and stochastic dynamics, but also uncertainty about the dynamics itself, and thus do not admit static off-line solutions.
Least-Squares Policy Iteration

We propose a new approach to reinforcement learning for control problems which combines value-function approximation with linear architectures and approximate policy iteration. This new approach is motivated by the least-squares temporal-difference learning algorithm (LSTD) for prediction problems, which is known for its efficient use of sample experiences compared to pure temporal-difference algorithms. Heretofore, LSTD has not had a straightforward application to control problems mainly because LSTD learns the state value function of a fixed policy which cannot be used for action selection and control without a model of the underlying process. Our new algorithm, least-squares policy iteration (LSPI), learns the state-action value function which allows for action selection without a model and for incremental policy improvement within a policy-iteration framework. LSPI is a model-free, off-policy method which can use efficiently (and reuse in each iteration) sample experiences collected in any manner. By separating the sample collection method, the choice of the linear approximation architecture, and the solution method, LSPI allows for focused attention on the distinct elements that contribute to practical reinforcement learning. LSPI is tested on the simple task of balancing an inverted pendulum and the harder task of balancing and riding a bicycle to a target location. In both cases, LSPI learns to control the pendulum or the bicycle by merely observing a relatively small number of trials where actions are selected randomly. LSPI is also compared against Q-learning (both with and without experience replay) using the same value function architecture. While LSPI achieves good performance fairly consistently on the difficult bicycle task, Q-learning variants were rarely able to balance for more than a small fraction of the time needed to reach the target location.
Efficient approximate policy iteration methods for sequential decision making in reinforcement learning

Reinforcement learning is a promising learning paradigm in which an agent learns how to make good decisions by interacting with an (unknown) environment. This learning framework can be extended along two dimensions: the number of decision makers (single- or multi-agent) and the nature of interaction (collaborative or competitive). This characterization leads to the four decision making situations that are considered in this thesis and are modeled as Markov decision processes, team Markov decision processes, zero-sum Markov games, and team zero-sum Markov games. 
Existing reinforcement learning algorithms have not been applied widely on real-world problems, mainly because the required resources grow fast as a function of the size of the problem. Exact, but impractical, solutions are commonly abandoned in favor of approximate, but practical, solutions. Unfortunately, research on efficient and stable approximate methods has focused mainly on the prediction problem, where an agent tries to learn the outcome of a fixed decision policy. This thesis contributes two efficient and stable algorithms based on the general framework of approximate policy iteration for the control problem, whereby the agent tries to learn a good decision policy. 
Least-Squares Policy Iteration (LSPI) is an algorithm for learning good policies based on a least-squares fixed-point approximation of the value function. LSPI makes efficient use of sample experience and, therefore, is most appropriate for domains where training data are expensive or a simulator of the process is not available. Rollout Classification Policy Iteration (RCPI) on the other hand learns good policies based on the use of rollouts (Monte-Carlo simulation estimates) to train a classifier that represents an approximate policy. For that reason, RCPI is most appropriate for domains where experience comes at no cost or where a simulator is available. Both algorithms exhibit nice theoretical properties, and they bear strong connections to other research areas, such as feature selection and classification learning, respectively. 
The proposed algorithms are demonstrated on a variety of learning tasks: chain walk, inverted pendulum balancing, bicycle balancing and riding, the game of Tetris, multiagent system administration, distributed power grid control, server-router flow control, two-player soccer game, and multiagent server-router flow control. These results demonstrate clearly the efficiency and applicability of the new algorithms on large reinforcement learning control problems.
Reinforcement Learning as Classification: Leveraging Modern Classifiers

The basic tools of machine learning appear in the inner loop of most reinforcement learning algorithms, typically in the form of Monte Carlo methods or function approximation techniques. To a large extent, however, current reinforcement learning algorithms draw upon machine learning techniques that are at least ten years old and, with a few exceptions, very little has been done to exploit recent advances in classification learning for the purposes of reinforcement learning. We use a variant of approximate policy iteration based on rollouts that allows us to use a pure classification learner, such as a support vector machine (SVM), in the inner loop of the algorithm. We argue that the use of SVMs, particularly in combination with the kernel trick, can make it easier to apply reinforcement learning as an "out-of-the-box" technique, without extensive feature engineering. Our approach opens the door to modern classification methods, but does not preclude the use of classical methods. We present experimental results in the pendulum balancing and bicycle riding domains using both SVMs and neural networks for classifiers.
A Listening Keyboard for Users with Motor Impairments—A Usability Study

Computer users with motor impairments find it difficult and, in many cases, impossible to access PC functionality through the physical keyboard-and-mouse interface. Studies show that even able-bodied users experience similar difficulties when interacting with mobile devices; this is due to the reduced size/usability of the input interfaces. Advances in speech recognition have made it possible to design speech interfaces for alphanumeric data entry and indirect manipulation (cursor control). Although several related commercial applications exist, such systems do not provide a complete solution for arbitrary keyboard and mouse access, such as the access needed for, say, typing, compiling, and executing a C++ program.We carried out a usability study to support the development of a speech user interface for arbitrary keyboard access and mouse control. The study showed that speech interaction with an ideal listening keyboard is better for users with motor impairments than handstick, in terms of task completion time (37% better), typing rate (74% better), and error rates (63% better). We believe that these results apply to both permanent and task-induced motor impairments. In particular, a follow-up experiment showed that handstick approximates conventional modes of alphanumeric input available on mobile devices (e.g., PDAs, cellular phones, and personal organizers). These modes of input include miniaturized keyboards, stylus “soft” keyboards, cellular phone numberpads, and handwriting recognition software. This result suggests that a listening keyboard would be an effective mode for alphanumeric input on future mobile devices.This study contributed to the development of SUITEKeys—a speech user interface for arbitrary keyboard and mouse access available for MS platforms as freeware.
Value Function Approximation in Zero-Sum Markov Games

This paper investigates value function approximation in the context of zero-sum Markov games, which can be viewed as a generalization of the Markov decision process (MDP) framework to the two-agent case. We generalize error bounds from MDPs to Markov games and describe generalizations of reinforcement learning algorithms to Markov games. We present a generalization of the optimal stopping problem to a two-player simultaneous move Markov game. For this special problem, we provide stronger bounds and can guarantee convergence for LSTD and temporal difference learning with linear value function approximation. We demonstrate the viability of value function approximation for Markov games by using the Least squares policy iteration (LSPI) algorithm to learn good policies for a soccer domain and a flow control problem.
Coordinated Reinforcement Learning

We present several new algorithms for multiagent reinforcement learning. A common feature of these algorithms is a parameterized, structured representation of a policy or value function. This structure is leveraged in an approach we call coordinated reinforcement learning, by which agents coordinate both their action selection activities and their parameter updates. Within the limits of our parametric representations, the agents will determine a jointly optimal action without explicitly considering every possible action in their exponentially large joint action space. Our methods differ from many previous reinforcement learning approaches to multiagent coordination in that structured communication and coordination between agents appears at the core of both the learning algorithm and the execution architecture.
Learning in Zero-Sum Team Markov Games Using Factored Value Functions

We present a new method for learning good strategies in zero-sum Markov games in which each side is composed of multiple agents collaborating against an opposing team of agents. Our method requires full observability and communication during learning, but the learned policies can be executed in a distributed manner. The value function is represented as a factored linear architecture and its structure determines the necessary computational resources and communication bandwidth. This approach permits a tradeoff between simple representations with little or no communication between agents and complex, computationally intensive representations with extensive coordination between agents. Thus, we provide a principled means of using approximation to combat the exponential blowup in the joint action space of the participants. The approach is demonstrated with an example that shows the efficiency gains over naive enumeration.
Model-Free Least-Squares Policy Iteration

We propose a new approach to reinforcement learning which combines least squares function approximation with policy iteration. Our method is model-free and completely off policy. We are motivated by the least squares temporal difference learning algorithm (LSTD), which is known for its efficient use of sample experiences compared to pure temporal difference algorithms. LSTD is ideal for prediction problems, however it heretofore has not had a straightforward application to control problems. Moreover, approximations learned by LSTD are strongly influenced by the visitation distribution over states. Our new algorithm, Least Squares Policy Iteration (LSPI) addresses these issues. The result is an off-policy method which can use (or reuse) data collected from any source. We have tested LSPI on several problems, including a bicycle simulator in which it learns to guide the bicycle to a goal efficiently by merely observing a relatively small number of completely random trials.
Universal Access to Mobile Computing Devices through Speech Input

This paper presents results on a user interface model for providing universal access to mobile computing devices. The model uses a continuous speech understanding engine to provide access to a virtual keyboard and mouse through speech input. This research has been targeted towards users with permanent motor disabilities. However, these results also apply to able-bodied users with temporary, taskinduced motor disabilities, such as users performing alphanumeric data entry through a cellular phone keypad. The proposed solution might complement (or even replace) miniaturized keyboards and other physical keyboard alternatives, such as stylus-type "soft" keyboards. Since it only requires a microphone (and perhaps a speaker for feedback) which are already included in many mobile devices, it may allow such devices to shrink considerably in size, as alphanumeric input is no longer bound to a physical area. The paper describes the underlying architecture employed by the system. It presents empirical results addressing the effectiveness of this interface over alternative input methods for alphanumeric data entry. Finally, it discusses implications and future directions.
2D DNA self-assembly for satisfiability

DNA self-assembly has been proposed as a way to cope with huge combinatorial NP-HARD problems, such as satis ability. However, the algorithmic designs for DNA self-assembly proposed so far are highly dependent on the instance to be solved. The required work (DNA synthesis, tile construction, encoding, etc.) can be done only after the instance is given. This paper presents an algorithmic design for solving satis ability problems using two-dimensional DNA self-assembly (tiling). The main driving factor in this work was the design and encoding of the algorithm in a general way that minimizes the dependency on particular instances. In e ect, a large amount of work and preparation can be done in advance as a batch process in the absence of any particular instance. In practice, it is likely that the total time (from the time an instance is given, to the time a solution is returned) will be decreased signi cantly and laboratory procedures will be simpli ed. 1 The Satis ability (SAT) Problem The Boolean Satis ability (SAT) problem is the most known representative of the NP-HARD class of problems. The non-polynomial (usually exponential) time required for optimal solutions to these problems, implies that solution of large instances becomes intractably di cult, if not practically impossible. A SAT instance consists of a number of Boolean variables x1; x2; :::; xm and a number of clauses C1; C2; :::; Cn. Each clause is a disjunction of distinct literals, whereby a literal is a single variable xi itself or its complement xi. A solution (satisfying assignment) is an assignment of binary values to the variables xi, such that the conjunction of all clauses is satis ed. Boolean formulas represented in this format are said to be in Conjunction Normal Form (CNF); a conjunction of disjunctions. An example with 5 variables and 8 clauses is given below: (x1 + x2 + x3)( x2 + x4)( x1 + x5)( x1 + x2 + x3 + x4 + x5)( x3)(x2 + x5)( x5)(x1) In this case, the assignment (x1; x2; x3; x4; x5) = (1; 1; 0; 1; 0) is satisfying. So far, there is no restriction on the number of literals that a single clause can contain. However, even if we require that each clause has exactly k literals (k 3), the problem (known as kCNF-SAT) is still intractable. Moreover, it is proven that any SAT instance can be turned into an equivalent kCNFSAT instance [GaJo79]. From these variations, most notable is the 3CNF-SAT variation, because of the small-sized clauses. Notice that for a given number of m variables, there can be m k 2
Neural maps for mobile robot navigation

Neural maps have been recently proposed as an alternative method for mobile robot path planning. However, these proposals are mostly theoretical and primarily concerned with biological plausibility. This paper addresses the applicability of neural maps to mobile robot navigation with focus on efficient implementations It is suggested that neural maps offer a promising alternative compared to the traditional distance transform and harmonic function methods. Applications of neural maps are presented for both global and local navigation. Experimental results (both simulated and real-world on a Nomad 200 mobile robot) demonstrate the validity of the approach. Our work reveals that a key issue for success of the method is the organization of the map that needs to be optimized for the situation at hand.
Robot Navigation with a Polar Neural Map

Center for Advanced Computer StudiesUniversity of Southwestern LouisianaLafayette, LA 70504maida Our purpose is to investigatetheir applicability on real robots.Information about the environment is mapped on atopologically ordered neural population. The diffusiondynamics force the network into a unique equilibriumstate that defines the navigation landscape for the giventarget. A path from any initial position to the target(corresponding to the peak of the activation surface)is derived by a steepest ascent procedure. The figuresbelow show an example on a 50 x 50 rectangular map(a. Environment, b. Contours of activation, c. Path).We attempted to implement the approach on a No-mad 200 mobile robot for sonar-based navigation. How-ever, we found that the neural map requires reorgani-zation in a polar topology that reflects the distributionof the sonar data points, the only source of informationabout the environment. The polar map covers the localcircular area around at the robot. Sonar data points aremapped scaled to the physical robot size. At each stepof the control loop, the dynamics of the map is usedto derive the angular and radial displacement requiredto reach the target from the current configuration. Asimplified example is shown below (bird’s eye view).Sensor uncertainty and noise is handled by a sonarshort-term memory and appropriate coordinate map-ping for reuse. Motion control is based on an optimiza-tion procedure that combines ideas from Fox, Burgard,and Thrun (1997) and Hong et al (1996), and takes account the kinematic and dynamic constraints of therobot. The complete architecture of the resulting local(sensor-based) navigation system is shown below.