nils-herrmann/clean_scirepeval huggingface dataset:
Large-scale dynamic optimization using teams of reinforcement learning agents

Recent algorithmic and theoretical advances in reinforcement learning (RL) are attracting widespread interest. RL algorithms have appeared that approximate dynamic programming (DP) on an incremental basis. Unlike traditional DP algorithms, these algorithms do not require knowledge of the state transition probabilities or reward structure of a system. This allows them to be trained using real or simulated experiences, focusing their computations on the areas of state space that are actually visited during control, making them computationally tractable on very large problems. RL algorithms can be used as components of multi-agent algorithms. If each member of a team of agents employs one of these algorithms, a new collective learning algorithm emerges for the team as a whole. In this dissertation we demonstrate that such collective RL algorithms can be powerful heuristic methods for addressing large-scale control problems. 
Elevator group control serves as our primary testbed. The elevator domain poses a combination of challenges not seen in most RL research to date. Elevator systems operate in continuous state spaces and in continuous time as discrete event dynamic systems. Their states are not fully observable and they are non-stationary due to changing passenger arrival rates. As a way of streamlining the search through policy space, we use a team of RL agents, each of which is responsible for controlling one elevator car. The team receives a global reinforcement signal which appears noisy to each agent due to the effects of the actions of the other agents, the random nature of the arrivals and the incomplete observation of the state. In spite of these complications, we show results that in simulation surpass the best of the heuristic elevator control algorithms of which we are aware. These results demonstrate the power of RL on a very large scale stochastic dynamic optimization problem of practical utility.
Linear Least-Squares Algorithms for Temporal Difference Learning

We introduce two new temporal difference (TD) algorithms based on the theory of linear least-squares function approximation. We define an algorithm we call Least-Squares TD (LS TD) for which we prove probability-one convergence when it is used with a function approximator linear in the adjustable parameters. We then define a recursive version of this algorithm, Recursive Least-Square TD (RLS TD). Although these new TD algorithms require more computation per time-step than do Sutton‘s TD(λ) algorithms, they are more efficient in a statistical sense because they extract more information from training experiences. We describe a simulation experiment showing the substantial improvement in learning rate achieved by RLS TD in an example Markov prediction problem. To quantify this improvement, we introduce the TD error variance of a Markov chain, σTD, and experimentally conclude that the convergence rate of a TD algorithm depends linearly on σTD. In addition to converging more rapidly, LS TD and RLS TD do not have control parameters, such as a learning rate parameter, thus eliminating the possibility of achieving poor performance by an unlucky choice of parameters.
Models of the cerebellum and motor learning

This article reviews models of the cerebellum and motor learning starting with the landmark papers by Marr and Albus and going through present times. The unique architecture of the cerebellar cortex is ideally suited for pattern recognition, but how is pattern recognition incorporated into motor control and learning systems? The present analysis begins with a discussion of what the cerebellar cortex needs to regulate through its anatomically defined projections to premotor networks. Next we examine various models of how the microcircuitry in the cerebellar cortex may be used to achieve its regulatory functions. Having thus defined what it is that Purkinje cells in the cerebellar cortex must learn, we then evaluate theories of motor learning. We examine current models of synaptic plasticity, credit assignment and the generation of training information, indicating how they could function cooperatively to guide the processes of motor learning. Lesion studies carried out in the 19th century demonstrated that the cerebellum is important for coordinating movements (Florens, 1824). However, mechanistic models of the cerebellum awaited an analysis of the histology of the cerebellum (Braitenberg & Atwood, 1958), and combined analyses of histology and electrophysiology (Marr, 1969; Albus, 1971). The clear orthogonal relationships between parallel and climbing fibers and the dendritic trees of Purkinje cells convinced Braitenberg that the cerebellum functions as a timing organ. He viewed the parallel fibers as delay lines and climbing fibers as clock read-out mechanisms, with Purkinje cells firing only when there was a coincidence of a parallel fiber volley and a climbing fiber activation. In one of his examples, a Purkinje cell innervating an antagonist muscle fired at an appropriate time delay to terminate a movement at its intended target. Even though parallel fibers have small diameters and low conduction velocities, the time delays only amount to a few milliseconds, rather short for most problems in motor control. Another limitation of this theory is its requirement for foci of synchronized activation in the granular layer. For these and other reasons, Braitenberg’ s timing theory has not been as vigorously pursued as the learned pattern recognition theories developed subsequently by Marr (1969) and Albus (1971).
A Predictive Switching Model of Cerebellar Movement Control

We present a hypothesis about how the cerebellum could participate in regulating movement in the presence of significant feedback delays without resorting to a forward model of the motor plant. We show how a simplified cerebellar model can learn to control end-point positioning of a nonlinear spring-mass system with realistic delays in both afferent and efferent pathways. The model's operation involves prediction, but instead of predicting sensory input, it directly regulates movement by reacting in an anticipatory fashion to input patterns that include delayed sensory feedback.
Adaptive Critics and the Basal Ganglia

One of the most active areas of research in artificial intelligence is the study of learning methods by which “embedded agents” can improve performance while acting in complex dynamic environments. An agent, or decision maker, is embedded in an environment when it receives information from, and acts on, that environment in an ongoing closed-loop interaction. An embedded agent has to make decisions under time pressure and uncertainty and has to learn without the help of an ever-present knowledgeable teacher. Although the novelty of this emphasis may be inconspicuous to a biologist, animals being the prototypical embedded agents, this emphasis is a significant departure from the more traditional focus in artificial intelligence on reasoning within circumscribed domains removed from the flow of real-world events. One consequence of the embedded agent view is the increasing interest in the learning paradigm called reinforcement learning (RL). Unlike the more widely studied supervised learning systems, which learn from a set of examples of correct input/output behavior, RL systems adjust their behavior with the goal of maximizing the frequency and/or magnitude of the reinforcing events they encounter over time.
Improving Elevator Performance Using Reinforcement Learning

This paper describes the application of reinforcement learning (RL) to the difficult real world problem of elevator dispatching. The elevator domain poses a combination of challenges not seen in most RL research to date. Elevator systems operate in continuous state spaces and in continuous time as discrete event dynamic systems. Their states are not fully observable and they are nonstationary due to changing passenger arrival rates. In addition, we use a team of RL agents, each of which is responsible for controlling one elevator car. The team receives a global reinforcement signal which appears noisy to each agent due to the effects of the actions of the other agents, the random nature of the arrivals and the incomplete observation of the state. In spite of these complications, we show results that in simulation surpass the best of the heuristic elevator control algorithms of which we are aware. These results demonstrate the power of RL on a very large scale stochastic dynamic optimization problem of practical utility.
The Topographic Product of Experts

We create a new form of topographic map which is based on a nonlinear mapping of a space of latent points. The mapping of these latent points into data space creates centres which are equivalent to those of the standard SOM. We relate this mapping to the Generative Topographic Mapping, GTM. We then show that it is rather simple and computationally inexpensive to grow one of these maps and that a probabilistic interpretation of these maps facilitates our investigation of alternative algorithms.
Training an AI Player to play Pong Using the GTM

We extend the work of [McGlinchey 2003], in which the author trained an AI player to play Pong from game observation data recorded from games played by humans. The data trained a Self Organising Map (SOM) , and it was found that the AI player played Pong with a human style of play. However one of the drawback of using the SOM was that the movement of the bat was jerky, due to quantisation of vectors. The author had to applying smoothing to the AI player’s bat to make the movement more realistic. It was also found that the AI players were easy to beat. In this paper we train the AI player using Generative Topographic Mapping (GTM) and we show that using the mean of the conditional density to estimate the bat’s position is better than using the mode.
Matching the dimensionality of maps with that of the data

Topographic maps are low dimensional maps which retain some topology of the original dataset. Many topographic mappings suffer from having the dimensionality of the map determined beforehand which is certain to be inappropriate for some data sets. In this paper, we develop a method of investigating a data set enabling the local dimensionality of the map to change. Our model of the data allows us to traverse the main manifold on which the data lies while giving information about the local dimensionality of the data around this main manifold.
Motocross and Artificial Neural Networks

In this paper, we investigate training artificial neural networks to ride simulated motorbikes in a new computer game using two different training techniques, Evolutionary Algorithms and the Backpropagation Algorithm. We show that the backpropagation algorithm creates a rider which is faster than that created by the evolutionary algorithm but at the price of requiring a training set created by a human playing the game. Also the evolutionary algorithm has the advantage that it can find solutions which no human has previously found. Both methods create human-like performance in the motocross game.
The combined use of self-organizing maps and andrews' curves

The use of self-organizing maps to analyze data often depends on finding effective methods to visualize the SOM's structure. In this paper we propose a new way to perform that visualization using a variant of Andrews' Curves. Also we show that the interaction between these two methods allows us to find sub-clusters within identified clusters. Perhaps more importantly, using the SOM to pre-process data by identifying gross features enables us to use Andrews' Curves on data sets which would have previously been too large for the methodology. Finally we show how a three way interaction between the human user and these two methods can be a valuable exploratory data analysis tool.
4 Artificial Neural Networks

In this chapter, we give an overview of the most common artificial neural networks which are being used today. Perhaps the crucial thing which differentiates modern searches for artificial intelligence from those of the past is that modern approaches tend to emphasise adaptivity. This is done in artificial neural networks with learning and we define two of the major approaches to learning as supervised (or learning with a teacher) and unsupervised. We illustrate both techniques with standard methods. Now the artificial neural networks which are in use today are not truly “smart adaptive systems”; even though they are based on models of a physical entity – the brain – which is truly a smart adaptive system, something seems to be missing. We conjecture that a combination of some of our modern searches for intelligence might be closer to a smart adaptive system and so we investigate a combination of statistical and neural techniques at the conclusion of this chapter.
Visualization of High-dimensional Data via Orthogonal Curves

Computers are still much less useful than the ability of the human eye for pattern matching. This ability can be used quite straightforwardly to identify structure in a data set when it is two or three dimensional. With data sets with more than 3 dimensions some kind of transformation is always necessary. In this paper we review in depth and present and extension of one of these mechanisms: Andrews' curves. With the Andrews' curves we use a curve to represent each data point. A human can run his eye along a set of curves (representing the members of the data set) and identify particular regions of the curves which are optimal for identifying clusters in the data set. Of interest in this context, is our extension in which a moving three-dimensional image is created in which we can see clouds of data points moving as we move along the curves; in a very real sense, the data which dance together are members of the same cluster.
Faster clustering of complex data with the generalised harmonic topographic mapping(G-HaToM)

In this paper we explain the Generalised Harmonic topographic Map (G-HaToM), an extension of the Harmonic Topographic map  and . This algorithm extends the mapping from data space to latent space using the pth power of the L2 distance, where the second power is the former version of the topographic mapping, HaToM. This generalization allows the mapping of more difficult data, reducing at the same time the computational cost of the mapping for the data already clustered by the original HaToM.
Hebbian Learning and Negative Feedback Networks

Part I - Single Stream Networks Background The Negative Feedback Network Peer-Inhibitory Neurons Multiple Cause Data Exploratory Data Analysis Topology Preserving Maps Maximum Likelihood Hebbian Learning Part II - Dual Stream Networks Two Neural Networks for Canonical Correlation Analysis Alternative Derivations of CCA Networks Kernel and Nonlinear Correlations Exploratory Correlation Analysis Multicollinearity and Partial Least Squares Twinned Principal curves The Future App. A. Negative Feedback Artificial Neural Networks B. Previous Factor Analysis Models C. Related Models for ICA D. Previous Dual Stream Approaches E. Data Sets References Index
Tight clusters and smooth manifolds with the harmonic topographic map

We review a new form of self-organizing map introduced in  which is based on a non-linear projection of latent points into data space, identical to that performed in the Generative Topographic Mapping (GTM) . We discuss a refinement of that mapping (M-HaToM) and show on real and artificial data how it both finds the true manifold on which a data set lies and also clusters data more tightly than the previous algorithm (D-HaToM).
Pre-processing using Topographic Mappings

We review two recently developed methods which are used to improve classifier accuracy, bagging and the random subspace method. Both of these methods (and other similar methods) may be characterized as deleting some of the information in the training set and creating classifiers which, though themselves sub-optimal, may be combined to create a better classifier than that created using the original data. We pre-process the data using an unsupervised method which creates topographic mappings and show that the resulting classifiers exhibit diversity and better performance
Local vs Global Models in Pong

We review two previous simulations in which opponent modelling was performed within the computer game of pong. These results suggested that sums of local models were better than a single global model on this data set. We compare two supervised methods, the multilayered perceptron, which is global, and the radial basis function network which is a sum of local models on this data and again find that the latter gives better performance. Finally we introduce a new topology preserving network which can give very local or more global estimates of results and show that, while the local estimates are more accurate, they result in game play which is less human-like in behaviour.
Development of a Global and Integral Model of Business Management Using an Unsupervised Model

In this paper, we use a recent artificial neural architecture called Cooperative Maximum Likelihood Hebbian Learning (CMLHL) in order to categorize the necessities for the Acquisition, Transfer and Updating of Knowledge of the different departments of a firm. We apply Maximum Likelihood Hebbian learning to an extension of a negative feedback network characterised by the use of lateral connections on the output layer. These lateral connections have been derived from the Rectified Gaussian distribution. This technique is used as a tool to develop a part of a Global and Integral Model of business Management, which brings about a global improvement in the firm, adding value, flexibility and competitiveness. From this perspective, the model tries to generalise the hypothesis of organizational survival and competitiveness, so that the organisation that is able to identify, strengthen, and use key knowledge will reach a pole position.
Simultaneous Identification of Face and Orientation

We have recently developed an extension of a Principal Component Analysis Artificial Neural Network which we have linked to the statistical technique of Factor Analysis. The learning rule can be shown to be optimal for data sets corrupted by Gaussian noise. We now derive from a new cost function a novel learning rule which is optimal for a standard data set. We compare both rules on a data set composed of 10 faces in a mixture of poses. The first learning rule performs best on the face data. Our conclusion is that the first rule which is optimal for Gaussian noise is more generally useful but that specific rules may be optimal for finding the independent factors underlying specific data sets dependent on the noise in the data set.
Maximum and Minimum Likelihood Hebbian Learning for Exploratory Projection Pursuit

In this paper, we review an extension of the learning rules in a Principal Component Analysis network which has been derived to be optimal for a specific probability density function. We note that this probability density function is one of a family of pdfs and investigate the learning rules formed in order to be optimal for several members of this family. We show that, whereas we have previously (Lai et al., 2000; Fyfe and MacDonald, 2002) viewed the single member of the family as an extension of PCA, it is more appropriate to view the whole family of learning rules as methods of performing Exploratory Projection Pursuit. We illustrate this on both artificial and real data sets.
A Family of Canonical Correlation Networks

We have previously introduced [2, 5] a neural implementation of Canonical Correlation Analysis (CCA). In this paper, we re-derive the learning method from a probabilistic perspective and then show that similar networks can be derived based on the pioneering work of Becker  if certain simplifying assumptions are made. Becker has shown that her network is able to find depth information from an abstraction of random dot stereogram data and so finally we note the similarity of the derived methods with those of Stone  which was used with a smooth stereo disparity data set to extract depth information.
Analysing Spectroscopic Data Using Hierarchical Cooperative Maximum Likelihood Hebbian Learning

A novel approach to feature selection is presented in this paper, in which the aim is to visualize and extract information from complex, high dimensional spectroscopic data. The model proposed is a mixture of factor analysis and exploratory projection pursuit based on a family of cost functions proposed by Fyfe and MacDonald  which maximizes the likelihood of identifying a specific distribution in the data while minimizing the effect of outliers [9,12]. It employs cooperative lateral connections derived from the Rectified Gaussian Distribution [8,14] to enforce a more sparse representation in each weight vector. We also demonstrate a hierarchical extension to this method which provides an interactive method for identifying possibly hidden structure in the dataset.
Traffic jams: an evolutionary investigation

Traffic at the rush hours is a big problem in every city so that city councils try to encourage citizens to choose mass transportation instead of driving cars. However, this effort is counterbalanced by the fact that travelers' utilities from car usage are more than those from taking buses in most cases. We build a traffic problem model with many agents who have the two options of choosing the car or the bus based on their memory of utilities achieved in previous journeys. Simulating this problem with a genetic algorithm can investigate commuters' behavior and can help identify if a Nash equilibrium exists in this model. We find that a Nash equilibrium exists in theoretical discussion but the population does not converge during the simulation to the Nash equilibrium point. Only a few travelers in the population moved between the two transport methods to establish dynamic stability and all others use only one means of transport throughout the simulation. However, some travelers choose to use the bus all the time (which increases the utilities of the population as a whole) even though their utilities are relatively lower than those of other travelers. Therefore, these simulations describe, the emergence of cooperation which is built on some players bearing worse utilities to obtain more for all the population.
A Neural Network for PCA and Beyond

Principal Component Analysis (PCA) has been implemented by several neural methods. We discuss a Network which has previously been shown to find the Principal Component subspace though not the actual Principal Components themselves. By introducing a constraint to the learning rule (we do not allow the weights to become negative) we cause the same network to find the actual Principal Components. We then use the network to identify individual independent sources when the signals from such sources are ORed together.
Internet agents who structure concept formation using kernel self-organising maps

We review a technique for creating self-organising maps (SOMs) in a feature space, which is nonlinearly related to the original data space. We show that convergence is remarkably fast for this method. We then consider the case where an internet agent quantises internet (HTML) documents using this method. The resulting map has two properties, which are interesting from a human perspective: first, the learning forms topology preserving mappings extremely quickly; second, the learning is most refined for those parts of the feature space which is learned first and which have most data. By considering the linear feature space, we show that it is the interaction between the overcomplete basis in which learning takes place and the mixture of one-shot and incremental learning which comprises the method that gives the method its power. Finally, as a tool to quantise document collections, the tool has the invaluable property that it scales with the number of documents on a site rather than the number of distinct words. We illustrate its use on a standard dataset.