nils-herrmann/clean_scirepeval huggingface dataset:
Finite size effects in on-line learning of multi-layer neural networks

We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.
On-line learning from finite training sets

We analyse on-line (gradient descent) learning of a rule from a finite set of training examples at non-infinitesimal learning rates η, calculating exactly the time-dependent generalization error for a simple model scenario. In the thermodynamic limit, we close the dynamical equation for the generating function of an infinite hierarchy of order parameters using "within-sample self-averaging". The resulting dynamics is non-perturbative in η, with a slow mode appearing only above a finite threshold ηmin. Optimal settings of η for given final learning time are determined and the results are compared with offline gradient descent.
Online Learning from Nite Training Sets

We analyse online (gradient descent) learning of a rule from a nite set of training examples at non-innnitesimal learning rates , calculating exactly the time-dependent generalization error for a simple model scenario. In the thermodynamic limit, we close the dynamical equation for the generating function of an innnite hierarchy of order parameters using`within-sample self-averaging'. The resulting dynamics is non-perturbative in , with a slow mode appearing only above a nite threshold min. Optimal settings of for given nal learning time are determined and the results are compared with ooine gradient descent. Neural networks have been the subject of much recent research because of their ability to learn rules from examples. One of the most common learning algorithms is online gradient descent: The weights of a network (`student') are updated each time a training example from the training set is presented, such that the error on this example is reduced. In ooine gradient descent, on the other hand, the total error on all examples in the training set is accumulated before a gradient descent weight update is made. For a given training set and starting weights, ooine learning is entirely deterministic. Online learning, on the other hand, is a stochastic process due to the random choice of training example (from the given training set) for each update. It becomes equivalent to ooine learning only in the limit where the learning rate ! 0 1]. In both cases, the main quantity of interest is the evolution of the generalization error: After a given number of weight updates, how well does the student approximate the input-output mapping (`teacher' rule) underlying the training examples? We do not consider in the following non-gradient descent learning algorithms, and also restrict ourselves to gradient descent on the most common measure of error on a training example, the squared output deviation (see eq. (1) below). For interesting recent results on more general, optimized online learning algorithms, see, e.g., 2, 3].
Radial Basis Functions: A Bayesian Treatment

Bayesian methods have been successfully applied to regression and classification problems in multi-layer perceptrons. We present a novel application of Bayesian techniques to Radial Basis Function networks by developing a Gaussian approximation to the posterior distribution which, for fixed basis function widths, is analytic in the parameters. The setting of regularization constants by cross-validation is wasteful as only a single optimal parameter estimate is retained. We treat this issue by assigning prior distributions to these constants, which are then adapted in light of the data under a simple re-estimation formula.
Finite-size effects in on-line learning of multilayer neural networks

We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multilayer networks by calculating fluctuations possessed by finite-dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.
Bayesian Model Comparison by Monte Carlo Chaining

The techniques of Bayesian inference have been applied with great success to many problems in neural computing including evaluation of regression functions, determination of error bars on predictions, and the treatment of hyper-parameters. However, the problem of model comparison is a much more challenging one for which current techniques have significant limitations. In this paper we show how an extended form of Markov chain Monte Carlo, called chaining, is able to provide effective estimates of the relative probabilities of different models. We present results from the robot arm problem and compare them with the corresponding results obtained using the standard Gaussian approximation framework.
Finite size effects in neural network algorithms

One approach to the study of learning in neural networks within the physics community has been to use statistical mechanics to calculate the expected error that a network will make on a typical novel example, termed the generalisation error. Such average case analyses have been mainly carried out with recourse to the thermodynamic limit in which the size of the network is taken to infinity. For the case of a finite sized network, however, the error is not self-averaging i.e., it remains dependent upon the actual set of examples used to train and test the network. The error estimated on a specific test set realisation, termed the test error, forms a finite sample approximation to the generalisation error. We present in this thesis a systematic examination of test error variances in finite sized networks trained by stochastic learning algorithms. Beginning with simple single layer systems, in particular, the linear perceptron, we calculate the test error variance arising from randomness in both the training examples and the stochastic Gibbs learning algorithm. This quantity enables us to examine the performance of networks in a limited data scenario, including the optimal partitioning of a data set into a training and testing set in order to minimize the average error that the network makes, whilst remaining confident that the average test error is representative. A detailed study of the variance of cross-validation errors is carried out, and a comparison made between different cross-validation schemes. We examine also the test error variance of the binary perceptron, comparing the results to the linear case. In addition, we study the effect of a finite system size on the on-line training of multilayer networks, in which we track the dynamic evolution of the error variance under the stochastic gradient descent algorithm used to train the network on an increasing amount of data. We find that the hidden unit symmetries of the multi-layer network give rise to relatively large finite size effects around the point at which the symmetries are broken. As the degree of symmetry in the initial conditions is increased, divergent finite size effects herald a phase transition in which the average case analysis breaks down. By including an easily implementable extra constraint on the training dynamics to encourage hidden unit asymmetry, we show that one can generically reduce both
Online Learning from Finite Training Sets: An Analytical Case Study

We analyse online learning from finite training sets at non-infinitesimal learning rates η. By an extension of statistical mechanics methods, we obtain exact results for the time-dependent generalization error of a linear network with a large number of weights N. We find, for example, that for small training sets of size p ≈ N, larger learning rates can be used without compromising asymptotic generalization performance or convergence speed. Encouragingly, for optimal settings of η (and, less importantly, weight decay λ) at given final learning time, the generalization performance of online learning is essentially as good as that of offline learning.
Gaussian Processes for Bayesian Classification via Hybrid Monte Carlo

The full Bayesian method for applying neural networks to a prediction problem is to set up the prior/hyperprior structure for the net and then perform the necessary integrals. However, these integrals are not tractable analytically, and Markov Chain Monte Carlo (MCMC) methods are slow, especially if the parameter space is high-dimensional. Using Gaussian processes we can approximate the weight space integral analytically, so that only a small number of hyperparameters need be integrated over by MCMC methods. We have applied this idea to classification problems, obtaining excellent results on the real-world problems investigated so far.
FINITE-SIZE EFFECTS AND OPTIMAL TEST SET SIZE IN LINEAR PERCEPTRONS

Fluctuations in the test error are important in the learning theory of finite-dimensional systems as they represent how well the test error matches the average test error. By explicitly finding the variance of the test error due to randomness present in both the data set and algorithm for a linear perceptron of dimension n, we are able to address such questions as the optimal test set size. Where exact results were not tractable, a good approximation is given to the variance. We find that the optimal test set size possesses a phase transition between linear and 2/3 power-law scaling in the system size n.
Named-Entity Recognition in Novel Domains with External Lexical Knowledge

We investigate the adaptation of structured classifiers to new domains. In particular, the problem of using a supervised Named-Entity Recognition (NER) system on data from a different source than the training data. We present a Semi-Markov Model, trained with the perceptron algorithm, coupled with an external dictionary with the goal of improving generalization on the novel domain. Preliminary experiments show promising results, obtained with very simple additional features.
Maximum Margin Semi-Supervised Learning for Structured Variables

Many real-world classification problems involve the prediction of multiple inter-dependent variables forming some structural dependency. Recent progress in machine learning has mainly focused on supervised classification of such structured variables. In this paper, we investigate structured classification in a semi-supervised setting. We present a discriminative approach that utilizes the intrinsic geometry of input patterns revealed by unlabeled data points and we derive a maximum-margin formulation of semi-supervised learning for structured variables. Unlike transductive algorithms, our formulation naturally extends to new test points.
Using Conditional Random Fields to Predict Pitch Accents in Conversational Speech

The detection of prosodic characteristics is an important aspect of both speech synthesis and speech recognition. Correct placement of pitch accents aids in more natural sounding speech, while automatic detection of accents can contribute to better word-level recognition and better textual understanding. In this paper we investigate probabilistic, contextual, and phonological factors that influence pitch accent placement in natural, conversational speech in a sequence labeling setting. We introduce Conditional Random Fields (CRFs) to pitch accent prediction task in order to incorporate these factors efficiently in a sequence model. We demonstrate the usefulness and the incremental effect of these factors in a sequence model by performing experiments on hand labeled data from the Switchboard Corpus. Our model outperforms the baseline and previous models of pitch accent prediction on the Switch-board Corpus.
A study of encoding and decoding techniques for syndrome-based video coding

In conventional video coding, the complexity of an encoder is generally much higher than that of a decoder because operations, such as motion estimation, consume significant computational resources. Such a codec architecture is suitable for a broadcast downlink transmission model. However, in contemporary applications of mobile wireless video uplink transmission, it is desirable to have a low complexity video encoder to meet the resource limitations on mobile devices. Recent advances in distributed video source coding provide a potential reverse in computational complexity for encoder and decoder (Puri, R. and Ramchandran, K., 2002; Aaron, A. and Girod, B., 2003). Various studies are done within the context of a scheme that is similar in spirit to that of Aaron and Girod. We conduct a study of encoding and decoding techniques for syndrome-based video encoding schemes for mobile wireless applications. Our innovation is the adoption of low resolution low quality reference frames for motion estimation at the decoder. To accomplish the goal of reducing computational complexity while maintaining good reconstruction quality at the decoder, we investigate the following related strategies: (1) compression of low resolution and low quality sequences; (2) rate-distortion study with complexity constraints at the encoder; (3) building enhanced evidence for a syndrome decoder. Extensive experimental results have confirmed the effectiveness of these techniques for syndrome based video coding.
Iterative Decoding Using Replicas

Replica shuffled versions of iterative decoders of low-density parity-check codes and turbo codes are presented in this paper. The proposed schemes can converge faster than standard and plain shuffled approaches. Two methods, density evolution and EXIT charts, are used to analyze the performance of the proposed algorithms. Both theoretical analysis and simulations show that the new schedules offer good trade-offs with respect to performance, complexity, latency and connectivity.
Secure Biometrics Via Syndromes

We consider the secure biometric storage problem and develop a solution using syndrome codes. Specifically, biometrics such as fingerprints, irises, and faces are often used for authentication, access control, and encryption instead of passwords. While it is well known that passwords should never be stored in the clear, current systems often store biometrics in the clear and are easily compromised. We describe the secure biometric storage problem by discussing the insecurities in current systems, the reasons why password hashing/encryption algorithms are unsuitable for biometrics, and propose a secure biometric storage framework based on syndrome codes and the Slepian-Wolf theorem. 43rd Annual Allerton Conference on Communications, Control, and Computing
Constructing free-energy approximations and generalized belief propagation algorithms

Important inference problems in statistical physics, computer vision, error-correcting coding theory, and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs. The belief propagation (BP) algorithm is an efficient way to solve these problems that is exact when the factor graph is a tree, but only approximate when the factor graph has cycles. We show that BP fixed points correspond to the stationary points of the Bethe approximation of the free energy for a factor graph. We explain how to obtain region-based free energy approximations that improve the Bethe approximation, and corresponding generalized belief propagation (GBP) algorithms. We emphasize the conditions a free energy approximation must satisfy in order to be a "valid" or "maxent-normal" approximation. We describe the relationship between four different methods that can be used to generate valid approximations: the "Bethe method", the "junction graph method", the "cluster variation method", and the "region graph method". Finally, we explain how to tell whether a region-based approximation, and its corresponding GBP algorithm, is likely to be accurate, and describe empirical results showing that GBP can significantly outperform BP.
Iterative Quantization Using Codes On Graphs

We study codes on graphs combined with an iterative message passing algorithm for quantization. Specifically, we consider the binary erasure quantization (BEQ) problem which is the dual of the binary erasure channel (BEC) coding problem. We show that duals of capacity achieving codes for the BEC yield codes which approach the minimum possible rate for the BEQ. In contrast, low density parity check codes cannot achieve the minimum rate unless their density grows at least logarithmically with block length. Furthermore, we show that duals of efficient iterative decoding algorithms for the BEC yield efficient encoding algorithms for the BEQ. Hence our results suggest that graphical models may yield near optimal codes in source coding as well as in channel coding and that duality plays a key role in such constructions.
Theory and Applied Computing: Observations and Anecdotes

While the kind of theoretical computer science being studied in academe is still highly relevant to systems-oriented research, it is less relevant to applications-oriented research. In applied computing, theoretical elements are used only when strictly relevant to the practical problem at hand. Theory is often combined judiciously with empiricism. And increasingly, theory is most useful when cross-pollinated with ideas and methods from other fields. We will illustrate these points by describing several recent projects at Mitsubishi Electric Research Labs that have heavy mathematical and algorithmic underpinnings. These projects include new algorithms for: traffic analysis; geometric layout; belief propagation in graphical models; dimensionality reduction; and shape representation. Practical applications of this work include elevator dispatch, stock cutting, error-correcting codes, data mining, and digital typography. In all cases theoretical concepts and results are used effectively to solve practical problems of commercial import.
New Spreading Transforms for Fading Channels

We introduce a new class of symbol-spreading transforms that are well-suited for use in errorcontrol coding systems operating in fading channels. These unitary transforms have the property that they allow for the unambigous determination of input symbols from just a single correctly received output symbol. The transforms can be represented using factor graphs that allow for ”fast” encoding. Our simulations show that on channels with fading or interference, systems based on these new transforms can greatly outperform previously proposed systems that use Walsh-Hadamard transforms.
Understanding belief propagation and its generalizations

"Inference" problems arise in statistical physics, computer vision, error-correcting coding theory, and AI. We explain the principles behind the belief propagation (BP) algorithm, which is an efficient way to solve inference problems based on passing local messages. We develop a unified approach, with examples, notation, and graphical models borrowed from the relevant disciplines.We explain the close connection between the BP algorithm and the Bethe approximation of statistical physics. In particular, we show that BP can only converge to a fixed point that is also a stationary point of the Bethe approximation to the free energy. This result helps explaining the successes of the BP algorithm and enables connections to be made with variational approaches to approximate inference.The connection of BP with the Bethe approximation also suggests a way to construct new message-passing algorithms based on improvements to Bethe's approximation introduced Kikuchi and others. The new generalized belief propagation (GBP) algorithms are significantly more accurate than ordinary BP for some problems. We illustrate how to construct GBP algorithms with a detailed example.
Generating Code Representations Suitable for Belief Propagation Decoding

We describe a method for transforming a generalized parity check (GPC) matrix representation of a block linear binary code into another GPC matrix representation of the same code. The output GPC matrix has some attractive features from the point of view of iterative decoding algorithms like belief propagation. In particular, the number of ones in each row is reduced, and there are no cycles of length four in the equivalent graphical representation of the code. We illustrate the method for toy examples including the Golay code, and also for a Euclidean Geometry (n=255,k=127) code. The decoding performance of the belief propagation algorithm using our new GPC matrices improves significantly when they are used on the binary erasure channel, but the results are mixed for the additive white Gaussian noise channel–for the Euclidean Geometry code, the performance still improves, but for the Golay code it deteriorates. to be published in the Proceedings of the 40th Annual Allerton Conference on Communications, Control, and Computing This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved.
Renormalization group approach to error-correcting codes

We explain an algorithm that approximately but efficiently assesses the performance of belief propagation decoding for particular parity check error-correcting codes of large, but finite, blocklength. This algorithm is based on the renormalization group approach from physics: the idea is to continually replace an error-correcting code with a simpler error-correcting code that has nearly identical performance, until the code is reduced to a small enough size that its performance can be computed exactly.