nils-herrmann/clean_scirepeval huggingface dataset:
Analysis of Optical Imaging Data Using Weak Models and ICA

This chapter will present techniques for the recovery of component spatial and temporal modes from spatio-temporal data sets, in particular from medical imaging data such as that obtained by functional magnetic resonance imaging functional magnetic resonance imaging (fMRI) and optical imaging optical imaging (OI) of brain activity. These techniques were developed in order to help address some current issues involving the nature of the haemodynamic response to neural activity (for more details of this research and further references see [7]).
Temporal Constraints on Visual Learning: A Computational Model

Given a constant stream of perceptual stimuli, how can the underlying invariances associated with a given input be learned? One approach consists of using generic truths about the spatiotemporal structure of the physical world as constraints on the types of quantities learned. The learning methodology employed here embodies one such truth: that perceptually salient properties (such as stereo disparity) tend to vary smoothly over time. Unfortunately, the units of an artificial neural network tend to encode superficial image properties, such as individual grey-level pixel values, which vary rapidly over time. However, if the states of units are constrained to vary slowly, then the network is forced to learn a smoothly varying function of the training data. We implemented this temporal-smoothness constraint in a backpropagation network which learned stereo disparity from random-dot stereograms. Temporal smoothness was formalised with the use of regularisation theory by modifying the standard cost function minimised during training of a network. Temporal smoothness was found to be similar to other techniques for improving generalisation, such as early stopping and weight decay. However, in contrast to these, the theoretical underpinnings of temporal smoothing are intimately related to fundamental characteristics of the physical world. Results are discussed in terms of regularisation theory and the physically realistic assumptions upon which temporal smoothing is based.
Independent Components Analysis for Step-wise Separation of Signals

We introduce a method for the step-wise extraction of each of N source signals from a set of M N signal mixtures. The mixtures x = (x1; : : : ; xM) T are formed by combining diierent proportions of the independent sources s = (s1 ; sN) T using an M N mixing matrix A, so that x = As. Porrill and Stone Porrill and Stone, 1997] showed that K N signals (s1 ; sK) T could be extracted using a K M rectangular un0mixing matrix W by maximising the joint entropy of K whitened, unmixed signals Y = (Wx), where is a monotonic, non-linear function. Rather than extracting K N signals simultaneously, we use a step-wise procedure in which each step consists of extracting K = 1 source signal, and then subtracting that signal from the signal mixtures. The relation between this ICA method and exploratory projection pursuit is discussed. The method is demonstrated on acoustic data.
Unsupervised discovery of invariances

The grey level profiles of adjacent image regions tend to be different, whilst the ‘hidden’ physical parameters associated with these regions (e.g. surface depth, edge orientation) tend to have similar values. We demonstrate that a network in which adjacent units receive inputs from adjacent image regions learns to code for hidden parameters. The learning rule takes advantage of the spatial smoothness of physical parameters in general to discover particular parameters embedded in grey level profiles which vary rapidly across an input image. We provide examples in which networks discover stereo disparity and feature orientation as invariances underlying image data.
Unicycling Helps Your French: Spontaneous Recovery of Associations by Learning Unrelated Tasks

We demonstrate that imperfect recall of a set of associations can usually be improved by training on a new, unrelated set of associations. This spontaneous recovery of associations is a consequence of the high dimensionality of weight spaces, and is therefore not peculiar to any single type of neural net. Accordingly, this work may have implications for spontaneous recovery of memory in the central nervous system.
A Canonical Microfunction for Learning Perceptual Invariances

An unsupervised method is presented which permits a set of model neurons, or a microcircuit, to learn low-level vision tasks, such as the extraction of surface depth. Each microcircuit implements a simple, generic strategy which is based on a key assumption: perceptually salient visual invariances, such as surface depth, vary smoothly over time. In the process of learning to extract smoothly varying invariances, each microcircuit maximises a microfunction. This is achieved by means of a learning rule which maximises the long-term variance of the state of a model neuron and simultaneously minimises its short-term variance. The learning rule involves a linear combination of anti-Hebbian and Hebbian weight changes, over short and long time scales, respectively. The method is demonstrated on a hyperacuity task: estimating subpixel stereo disparity from a temporal sequence of random-dot stereograms. After learning, the microcircuit generalises, without additional learning, to previously unseen image sequences. It is proposed that the approach adopted here may be used to define a canonical microfunction, which can be used to learn many perceptually salient invariances.
Learning Perceptually Salient Visual Parameters Using Spatiotemporal Smoothness Constraints

A model is presented for unsupervised learning of low level vision tasks, such as the extraction of surface depth. A key assumption is that perceptually salient visual parameters (e.g., surface depth) vary smoothly over time. This assumption is used to derive a learning rule that maximizes the long-term variance of each unit's outputs, whilst simultaneously minimizing its short-term variance. The length of the half-life associated with each of these variances is not critical to the success of the algorithm. The learning rule involves a linear combination of anti-Hebbian and Hebbian weight changes, over short and long time scales, respectively. This maximizes the information throughput with respect to low-frequency parameters implicit in the input sequence. The model is used to learn stereo disparity from temporal sequences of random-dot and gray-level stereograms containing synthetically generated subpixel disparities. The presence of temporal discontinuities in disparity does not prevent learning or generalization to previously unseen image sequences. The implications of this class of unsupervised methods for learning in perceptual systems are discussed.
Learning stereo disparity using temporal smoothness constraints: a computational model.

An unsupervised learning algorithm is presented for learning stereo disparity. A key assumption is that surface depth varies smoothly over time. This assumption is consistent with a learning rule which maximizes the long-term variance of each unit's outputs, whilst simultaneously minimizing its short-term variance. The learning rule involves a linear combination of anti-Hebbian and Hebbian weight changes, over short and long time scales, respectively. The model is demonstrated on a hyperacuity task: estimating sub-pixel stereo disparity from a temporal sequence of stereograms. The algorithm generalizes, without additional learning, to previously unseen image sequences.
Adaptive Scale Filtering: A General Method for Obtaining Shape From Texture

Introduces adaptive scale filtering, a general method for deriving shape from texture under perspective projection without recourse to prior segmentation of the image into geometric texture elements (texels), and without thresholding of filtered images. If texels on a given surface can be identified in an image then the orientation of that surface can be obtained. However, there is no general characterization of texels for arbitrary textures. Furthermore, even if the size and shape of texels on the surface is invariant with regard to position, perspective projection ensures that the size and shape of the corresponding image texels vary by orders of magnitude. Commencing with an initial set F/sub O/ of identical image filters, adaptive scale filtering iteratively derives a set F/sub N/ which contains a unique filter for each image position. Each element of F/sub N/ is tuned to the three-dimensional structure of the surface; that is, all image filters in F/sub N/ back-project to an identical shape and size on the surface. Thus image texels of various sizes, but associated with a single spatial scale on the surface, can be identified in different parts of the image. When combined with conventional shape from texture methods, edges derived using F/sub N/ provide accurate estimates of surface orientation. Results for planar surfaces are presented.
A learning rule for extracting spatio-temporal invariances

The inputs to photoreceptors tend to change rapidly over time, whereas physical parameters (e.g. surface depth) underlying these changes vary more slowly. Accordingly, if a neuron codes for a physical parameter then its output should also change slowly, despite its rapidly fluctuating inputs. We demonstrate that a model neuron which adapts to make its output vary smoothly over time can learn to extract invariances implicit in its input. This learning consists of a linear combination of Hebbian and anti-Hebbian synaptic changes, operating simultaneously upon the same connection weights but at different time scales. This is shown to be sufficient for the unsupervised learning of simple spatio-temporal invariances.
A Learning Rule for Extracting Temporal Invariances

Neurons recode information implicit in their inputs. Within a population of neurons, this recoding process is often associated with the removal of redundant information. Clearly, an important objective is to extract a perceptually salient invariance, whilst also specifying its value. A linear combination of Hebbian and anti-Hebbiaa adaptation (HAH learning), operating simultaneously upon the same connection weights but at different time scales, is shown to be sufficient for the unsupervised learning of temporal invariances. A model neuron which implements this rule learns to detect an invariance, and also specifies the value of that invariance.
An empirical study of the time complexity of various error functions with conjugate gradient backpropagation

Describes an empirical comparison of the scaling behaviour of six error functions, on a conjugate gradient form of backpropagation. The authors classify the functions according to the limit behaviours of their respective error signals, as the target value and the actual output value approach opposite extremes. These limit behaviours are zero limit, finite limit, and infinite limit. Despite such a wide divergence in their limit behaviours, the authors find that all six error functions exhibit a median run-time order of approximately O(N/sup 4/) on the N-2-N encoder. This result indicates that, while some factors affecting the scaling behaviour of standard and conjugate gradient backpropagation have been previously identified (such as saturation), other factors remain unidentified.
Can artificial neural networks discover useful regularities

Argues that the success of backpropagation artificial neural networks (BP ANNs) depends on input representations which recode input parameters in terms of low-order input statistics. We provide an analysis of how BP ANNs can take advantage of two types of low-order statistical effects to learn a given problem. First, BP ANNs utilise the spurious low-order statistics associated with almost any 'natural' problem. These statistics provide partial information about the underlying mapping, and therefore do not, in general, permit accurate generalisation. Second, it as always possible to hand-craft inputs so that a parameter which was originally coded by relations between input components is coded by a single input component. Such a recoding enables ANNs to perform tasks on which they would otherwise fail. We conjecture that many 'natural' problems represent sparse codings of some underlying relational mapping. If true, this suggests that the ability of BP ANNs to solve such problems is more apparent than real, because BP ANNs rely upon informative, but essentially spurious, correlations between input and output variables.
Learning Invariances Via Spatio-Temporal Constraints

A model for unsupervised learning of visual invariances is presented. The learning method involves a linear combination of anti-Hebbian and Hebbian weight changes, over short and long time scales, respectively. The model is demonstrated on the problem of estimating sub-pixel stereo disparity from a temporal sequence of unprocessed image pairs. After learning on a given image sequence, the model’s ability to detect sub-pixel disparity generalises, without additional learning, to image pairs from other sequences.
Hierachical learning of visual invariances via spatio-temporal constraints

An unsupervised learning algorithm is presented for the acquisition of low-level vision tasks, such as the extraction of surface depth. A key assumption is that perceptually salient parameters in the image of a moving object vary smoothly over time. This assumption is consistent with a learning rule which maximises the long-term variance of a unit's outputs, whilst simultaneously minimising its short-term variance. This learning rule involves a linear combination of anti-Hebbian and Hebbian weight changes, over short and long time scales respectively. It can be shown that the learning rule maximises the mutual information between a unit's output and the short-term exponentially weighted mean of that unit's output. The model is demonstrated on a hyper-acuity task: estimating sub-pixel stereo disparity from a temporal sequence of stereograms. The model can discriminate disparities as small as one-tenth of the width of an input receptor, which compares with human performance on stereo hyper-acuity tasks. Furthermore, the algorithm generalises, without additional learning, to previously unseen image sequences. The results of applying the learning method independently to successive layers of a hierarchical network are presented.
Hierarchical testing designs for pattern recognition

We explore the theoretical foundations of a twenty questions approach to pattern recognition. The object of the analysis is the computational process itself rather than probability distributions (Bayesian inference) or decision boundaries (statistical learning). Our formulation is motivated by applications to scene interpretation in which there are a great many possible explanations for the data, one (background) is statistically dominant, and it is imperative to restrict intensive computation to genuinely ambiguous regions. The focus here is then on pattern filtering: Given a large set Y of possible patterns or explanations, narrow down the true one Y to a small (random) subset Y ⊂ Y, of detected patterns to be subjected to further, more intense, processing. To this end, we consider a family of hypothesis tests for Y e A versus the nonspecific alternatives Y ∈ A c . Each test has null type I error and the candidate sets A ⊂ Y are arranged in a hierarchy of nested partitions. These tests are then characterized by scope (|A|), power (or type II error) and algorithmic cost. We consider sequential testing strategies in which decisions are made iteratively, based on past outcomes, about which test to perform next and when to stop testing. The set Y is then taken to be the set of patterns that have not been ruled out by the tests performed. The total cost of a strategy is the sum of the testing cost and the postprocessing cost (proportional to |Y|) and the corresponding optimization problem is analyzed. As might be expected, under mild assumptions good designs for sequential testing strategies exhibit a steady progression from broad scope coupled with low power to high power coupled with dedication to specific explanations. In the assumptions ensuing this property a key role is played by the ratio cost/power. These ideas are illustrated in the context of detecting rectangles amidst clutter.
Non-Gaussian Component Analysis: a Semi-parametric Framework for Linear Dimension Reduction

We propose a new linear method for dimension reduction to identify non-Gaussian components in high dimensional data. Our method, NGCA (non-Gaussian component analysis), uses a very general semi-parametric framework. In contrast to existing projection methods we define what is uninteresting (Gaussian): by projecting out uninterestingness, we can estimate the relevant non-Gaussian subspace. We show that the estimation error of finding the non-Gaussian components tends to zero at a parametric rate. Once NGCA components are identified and extracted, various tasks can be applied in the data analysis process, like data visualization, clustering, denoising or classification. A numerical study demonstrates the usefulness of our method.
Pattern Recognition from One Example by Chopping

We investigate the learning of the appearance of an object from a single image of it. Instead of using a large number of pictures of the object to recognize, we use a labeled reference database of pictures of other objects to learn invariance to noise and variations in pose and illumination. This acquired knowledge is then used to predict if two pictures of new objects, which do not appear on the training pictures, actually display the same object. 
 
We propose a generic scheme called chopping to address this task. It relies on hundreds of random binary splits of the training set chosen to keep together the images of any given object. Those splits are extended to the complete image space with a simple learning algorithm. Given two images, the responses of the split predictors are combined with a Bayesian rule into a posterior probability of similarity. 
 
Experiments with the COIL-100 database and with a database of 150 degraded LATEX symbols compare our method to a classical learning with several examples of the positive class and to a direct learning of the similarity.
On the Convergence of Eigenspaces in Kernel Principal Component Analysis

This paper presents a non-asymptotic statistical analysis of Kernel-PCA with a focus different from the one proposed in previous work on this topic. Here instead of considering the reconstruction error of KPCA we are interested in approximation error bounds for the eigenspaces themselves. We prove an upper bound depending on the spacing between eigenvalues but not on the dimensionality of the eigenspace. As a consequence this allows to infer stability results for these estimated spaces.
Kernel Projection Machine: a New Tool for Pattern Recognition

This paper investigates the effect of Kernel Principal Component Analysis (KPCA) within the classification framework, essentially the regularization properties of this dimensionality reduction method. KPCA has been previously used as a pre-processing step before applying an SVM but we point out that this method is somewhat redundant from a regularization point of view and we propose a new algorithm called Kernel Projection Machine to avoid this redundancy, based on an analogy with the statistical framework of regression for a Gaussian white noise model. Preliminary experimental results show that this algorithm reaches the same performances as an SVM.
BCI competition 2003-data set IIa: spatial patterns of self-controlled brain rhythm modulations

A brain-computer interface (BCI) is a system that should in its ultimate form translate a subject's intent into a technical control signal without resorting to the classical neuromuscular communication channels. By using that signal to, e.g., control a wheelchair or a neuroprosthesis, a BCI could become a valuable tool for paralyzed patients. One approach to implement a BCI is to let users learn to self-control the amplitude of some of their brain rhythms as extracted from multichannel electroencephalogram. We present a method that estimates subject-specific spatial filters which allow for a robust extraction of the rhythm modulations. The effectiveness of the method was proved by achieving the minimum prediction error on data set IIa in the BCI Competition 2003, which consisted of data from three subjects recorded in ten sessions.
Oracle Bounds and Exact Algorithm for Dyadic Classification Trees

This paper introduces a new method using dyadic decision trees for estimating a classification or a regression function in a multiclass classification problem. The estimator is based on model selection by penalized empirical loss minimization. Our work consists in two complementary parts: first, a theoretical analysis of the method leads to deriving oracle-type inequalities for three different possible loss functions. Secondly, we present an algorithm able to compute the estimator in an exact way.
Statistical Properties of Kernel Principal Component Analysis

We study the properties of the eigenvalues of Gram matrices in a non-asymptotic setting. Using local Rademacher averages, we provide data-dependent and tight bounds for their convergence towards eigenvalues of the corresponding kernel operator. We perform these computations in a functional analytic framework which allows to deal implicitly with reproducing kernel Hilbert spaces of infinite dimension. This can have applications to various kernel algorithms, such as Support Vector Machines (SVM). We focus on Kernel Principal Component Analysis (KPCA) and, using such techniques, we obtain sharp excess risk bounds for the reconstruction error. In these bounds, the dependence on the decay of the spectrum and on the closeness of successive eigenvalues is made explicit.
Different Paradigms for Choosing Sequential Reweighting Algorithms

Analyses of the success of ensemble methods in classification have pointed out the important role played by the margin distribution function on the training and test sets. While it is acknowledged that one should generally try to achieve high margins on the training set, the more precise shape of the empirical margin distribution function one should favor in practice is subject to different approaches. We first present two concurrent philosophies for choosing the empirical margin profile: the minimax margin paradigm and the mean and variance paradigm. The best-known representative of the first paradigm is the AdaBoost algorithm, and this philosophy has been shown by several other authors to be closely related to the principle of the support vector machine. We show that the second paradigm is very close in spirit to Fisher's linear discriminant (in a feature space). We construct two boosting-type algorithms, very similar in their form, dedicated to one or the other philosophy. We consequently derive by interpolation a very simple family of iterative reweighting algorithms that can be understood as different trade-offs between the two paradigms and argue from experiments that this can allow for a suitable adaptivity to different classification problems, particularly in the presence of noise or excessive complexity of the base classifiers.