nils-herrmann/clean_scirepeval huggingface dataset:
A PAC analysis of a Bayesian estimator

Bayesian analysis of generalisation can place a prior distribution on the hypotheses and estimate the volume of this space that is consistent with the training data. The larger this volume the greater the confidence in the classifier obtained. The key feature of such estimators is that they provide a posteriori estimates of generalisation based on properties of the hypothesis and the training data. This contrasts with a ‘classical’ PAC analysis which provides only a priori (worst case) bounds. Following results in [26] showing that Datasensitive analysis of generalisation in the PAC sense is possible, the paper uses the techniques to give the first PAC style analysis of a Bayesian inspired estimator of generalisation. The estimator concerned is the size of a ball which can be placed in the consistent region of parameter space. The ball gives a lower bound on the volume of parameter space consistent with the training set. The larger the ball the better the bound on the generalisation obtained. In all cases the bounds are of good generalisation with high confidence, hence bounding the tail of the distribution of generalisation errors that might occur. The resulting bounds are independent of the complexity of the function class though they depend linearly on the dimensionality of the parameter space. Permission to make digitnl/hnrd copies ofnll or part ofthis mntcl-inl fol personal or clw.mnm use is gr:u’ted withaut I kc pwvidcd 11~1 lhc oop~s are 1101 made or distrihutrd For profit or comnwx~l advnm~gr. the copy right notice, the title of the puhlwxtion and its dnie appear. and n&t: is given that copyright is hy psmksion of~hr .Ac’M. lnc To copy othmvisc to republish. IO post on servers or to rrdistrilwtr 10 lists. rquires ~pcxilic pemission m&or fee COLT 97 Nashville.
Detecting Cellular Fraud Using Adaptive Prototypes.

This paper discusses the current status of research on fraud detection undertaken a.s part of the European Commissionfunded ACTS ASPECT (Advanced Security for Personal Communications Technologies) project, by Royal Holloway University of London. Using a recurrent neural network technique, we uniformly distribute prototypes over Toll Tickets. sampled from the U.K. network operator, Vodafone. The prototypes, which continue to adapt to cater for seasonal or long term trends, are used to classify incoming Toll Tickets to form statistical behaviour proFdes covering both the short and long-term past. These behaviour profiles, maintained as probability distributions, comprise the input to a differential analysis utilising a measure known as the HeUinger distance[5] between them as an alarm criteria. Fine tuning the system to minimise the number of false alarms poses a significant task due to the low fraudulent/non fraudulent activity ratio. We benefit from using unsupervised learning in that no fraudulent examples ate requited for training. This is very relevant considering the currently secure nature of GSM where fraud scenarios, other than Subscription Fraud, have yet to manifest themselves. It is the aim of ASPECT to be prepared for the would-be fraudster for both GSM and UMTS,
Data-Dependent Structural Risk Minimization for Perceptron Decision Trees

Perceptron Decision Trees (also known as Linear Machine DTs, etc.) are analysed in order that data-dependent Structural Risk Minimisation can be applied. Data-dependent analysis is performed which indicates that choosing the maximal margin hyperplanes at the decision nodes will improve the generalization. The analysis uses a novel technique to bound the generalization error in terms of the margins at individual nodes. Experiments performed on real data sets confirm the validity of the approach.
Valid Generalisation from Approximate Interpolation

Let and be sets of functions from domain X to ℝ. We say that validly generalises from approximate interpolation if and only if for each η > 0 and ∈, δ ∈ (0,1) there is m0(η, ∈, δ) such that for any function t ∈ and any probability distribution on X, if m > m0 then with m-probability at least 1 – δ, a sample X = (x1, X2,…,xm) ∈ Xm satisfies We find conditions that are necessary and sufficient for to validly generalise from approximate interpolation, and we obtain bounds on the sample length m0{η,∈,δ) in terms of various parameters describing the expressive power of .
Learning to compress ergodic sources

We present an adaptive coding technique which is shown to achieve optimal coding in the limit as the size of the text grows, while the data structures associated with the code only grow linearly with the text. The approach relies on Huffman codes which are generated relative to the context in which a particular character occurs. The Huffman codes themselves are inferred from the data that has already been seen. A key part of the paper involves showing that the loss per character incurred by the learning process tends to zero as the size of the text tends to infinity. This involves an analysis in an on-line learning framework bounding the cumulative loss, where loss is defined to be the excess code length. By using the Bayes prediction distribution and code the expected loss per character converges to zero at the best possible rate of O(log n/n). By allowing the length of contexts to grow in response to commonly occurring subsequences, the coding is efficient precisely where it needs to be, hence achieving a high compression rate at a relatively low overhead in terms of data structure storage.
A recurrent network with stochastic weights

Stochastic neural networks for global optimization are usually built by introducing random fluctuations into the network. A natural method is to use stochastic weights rather than stochastic activation functions. We propose a new model in which each neuron has very simple functionality but all the weights are stochastic. It is shown that the stationary distribution of the network uniquely exists and it is approximately a Boltzmann-Gibbs distribution when the size of the network is not too small. A new technique to implement simulated annealing is proposed. Simulation results on the graph bisection problem show that the power of the network is comparable with that of a Boltzmann machine.
A framework for structural risk minimisation

The paper introduces a framework for studying structural risk minimisation. The model views structural risk minimisation in a PAC context. It then considers the more general case when the hierarchy of classes is chosen in response to the data. This theoretically explains the impressive performance of the maximal margin hyperplane algorithm of Vapnik. It may also provide a general technique for exploitingserendipitous simplicity in observed data to obtain better prediction accuracy from small training sets.
Conndence Estimates of Classiication Accuracy on New Examples Produced as Part of the Esprit Working Group in Neural and Computational Learning, Neurocolt 8556

Following recent results 5] showing the importance of the fat shattering dimension in explaining the beneecial eeect of a large margin on generalization performance, the current paper investigates how the margin on a test example can be used to give greater certainty of correct classii-cation in the distribution independent model. The results show that even if the classiier does not classify all of the training examples correctly, the fact that a new example has a larger margin than that on the misclassiied examples, can be used to give very good estimates for the generalization performance in terms of the fat shattering dimension measured at a scale proportional to the excess margin. The estimate relies on a suuciently large number of the correctly classiied training examples having a margin roughly equal to that used to estimate generalization, indicating that the corresponding output values need to bèwell sampled'.
Sigmoid neural transfer function realized by percolation.

An experiment using the phenomenon of percolation has been conducted to demonstrate the implementation of neural functionality (summing and sigmoid transfer). A simple analog approximation to digital percolation is implemented. The device consists of a piece of amorphous silicon with stochastic bit-stream optical inputs, in which a current percolating from one end to the other defines the neuron output, also in the form of a stochastic bit stream. Preliminary experimental results are presented.
Fast String Matching in Stationary Ergodic Sources

A connection is made between the theory of ergodicity and the expected complexity of string searching. In particular, a substring search algorithm is introduced which, when applied to searching in text that has been produced by an appropriate stationary ergodic source, has an expected running time of O((N/m + m)logm), for a text string of length N and search string of length m. Similar expected complexity results have been obtained before, but the analysis is performed in a significantly more general framework, which models with greater accuracy the statistics of many types of strings, including natural language. The analysis also sheds light on the performance of the Boyer-Moore algorithm and the Sunday algorithm when applied to natural language.
Techniques in Neural Learning Produced as Part of the Esprit Working Group in Neural and Computational Learning, Neurocolt 8556

This paper takes ideas developed in a theoretical framework by Maass 8] and adapts them for a practical learning algorithm for feedforward sigmoid neural networks. A number of diierent techniques are presented which are based loosely around the common theme of taking advantage of the lin-earity of the net input to a neuron, or in other words the fact that there is only a single non-linearity per neuron. Some experimental results are included, though many of the ideas are as yet untested. The paper can therefore be viewed as a tool box ooering a selection of possible techniques for incorporation in practical, heuristic learning algorithms for multi-layer perceptrons.
Neural networks for invariant pattern recognition

In this paper, we discuss a methodology for applying feedforward networks to problems of invariant pattern recognition. We present the Group Representation Network (GRN), a type of feedforward network with the property that its output is invariant under a group of transformations of its input. Since the invariance of such a network is inbuilt, it does not need to be learned. Consequently it is capable of a better generalization performance than a conventional network for solving the same symmetric problem. In addition, the GRN has fewer free parameters than connections and we can hence expect it to train faster than an ordinary network of the same connectivity.
Consciousness at the Crossroads of Philosophy and Cognitive Science, Special issue of Journal of Consciousness Studies

Edited by Bojan Borstner and John Shawe-Taylor, these are the proceedings of the conference held in August 1994 as part of the European Community TEMPUS project. Speakers ranged from veteran philosopher Ullin Place, to physicist Basil Hiley, best known for his collaboration with David Bohm. These edited selected proceedings include original contributions and review papers covering neuropsychology, physics, philosophy and cognitive science.
Generalisation Analysis for Classes of Continuous Neural Networks

Thas paper reviews some of the recent results in applying the theory of Probably Approximately Correct (PAC) learning to feedforward neural networks with continuous activation functions. Despite the best known upper bound on the VC dimension of sigmoid networks being O ( ( W N ) ’ ) , for W parameters and N computational nodes, at is shown that the asymptotic bound on the sample size requzred f o r learning sigmoid networks is better than would be expected from a naive use of the VC dimension result. We propose a way of using boolean circuits to perform real valued computation in a way that naturally extends their boolean functionality. The functionality of multiple fan in threshold gates in this model is shown t o mimic that of a hardware implementation of continuous Neural Networks. The sample sizes obtained for these networks are significantly lower than those obtained for sigmoidal networks.
Stochastic connection neural networks

We investigate a novel neural network model which uses stochastic weights. It is shown that the functionality of the network is comparable to that of a general stochastic neural network using standard sigmoid activation functions. For the multilayer feedforward structure we demonstrate the network can be successfully used to solve a real problem like handwritten digit recognition. It is also shown that the recurrent network is as powerful as a Boltzmann machine. A new technique to implement simulated annealing is presented. Simulation results on the graph bisection problem demonstrate the model is efficient for global optimization.
Bitstream neurons for graph colouring

In Shawe Taylor and Zerovnik introduced the Generalised Boltzmann Machine GBM as an extension to the Boltzmann machine that enables us to map constraint problems requiring more than two states onto a recurrent neural network In experiments were performed using the Mean Field Annealing approach to graph colouring using the Petford and Welsh algorithm as a GBM In this paper we extend the use of bitstreams from the bi polar stochastically connected Boltzmann machine to a recurrent network of stochastically connected multi state bit stream neurons MSBSN We compare the performance of the re sultant network with the results obtained using Mean Field Annealing described in and the Petford and Welsh algorithm The results obtained from simulations of the MSBSN show that there is no increase in the number of iterations taken to solve a typical randomly generated k colourable graph Clearly the digital nature of the neuron combined with its simple functionality described by Shawe Taylor Van Daalen and Zhao in could lead to a fast and inexpensive hardware implementaion
Sample sizes for sigmoidal neural networks

This paper applies the theory of Probably Approximately Correct (PAC) learning to feedforward neural networks with sigmoidal activation functions. Despite the best known upper bound on the VC dimension of such networks being 0((WlV)2), for W parameters and N computational nodes, it is shown that the asymptotic bound on the sample size required for learning with increasing accuracy 1 – c and decreasing probabllty of failure d is o((l/e)(wlog(l/c) + (WN)2 +log(l/6)), For practical values of .s and d the formula obtained for the sample sizes is a factor 2 log(2e/c) smaller than a naive use of the VC dimension result would give. Similar results are obtained for learning where the hypothesis is only guaranteed to correctly classify a given proportion of the training sample. The results are formulated in general terms and show that for many learning classes defined by smooth functions thresholded at the output, the sample size for a class with VC-dimension d and 1 parameters is 0(%10’ (:)+0 [ ’3’+10w *Work supported by the ESPRIT Working Group NeuroCOLT, No.
Pattern Recognition

ive Multi-Document Text Summarization Using a Genetic Algorithm.  Eduardo Millán-Hernández, René Arnulfo García-Hernández, Yulia Ledeneva, and Ángel Hernández-Castañeda Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Bias in Estimating the Variance of K-Fold Cross-Validation

Most machine learning researchers perform quantitative experiments to estimate generalization error and compare the perforniance of different algorithms (in particular, their proposed algorithmn). In order to be able to draw statistically convincing conclusions, it is important to estimate the uncertainty of such estimates. This paper studies the very commonly used K-fold cross-validation estimator of generalization performance. The main theorem shows that there exists no universal (valid under all distributions) unbiased estimator of the variance of K-fold cross-validation, based on a single computation of the K-fold cross-validation estimator. The analysis that accompanies this result is based on the eigen-decomposition of the covariance matrix of errors, which has only three different eigenvalues corresponding to three degrees of freedom of the matrix and three components of the total variance. This analysis helps to better understand the nature of the problem and how it can make naive estimators (that don't take into account the error correlations due to the overlap between training and test sets) grossly underestimate variance. This is confirmed by numerical experiments in which the three components of the variance are compared when the difficulty of the learning problem and the number of folds are varied.
Convex Neural Networks

Convexity has recently received a lot of attention in the machine learning community, and the lack of convexity has been seen as a major disadvantage of many learning algorithms, such as multi-layer artificial neural networks. We show that training multi-layer neural networks in which the number of hidden units is learned can be viewed as a convex optimization problem. This problem involves an infinite number of variables, but can be solved by incrementally inserting a hidden unit at a time, each time finding a linear classifier that minimizes a weighted sum of errors.
Efficient Non-Parametric Function Induction in Semi-Supervised Learning

There has been an increase of interest for semi-supervised learning recently, because of the many datasets with large amounts of unlabeled examples and only a few labeled ones. This paper follows up on proposed non-parametric algorithms which provide an estimated continuous label for the given unlabeled examples. It extends them to function induction algorithms that correspond to the minimization of a regularization criterion applied to an out-of-sample example, and happens to have the form of a Parzen windows regressor. The advantage of the extension is that it allows predicting the label for a new example without having to solve again a linear system of dimension 'n' (the number of unlabeled and labeled training examples), which can cost O(n^3). Experiments show that the extension works well, in the sense of predicting a label close to the one that would have been obtained if the test example had been included in the unlabeled set. This relatively efficient function induction procedure can also be used when 'n' is large to approximate the solution by writing it only in terms of a kernel expansion with 'm' Keywords: non-parametric models, classification, regression, semi-supervised learning, modeles non parametriques, classification, regression, apprentissage semi-supervise
Greedy Spectral Embedding

Spectral dimensionality reduction methods and spectral clustering methods require computation of the principal eigenvectors of an n × n matrix where n is the number of examples. Following up on previously proposed techniques to speed-up kernel methods by focusing on a subset of m examples, we study a greedy selection procedure for this subset, based on the featurespace distance between a candidate example and the span of the previously chosen ones. In the case of kernel PCA or spectral clustering this reduces computation to O(m2n). For the same computational complexity, we can also compute the feature space projection of the non-selected examples on the subspace spanned by the selected examples, to estimate the embedding function based on all the data, which yields considerably better estimation of the embedding function. This algorithm can be formulated in an online setting and we can bound the error on the approximation of the Gram matrix.
Non-Local Manifold Parzen Windows

To escape from the curse of dimensionality, we claim that one can learn non-local functions, in the sense that the value and shape of the learned function at x must be inferred using examples that may be far from x. With this objective, we present a non-local non-parametric density estimator. It builds upon previously proposed Gaussian mixture models with regularized covariance matrices to take into account the local shape of the manifold. It also builds upon recent work on non-local estimators of the tangent plane of a manifold, which are able to generalize in places with little training data, unlike traditional, local, non-parametric models.
The Curse of Dimensionality for Local Kernel Machines

We present a series of theoretical arguments supporting the claim that a large class of modern learning algorithms based on local kernels are sensitive to the curse of dimensionality. These include local manifold learning algorithms such as Isomap and LLE, support vector classifiers with Gaussian or other local kernels, and graph-based semisupervised learning algorithms using a local similarity function. These algorithms are shown to be local in the sense that crucial properties of the learned function at x depend mostly on the neighbors of x in the training set. This makes them sensitive to the curse of dimensionality, well studied for classical non-parametric statistical learning. There is a large class of data distributions for which non-local solutions could be expressed compactly and potentially be learned with few examples, but which will require a large number of local bases and therefore a large number of training examples when using a local learning algorithm.
The Curse of Highly Variable Functions for Local Kernel Machines

We present a series of theoretical arguments supporting the claim that a large class of modern learning algorithms that rely solely on the smoothness prior - with similarity between examples expressed with a local kernel - are sensitive to the curse of dimensionality, or more precisely to the variability of the target. Our discussion covers supervised, semi-supervised and unsupervised learning algorithms. These algorithms are found to be local in the sense that crucial properties of the learned function at x depend mostly on the neighbors of x in the training set. This makes them sensitive to the curse of dimensionality, well studied for classical non-parametric statistical learning. We show in the case of the Gaussian kernel that when the function to be learned has many variations, these algorithms require a number of training examples proportional to the number of variations, which could be large even though there may exist short descriptions of the target function, i.e. their Kolmogorov complexity may be low. This suggests that there exist non-local learning algorithms that at least have the potential to learn about such structured but apparently complex functions (because locally they have many variations), while not using very specific prior domain knowledge.
Hierarchical Probabilistic Neural Network Language Model

In recent years, variants of a neural network architecture for statistical language modeling have been proposed and successfully applied, e.g. in the language modeling component of speech recognizers. The main advantage of these architectures is that they learn an embedding for words (or other symbols) in a continuous space that helps to smooth the language model and provide good generalization even when the number of training examples is insufficient. However, these models are extremely slow in comparison to the more commonly used n-gram models, both for training and recognition. As an alternative to an importance sampling method proposed to speed-up training, we introduce a hierarchical decomposition of the conditional probabilities that yields a speed-up of about 200 both during training and recognition. The hierarchical decomposition is a binary hierarchical clustering constrained by the prior knowledge extracted from the WordNet semantic hierarchy.