nils-herrmann/clean_scirepeval huggingface dataset:
On the Optimal Parameter Choice for v-Support Vector Machines

We determine the asymptotically optimal choice of the parameter /spl nu/ for classifiers of /spl nu/-support vector machine (/spl nu/-SVM) type which has been introduced by Scholkopf et al. (2000). It turns out that /spl nu/ should be a close upper estimate of twice the optimal Bayes risk provided that the classifier uses a so-called universal kernel such as the Gaussian RBF kernel. Moreover, several experiments show that this result can be used to implement some modified cross validation procedures which improve standard cross validation for /spl nu/-SVMs.
Support Vector Machines are Universally Consistent

We show that support vector machines of the 1-norm soft margin type are universally consistent provided that the regularization parameter is chosen in a distinct manner and the kernel belongs to a specific class-the so-called universal kernels-which has recently been considered by the author. In particular it is shown that the 1-norm soft margin classifier with Gaussian RBF kernel on a compact subset X of Rd and regularization parameter cn = nβ-1 is universally consistent, if n is the training set size and 0 >β> 1/d.
On the Inuence of the Kernel on the Consistency of

In this article we study the generalization abilities of several classiers of support vector machine (SVM) type using a certain class of kernels that we call universal. It is shown that the soft margin algorithms with universal kernels are consistent for a large class of classication problems including some kind of noisy tasks provided that the regularization parameter is chosen well. In particular we derive a simple sucient condition for this parameter in the case of Gaussian RBF kernels. On the one hand our considerations are based on an investigation of an approximation property|the so-called universality|of the used kernels that ensures that all continuous functions can be approximated by certain kernel expressions. This approximation property also gives a new insight into the role of kernels in these and other algorithms. On the other hand the results are achieved by a precise study of the underlying optimization problems of the classiers. Furthermore, we show consistency for the maximal margin classier as well as for the soft margin SVM’s in the presence of large margins. In this case it turns out that also constant regularization parameters ensure consistency for the soft margin SVM’s. Finally we prove that even for simple, noise free classication
Metric entropy of convex hulls in type p spaces-the critical case

Given a precompact subset A of a type p Banach space E, where p ∈ (1, 2], we prove that for every β E [0,1) and all n ∈ N sup k 1/p' (log k) β-1 e k (aco A) ≤ c sup k 1/p' (log k) β e k (A) k≤n holds, where acoA is the absolutely convex hull of A and e k (.) denotes the k th dyadic entropy number. With this inequality we show in particular that for given A and β ∈ (-∞,1) with e n (A) ≤ n -1/p' (log n) -β for all n ∈ N the inequality e n (aco A) < c n -1/p' (log n) -β+1 holds true for all n E N. We also prove that this estimate is asymptotically optimal whenever E has no better type than p. For β = 0 this answers a question raised by Carl, Kyrezi, and Pajor which has been solved up to now only for the Hilbert space case by F. Gao.
On the Influence of the Kernel on the Consistency of Support Vector Machines

In this article we study the generalization abilities of several classifiers of support vector machine (SVM) type using a certain class of kernels that we call universal. It is shown that the soft margin algorithms with universal kernels are consistent for a large class of classification problems including some kind of noisy tasks provided that the regularization parameter is chosen well. In particular we derive a simple sufficient condition for this parameter in the case of Gaussian RBF kernels. On the one hand our considerations are based on an investigation of an approximation property---the so-called universality---of the used kernels that ensures that all continuous functions can be approximated by certain kernel expressions. This approximation property also gives a new insight into the role of kernels in these and other algorithms. On the other hand the results are achieved by a precise study of the underlying optimization problems of the classifiers. Furthermore, we show consistency for the maximal margin classifier as well as for the soft margin SVM's in the presence of large margins. In this case it turns out that also constant regularization parameters ensure consistency for the soft margin SVM's. Finally we prove that even for simple, noise free classification problems SVM's with polynomial kernels can behave arbitrarily badly.
Entropy of C(K)-Valued Operators

We investigate how the entropy numbers (e"n(T)) of an arbitrary Holder-continuous operator T:E->C(K) are influenced by the entropy numbers (@e"n(K)) of the underlying compact metric space K and the geometry of E. We derive diverse universal inequalities relating finitely many @e"n(K)'s with finitely many e"n(T)'s which yield statements about the asymptotically optimal behaviour of the sequence (e"n(T)) in terms of the sequence (@e"n(K)). As an application we present new methods for estimating the entropy numbers of a precompact and convex subset in a Banach space E, provided that the entropy numbers of its extremal points are known.
Learning reliably and with one-sided error

Reliable and probably useful learning, proposed by Rivest and Sloan, is a variant of probably approximately correct learning. In this model the hypothesis must never misclassify an instance but is allowed to answer “I don't know” with a low probability. We derive upper and lower bounds for the sample complexity of reliable and probably useful learning in terms of the combinatorial characteristics of the concept class to be learned. This is done by reducing reliable and probably useful learning to learning with one-sided error. The bounds also hold for a slightly weaker model that allows the learner to output with a low probability a hypothesis that makes misclassifications. We see that in these models learning with one oracle is more difficult than learning with two oracles. Our results imply that monotone Boolean conjunctions or disjunctions cannot be learned reliably and probably usefully from a polynomial number of examples. Rectangles in ℝn forn ≥ 2 cannot be learned from any finite number of examples.
Relative Loss Bounds for Multidimensional Regression Problems

We study on-line generalized linear regression with multidimensional outputs, i.e., neural networks with multiple output nodes but no hidden nodes. We allow at the final layer transfer functions such as the softmax function that need to consider the linear activations to all the output neurons. The weight vectors used to produce the linear activations are represented indirectly by maintaining separate parameter vectors. We get the weight vector by applying a particular parameterization function to the parameter vector. Updating the parameter vectors upon seeing new examples is done additively, as in the usual gradient descent update. However, by using a nonlinear parameterization function between the parameter vectors and the weight vectors, we can make the resulting update of the weight vector quite different from a true gradient descent update. To analyse such updates, we define a notion of a matching loss function and apply it both to the transfer function and to the parameterization function. The loss function that matches the transfer function is used to measure the goodness of the predictions of the algorithm. The loss function that matches the parameterization function can be used both as a measure of divergence between models in motivating the update rule of the algorithm and as a measure of progress in analyzing its relative performance compared to an arbitrary fixed model. As a result, we have a unified treatment that generalizes earlier results for the gradient descent and exponentiated gradient algorithms to multidimensional outputs, including multiclass logistic regression.
Computational Learning Theory

We consider the sample complexity of agnostic learning with respect to squared loss. It is known that if the function class F used for learning is convex then one can obtain better sample complexity bounds than usual. It has been claimed that there is a lower bound that showed there was an essential gap in the rate. In this paper we show that the lower bound proof has a gap in it. Although we do not provide a definitive answer to its validity. More positively, we show one can obtain “fast” sample complexity bounds for nonconvex F for “most” target conditional expectations. The new bounds depend on the detailed geometry of F , in particular the distance in a certain sense of the target’s conditional expectation from the set of nonuniqueness points of the class F .
Online Learning of Linear Classifiers

This paper surveys some basic techniques and recent results related to online learning. Our focus is on linear classification. The most familiar algorithm for this task is the perceptron. We explain the perceptron algorithm and its convergence proof as an instance of a generic method based on Bregman divergences. This leads to a more general algorithm known as the p-norm perceptron. We give the proof for generalizing the perceptron convergence theorem for the p-norm perceptron and the non-separable case. We also show how regularization, again based on Bregman divergences, can make an online algorithm more robust against target movement.
Computational Learning Theory: 15th Annual Conference on Computational Learning Theory, COLT 2002, Sydney, Australia, July 8-10, 2002. Proceedings

Statistical Learning Theory.- Agnostic Learning Nonconvex Function Classes.- Entropy, Combinatorial Dimensions and Random Averages.- Geometric Parameters of Kernel Machines.- Localized Rademacher Complexities.- Some Local Measures of Complexity of Convex Hulls and Generalization Bounds.- Online Learning.- Path Kernels and Multiplicative Updates.- Predictive Complexity and Information.- Mixability and the Existence of Weak Complexities.- A Second-Order Perceptron Algorithm.- Tracking Linear-Threshold Concepts with Winnow.- Inductive Inference.- Learning Tree Languages from Text.- Polynomial Time Inductive Inference of Ordered Tree Patterns with Internal Structured Variables from Positive Data.- Inferring Deterministic Linear Languages.- Merging Uniform Inductive Learners.- The Speed Prior: A New Simplicity Measure Yielding Near-Optimal Computable Predictions.- PAC Learning.- New Lower Bounds for Statistical Query Learning.- Exploring Learnability between Exact and PAC.- PAC Bounds for Multi-armed Bandit and Markov Decision Processes.- Bounds for the Minimum Disagreement Problem with Applications to Learning Theory.- On the Proper Learning of Axis Parallel  A Consistent Strategy for Boosting Algorithms.- The Consistency of Greedy Algorithms for Classification.- Maximizing the Margin with Boosting.- Other Learning  Performance Guarantees for Hierarchical Clustering.- Self-Optimizing and Pareto-Optimal Policies in General Environments Based on Bayes-Mixtures.- Prediction and Dimension.- Invited Talk.- Learning the Internet.
Averaging Expert Predictions

We consider algorithms for combining advice from a set of experts. In each trial, the algorithm receives the predictions of the experts and produces its own prediction. A loss function is applied to measure the discrepancy between the predictions and actual observations. The algorithm keeps a weight for each expert. At each trial the weights are first used to help produce the prediction and then updated according to the observed outcome. Our starting point is Vovk's Aggregating Algorithm, in which the weights have a simple form: the weight of an expert decreases exponentially as a function of the loss incurred by the expert. The prediction of the Aggregating Algorithm is typically a non-linear function of the weights and the experts' predictions. We analyze here a simplified algorithm in which the weights are as in the original Aggregating Algorithm, but the prediction is simply the weighted average of the experts' predictions. We show that for a large class of loss functions, even with the simplified prediction rule the additional loss of the algorithm over the loss of the best expert is at most c ln n, where n is the number of experts and c a constant that depends on the loss function. Thus, the bound is of the same form as the known bounds for the Aggregating Algorithm, although the constants here are not quite as good. We use relative entropy to rewrite the bounds in a stronger form and to motivate the update.
Boosting as entropy projection

We consider the AdaBoost procedure for boosting weak learners. In AdaBoost, a key step is choosing a new distribution on the training examples based on the old distribution and the mistakes made by the present weak hypothesis. We show how AdaBoost’s choice of the new distribution can be seen as an approximate solution to the following problem: Find a new distribution that is closest to the old distribution subject to the constraint that the new distribution is orthogonal to the vector of mistakes of the current weak hypothesis. The distance (or divergence) between distributions is measured by the relative entropy. Alternatively, we could say that AdaBoost approximately projects the distribution vector onto a hyperplane defined by the mistake vector. We show that this new view of AdaBoost as an entropy projection is dual to the usual view of AdaBoost as minimizing the normalization factors of the updated distributions.
Relative loss bounds for single neurons

We analyze and compare the well-known gradient descent algorithm and the more recent exponentiated gradient algorithm for training a single neuron with an arbitrary transfer function. Both algorithms are easily generalized to larger neural networks, and the generalization of gradient descent is the standard backpropagation algorithm. In this paper we prove worst-case loss bounds for both algorithms in the single neuron case. Since local minima make it difficult to prove worst-case bounds for gradient-based algorithms, we must use a loss function that prevents the formation of spurious local minima. We define such a matching loss function for any strictly increasing differentiable transfer function and prove worst-case loss bounds for any such transfer function and its corresponding matching loss. For example, the matching loss for the identity function is the square loss and the matching loss for the logistic transfer function is the entropic loss. The different forms of the two algorithms' bounds indicates that exponentiated gradient outperforms gradient descent when the inputs contain a large number of irrelevant components. Simulations on synthetic data confirm these analytical results.
Sequential Prediction of Individual Sequences Under General Loss Functions

We consider adaptive sequential prediction of arbitrary binary sequences when the performance is evaluated using a general loss function. The goal is to predict on each individual sequence nearly as well as the best prediction strategy in a given comparison class of (possibly adaptive) prediction strategies, called experts. By using a general loss function, we generalize previous work on universal prediction, forecasting, and data compression. However, here we restrict ourselves to the case when the comparison class is finite. For a given sequence, we define the regret as the total loss on the entire sequence suffered by the adaptive sequential predictor, minus the total loss suffered by the predictor in the comparison class that performs best on that particular sequence. We show that for a large class of loss functions, the minimax regret is either /spl theta/(log N) or /spl Omega/(/spl radic//spl Lscr/log N), depending on the loss function, where N is the number of predictors in the comparison class and/spl Lscr/ is the length of the sequence to be predicted. The former case was shown previously by Vovk (1990); we give a simplified analysis with an explicit closed form for the constant in the minimax regret formula, and give a probabilistic argument that shows this constant is the best possible. Some weak regularity conditions are imposed on the loss function in obtaining these results. We also extend our analysis to the case of predicting arbitrary sequences that take real values in the interval [0,1].
Exponentiated Gradient Versus Gradient Descent for Linear Predictors

We consider two algorithm for on-line prediction based on a linear model. The algorithms are the well-known Gradient Descent (GD) algorithm and a new algorithm, which we call EG(+/-). They both maintain a weight vector using simple updates. For the GD algorithm, the update is based on subtracting the gradient of the squared error made on a prediction. The EG(+/-) algorithm uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively. We present worst-case loss bounds for EG(+/-) and compare them to previously known bounds for the GD algorithm. The bounds suggest that the losses of the algorithms are in general incomparable, but EG(+/-) has a much smaller loss if only a few components of the input are relevant for the predictions. We have performed experiments, which show that our worst-case upper bounds are quite tight already on simple artificial data.
Additive versus exponentiated gradient updates for linear prediction

We consider two algorithms for on-line prediction based on a linear model. The algorithms are the well-known Gradient Descent (GD) algorithm and a new algorithm, which we call EG *. They both maintain a weight vector using simple updates. For the GD algorithm, the weight vector is updated by subtracting from it the gradient of the squared error made on a prediction multiplied by a parameter called the learning rate. The EG* uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively. We present worst-case on-line loss bounds for EG* and compare them to previously known bounds for the GD algorithm. The bounds suggest that although the on-line losses of the algorithms are in general incomparable, EG * has a much smaller loss if only few of the input variables are relevant for the predictions. Experiments show that the worst-case upper bounds are quite tight already on simple artificial data. Our main methodological idea is using a distance function between weight vectors both in motivating the algorithms and as a potential function in an amortized analysis that leads to worst-case loss bounds. Using squared Euclidean distance leads to the GD algorithm, and using the relative entropy leads to the EG* algorithm.
Worst-case Loss Bounds for Single Neurons

We analyze and compare the well-known Gradient Descent algorithm and a new algorithm, called the Exponentiated Gradient algorithm, for training a single neuron with an arbitrary transfer function. Both algorithms are easily generalized to larger neural networks, and the generalization of Gradient Descent is the standard back-propagation algorithm. In this paper we prove worst-case loss bounds for both algorithms in the single neuron case. Since local minima make it difficult to prove worst-case bounds for gradient-based algorithms, we must use a loss function that prevents the formation of spurious local minima. We define such a matching loss function for any strictly increasing differentiable transfer function and prove worst-case loss bound for any such transfer function and its corresponding matching loss. For example, the matching loss for the identity function is the square loss and the matching loss for the logistic sigmoid is the entropic loss. The different structure of the bounds for the two algorithms indicates that the new algorithm out-performs Gradient Descent when the inputs contain a large number of irrelevant components.
The perceptron algorithm vs. Winnow: linear vs. logarithmic mistake bounds when few input variables are relevant

We give an adversary strategy that forces the Perceptron algorithm to make (N-k+1)/2 mistakes when learning k-literal disjunctions over N variables. Experimentally we see that even for simple random data, the number of mistakes made by the Perceptron algorithm grows almost linearly with N, even if the number k of relevant variable remains a small constant. In contrast, Littlestone''s algorithm Winnow makes at most O(k log N) mistakes for the same problem. Both algorithms use thresholded linear functions as their hypotheses. However, Winnow does multiplicative updates to its weight vector instead of the additive updates of the Perceptron algorithm.
Probabilistic Optimization for Energy-Efficient Broadcast in All-Wireless Networks

We study optimization methods for source-initiated energyefficient broadcast in all-wireless networks. The minimum energy broadcast (MEB) problem in wireless networks proved to be NP-complete [1,4]. Past studies on MEB problem  focus on deterministic optimization to construct energy-efficient broadcast tree. In this paper, we present a probabilistic optimization approach, the Randomized Tree Optimization (RTO) algorithm, which is based on the cross-entropy (CE) method introduced by Rubinstein et al in [3,6,7]. The experimental results indicate that the RTO method obtains the best performance compared to state-of-the-art methods.
Basis Function Adaptation in Temporal Difference Reinforcement Learning

Reinforcement Learning (RL) is an approach for solving complex multi-stage decision problems that fall under the general framework of Markov Decision Problems (MDPs), with possibly unknown parameters. Function approximation is essential for problems with a large state space, as it facilitates compact representation and enables generalization. Linear approximation architectures (where the adjustable parameters are the weights of pre-fixed basis functions) have recently gained prominence due to efficient algorithms and convergence guarantees. Nonetheless, an appropriate choice of basis function is important for the success of the algorithm. In the present paper we examine methods for adapting the basis function during the learning process in the context of evaluating the value function under a fixed control policy. Using the Bellman approximation error as an optimization criterion, we optimize the weights of the basis function while simultaneously adapting the (non-linear) basis function parameters. We present two algorithms for this problem. The first uses a gradient-based approach and the second applies the Cross Entropy method. The performance of the proposed algorithms is evaluated and compared in simulations.
Reinforcement learning with kernels and Gaussian processes

Kernel methods have become popular in many sub-fields of machine learning with the exception of reinforcement learning; they facilitate rich representations, and enable machine learning techniques to work in diverse input spaces. We describe a principled approach to the policy evaluation problem of reinforcement learning. We present a temporal difference (TD) learning using kernel functions. Our approach allows the TD algorithm to work in arbitrary spaces as long as a kernel function is defined in this space. This kernel function is used to measure similarity between states. The value function is described as a Gaussian process and we obtain a Bayesian solution by solving a generative model. A SARSA based extension of the kernel-based TD algorithm is also mentioned.