nils-herrmann/clean_scirepeval huggingface dataset:
Support vector algorithms as regularization networks

Many recently proposed learning algorithms are clearly inspired by Suppport Vector Machines. Some of them were developed while trying to simplify the quadratic programming problem that has to be solved in training SVMs. Some others have been proposed to solve problems other than binary classification (for example one-class SVM for novelty detection). Though indeed attractive, for most of the learning machine community the above algorithms lack of a clear theoretical motivation. In this context it seems that the connection to regularization networks is most promising both from a theoretical and a practical point of view and might be of great use to understand the mathematical properties of various SV algorithms. In this paper we contribute to fill the existing gap reviewing several SV algorithms from a regularization point of view.
ON EARLY STOPPING IN GRADIENT DESCENT BOOSTING

In this paper, we study a family of gradient descent algorithms to approximate the regression function from reproducing kernel Hilbert spaces. Here early stopping plays a role of regularization, where given a finite sample and some regularity condition on the regression function, a stopping rule is given and some probabilistic upper bounds are obtained for the distance between the function iterated at the stopping time and the regression function. A crucial advantage over other regularized least square algorithms studied recently lies in that it breaks through the saturation phenomenon where the convergence rate no longer improves after a certain level of regularity achieved by the regression function. These upper bounds show that in some situations we can achieve optimal convergence rates. We also discuss the implication of these results in the context of classification. Some connections are addressed with the Landweber iteration for regularization in inverse problems and the online learning algorithms as stochastic approximations of gradient descent method.
Model Selection for Regularized Least-Squares Algorithm in Learning Theory

AbstractWe investigate the problem of model selection for learning algorithms depending on a continuous parameter. We propose a model selection procedure based on a worst-case analysis and on a data-independent choice of the parameter. For the regularized least-squares algorithm we bound the generalization error of the solution by a quantity depending on a few known constants and we show that the corresponding model selection procedure reduces to solving a bias-variance problem. Under suitable smoothness conditions on the regression function, we estimate the optimal parameter as a function of the number of data and we prove that this choice ensures consistency of the algorithm.
Empirical Effective Dimension and Optimal Rates for Regularized Least Squares Algorithm

This paper presents an approach to model selection for regularized least-squares on reproducing kernel Hilbert spaces in the semi-supervised setting. The role of effective dimension was recently shown to be crucial in the definition of a rule for the choice of the regularization parameter, attaining asymptotic optimal performances in a minimax sense. The main goal of the present paper is showing how the effective dimension can be replaced by an empirical counterpart while conserving optimality. The empirical effective dimension can be computed from independent unlabelled samples. This makes the approach particularly appealing in the semi-supervised setting.
Learning from Examples as an Inverse Problem

Many works related learning from examples to regularization techniques for inverse problems, emphasizing the strong algorithmic and conceptual analogy of certain learning algorithms with regularization algorithms. In particular it is well known that regularization schemes such as Tikhonov regularization can be effectively used in the context of learning and are closely related to algorithms such as support vector machines. Nevertheless the connection with inverse problem was considered only for the discrete (finite sample) problem and the probabilistic aspects of learning from examples were not taken into account. In this paper we provide a natural extension of such analysis to the continuous (population) case and study the interplay between the discrete and continuous problems. From a theoretical point of view, this allows to draw a clear connection between the consistency approach in learning theory and the stability convergence property in ill-posed inverse problems. The main mathematical result of the paper is a new probabilistic bound for the regularized least-squares algorithm. By means of standard results on the approximation term, the consistency of the algorithm easily follows.
Learning, Regularization and Ill-Posed Inverse Problems

Many works have shown that strong connections relate learning from examples to regularization techniques for ill-posed inverse problems. Nevertheless by now there was no formal evidence neither that learning from examples could be seen as an inverse problem nor that theoretical results in learning theory could be independently derived using tools from regularization theory. In this paper we provide a positive answer to both questions. Indeed, considering the square loss, we translate the learning problem in the language of regularization theory and show that consistency results and optimal regularization parameter choice can be derived by the discretization of the corresponding inverse problem.
Are Loss Functions All the Same?

In this letter, we investigate the impact of choosing different loss functions from the viewpoint of statistical learning theory. We introduce a convexity assumption, which is met by all loss functions commonly used in the literature, and study how the bound on the estimation error changes with the loss. We also derive a general result on the minimizer of the expected risk for a convex loss function in the case of classification. The main outcome of our analysis is that for classification, the hinge loss appears to be the loss of choice. Other things being equal, the hinge loss leads to a convergence rate practically indistinguishable from the logistic loss rate and much better than the square loss rate. Furthermore, if the hypothesis space is sufficiently rich, the bounds obtained for the hinge loss are not loosened by the thresholding stage.
Some Properties of Regularized Kernel Methods

In regularized kernel methods, the solution of a learning problem is found by minimizing functionals consisting of the sum of a data and a complexity term. In this paper we investigate some properties of a more general form of the above functionals in which the data term corresponds to the expected risk. First, we prove a quantitative version of the representer theorem holding for both regression and classification, for both differentiable and non-differentiable loss functions, and for arbitrary offset terms. Second, we show that the case in which the offset space is non trivial corresponds to solving a standard problem of regularization in a Reproducing Kernel Hilbert Space in which the penalty term is given by a seminorm. Finally, we discuss the issues of existence and uniqueness of the solution. From the specialization of our analysis to the discrete setting it is immediate to establish a connection between the solution properties of sparsity and coefficient boundedness and some properties of the loss function. For the case of Support Vector Machines for classification, we also obtain a complete characterization of the whole method in terms of the Khun-Tucker conditions with no need to introduce the dual formulation.
Langmuir-Blodgett Films of a Functionalized Molecule with Cross-Sectional Mismatch Between Head and Tail

filling a n d suitable interful lerene spacing. I t Laboratory. Reliable cubic harmonic C, and ( 4 ~ ) " ~ f d Q ) ill~(OR) CI, Kd@o,+o) (4) C,,,, coeffic~ents were routinely extracted from appears t h a t t h e or ientat ional state o f the 1 % the ref~nement and were in qood aqreement with fullerenes a n d t h e intercalate-carbon in te r the values deduced by chow eta/ . . (9) from where jl (OR) are the 1-th order spherical Bessel a c t i o n subtly c o n t r o l the p i r b i n d i n g mechsingle-crystal data. We thank P. A Heiney for . . .. . functions and f,(Q) is the carbon scattering facmakina his raw data available to us IJ. Phvs. . , an ism in t h e tullerides. tor. Only the coeffic~ents. C,,,, that transform as the Chem; Solids 53.
Rough Times Ahead

Clusters of conventional computers have almost replaced vector-based supercomputers among the world9s 500 biggest computing complexes, thereby tapping a computing resource with lower costs. But as
 Kirkpatrick
 explains in his Perspective, this approach brings new problems. He highlights the report by
 Korniss et al ., who demonstrate that the simulations times of the different processors can diverge very quickly, leading to temporal "roughening." As a result, it can be difficult to synchronize computations performed on many different computers. Intermittent synchronization over random distances can suppress the roughening.
1.2 SATISFIABILITY AND HARD-PROBLEM INSTANCES....... iv

“NP-complete”problemsareat thecoreof many computational tasksof practical interest,whichmeansthatthecostof solutionwill grow exponentiallywith problem sizein the worst case(providedP  NP). Althoughheuristicscanbe quite effective on suchproblemsin most cases,thereis a growing appreciationthat theseproblemscontainphasetransitions,andat thephaseboundariesexponential complexity becomesthetypical outcome,not just theworstcase.Usingmethods from statisticalphysics,a much betterunderstandingof suchphasetransition phenomenain computationalproblemshasbeenobtainedin recentyears. We will review severalkey resultsin this area,therebyillustratingsomeof thedeep connectionsbetweencomputerscienceandstatisticalphysics.Theseminalwork by Fu andAnderson[13, 14] providedtheinitial impetusfor muchof this work.
Spectroscopic ellipsometry measurements of chromium nitride coatings

Coatings of chromium and chromium nitride with various compositions were deposited on silicon substrates using ion-assisted reactive magnetron sputtering. These coatings were characterized using x-ray diffraction (XRD), Rutherford backscattering (RBS), and spectroscopic ellipsometry (SE). The primary chromium nitride phases (Cr2N and CrN) in the chromium nitride films were identified using XRD. The chemical composition of selected samples was determined from RBS measurements. The refractive indices of Cr, Cr2N, and CrN were deduced from the analysis of the SE data. Based on these refractive indices, ellipsometry was used successfully to identify the microstructure and the roughness of a number of chromium nitride coatings grown under different deposition conditions.
2+p-SAT: relation of typical-case complexity to the nature of the phase transition

Heuristic methods for solution of problems in the NP-complete class of decision problems often reach exact solutions, but fail badly at “phase boundaries,” across which the decision to be reached changes from almost always having one value to almost always having a different value. We report an analytic solution and experimental investigations of the phase transition that occurs in the limit of very large problems in K-SAT. Studying a model which interpolates K-SAT between K=2 and K=3, we find a change from a continuous to a discontinuous phase transition when K, the average number of inputs per clause, exceeds 0.4. The cost of finding solutions also increases dramatically above this changeover. The nature of its “random first-order” phase transition, seen at values of K large enough to make the computational cost of solving typical instances increase exponentially with problem size, suggests a mechanism for the cost increase. There has been evidence for features like the “backbone” of frozen inputs which characterizes the UNSAT phase in K-SAT in the study of models of disordered materials, but this feature and this transition are uniquely accessible to analysis in K-SAT. The random first-order transition combines properties of the first-order (discontinuous onset of order) and second-order (with power law scaling, e.g., of the width of the critical region in a finite system) transitions known in the physics of pure solids. Such transitions should occur in other combinatoric problems in the large N limit. Finally, improved search heuristics may be developed when a “backbone” is known to exist.
Determining computational complexity from characteristic ‘phase transitions’

Non-deterministic polynomial time (commonly termed ‘NP-complete’) problems are relevant to many computational tasks of practical interest—such as the ‘travelling salesman problem’—but are difficult to solve: the computing time grows exponentially with problem size in the worst case. It has recently been shown that these problems exhibit ‘phase boundaries’, across which dramatic changes occur in the computational difficulty and solution character—the problems become easier to solve away from the boundary. Here we report an analytic solution and experimental investigation of the phase transition in K -satisfiability, an archetypal NP-complete problem. Depending on the input parameters, the computing time may grow exponentially or polynomially with problem size; in the former case, we observe a discontinuous transition, whereas in the latter case a continuous (second-order) transition is found. The nature of these transitions may explain the differing computational costs, and suggests directions for improving the efficiency of search algorithms. Similar types of transition should occur in other combinatorial problems and in glassy or granular materials, thereby strengthening the link between computational models and properties of physical systems.
2+p-SAT: Relation of typical-case complexity to the nature of the phase transition

Heuristic methods for solution of problems in the NP-Complete class of decision problems often reach exact solutions, but fail badly at "phase boundaries", across which the decision to be reached changes from almost always having one value to almost having a different value. We report an analytic solution and experimental investigations of the phase transition that occurs in the limit of very large problems in K-SAT. The nature of its "random first-order" phase transition, seen at values of K large enough to make the computational cost of solving typical instances increase exponenitally with problem size, suggest a mechanism for the cost increase. There has been evidence for features like the "backbone" of frozen inputs which characterizes the UNSAT phase in K-SAT in the study of models of disordered materials, but this feature and this transition are uniquely accessible to analysis in K-SAT. The random first order transition combines properties of the 1st order (discontinuous onset of order) and 2nd order (with power law scaling, e.g. of the width of the the critical region in a finite system) transitions known in the physics of pure solids. Such transitions should occur in other combinatoric problems in the large N limit. Finally, improved search heuristics may be developed when a "backbone" is known to exist.
Simulated annealing

Simulated annealing (SA) is a generic probabilistic metaheuristic for the global optimization problem of locating a good approximation to the global optimum of a given function in a large search space. It is often used when the search space is discrete (e.g., all tours that visit a given set of cities). For certain problems, simulated annealing may be more efficient than exhaustive enumeration — provided that the goal is merely to find an acceptably good solution in a fixed amount of time, rather than the best possible solution. The name and inspiration come from annealing in metallurgy, a technique involving heating and controlled cooling of a material to increase the size of its crystals and reduce their defects, both are attributes of the material that depend on its thermodynamic free energy. Heating and cooling the material affects both the temperature and the thermodynamic free energy. While the same amount of cooling brings the same amount of decrease in temperature it will bring a bigger or smaller decrease in the thermodynamic free energy depending to the rate that it occurs, with a slower rate producing a bigger decrease. This notion of slow cooling is implemented in the Simulated Annealing algorithm as a slow decrease in the probability of accepting worse solutions as it explores the solution space. Accepting worse solutions is a fundamental property of metaheuristics because it allows for a more extensive search for the optimal solution. The method is an adaptation of the Metropolis-Hastings algorithm, a Monte Carlo method to generate sample states of a thermodynamic system, invented by M.N. Rosenbluth and published in a paper by N. Metropolis et al. in 1953.
Phase transition and search cost in the 2+p-sat problem

We give analytical and numerical results concerning a new Satissability problem for random Boolean expressions. Our model smoothly interpolates between the polynomial 2{SAT problem and the NP{complete 3{SAT problem. Possible consequences on the relationship between the statistical mechanics characterization of phase transitions { particularly smooth second order RSB and rst order RSB transitions { and the onset of exponential behaviour in search algorithms are identiied.
Technology directions for portable computers

This paper contains an evaluation of trends in the key system parameters (e.g., size, weight, function, performance, battery life) for battery-powered portable computers, together with a review of development trends in the technologies required for such systems. The discussion focuses on notebook-size portable computers. Those technologies which will have substantial impact on battery life and power budgets of future notebook computers receive the primary emphasis in this paper for example, liquid crystal displays, storage technology, wireless communication technology, and low power electronics. System power management is also be addressed. The basic theme of this paper is first to develop a view of what the key attributes of future notebook computers will be, and then to discuss how technologies must evolve to allow such systems to be advanced over the current state of the art in terms of portability and battery life.
An Expectation Maximisation Algorithm for One-to-Many Record Linkage, Illustrated on the Problem of Matching Far Infra-Red Astronomical Sources to Optical Counterparts

The problem of record linkage is often seen simply in terms of making links between data points that might be generated from the same source. However, in many cases the grounds for linking items is itself not certain. In fact it is often desirable to learn, in an unsupervised manner, what form linked objects take in dierent databases. One simple case of this is the "one to many" linkage problem, where each object in one dataset is potentially linked to one of many objects in another dataset, and where the candidate matches are mutually exclusive. We show how the Expectation Maximisation algorithm can be used for this matching problem, both to calculate the probability of a match, and to learn something about the characteristics that matched objects have. The approach is derived for the specific astronomical problem of linking far infra-red observations to optical counterparts, but is generally applicable. This report outlines the theory of this record linkage procedure, but does not discuss its application or any implementational details.
Probabilistic inference for computing optimal policies in MDPs

We investigate how the problem of planning in a stochastic environment can be translated into a problem of inference. Previous work on planning by probabilistic inference was limited in that a total time T has to be fixed and that the computed policy is not optimal w.r.t. expected rewards. The generative model we propose considers the total time T as a random variable and we show equivalence to maximizing the expected future return for arbitrary reward functions. Optimal policies are computed via Expectation-Maximization.
The 2005 PASCAL Visual Object Classes Challenge

The PASCAL Visual Object Classes Challenge ran from February to March 2005. The goal of the challenge was to recognize objects from a number of visual object classes in realistic scenes (i.e. not pre-segmented objects). Four object classes were selected: motorbikes, bicycles, cars and people. Twelve teams entered the challenge. In this chapter we provide details of the datasets, algorithms used by the teams, evaluation criteria, and results achieved.
Cosine Transform Priors for Enhanced Decoding of Compressed Images

Image compression methods such as JPEG use quantisation of discrete cosine transform (DCT) coefficients of image blocks to produce lossy compression. During decoding, an inverse DCT of the quantised values is used to obtain the lossy image. These methods suffer from blocky effects from the region boundaries, and can produce poor representations of regions containing sharp edges. Such problems can be obvious artefacts in compressed images but also cause significant problems for many super-resolution algorithms. Prior information about the DCT coefficients of an image and the continuity between image blocks can be used to improve the decoding using the same compressed image information. This paper analyses empirical priors for DCT coefficients, and shows how they can be combined with block edge contiguity information to produce decoding methods which reduce the blockiness of images. We show that the use of DCT priors is generic can be useful in many other circumstances.
Cleaning sky survey data bases using Hough transform and renewal string approaches

Large astronomical data bases obtained from sky surveys such as the SuperCOSMOS Sky Survey (SSS) invariably suffer from spurious records coming from the artefactual effects of the telescope, satellites and junk objects in orbit around the Earth and physical defects on the photographic plate or CCD. Though relatively small in number, these spurious records present a significant problem in many situations, where they can become a large proportion of the records potentially of interest to a given astronomer. Accurate and robust techniques are needed for locating and flagging such spurious objects, and we are undertaking a programme investigating the use of machine learning techniques in this context. In this paper we focus on the four most common causes of unwanted records in the SSS: satellite or aeroplane tracks, scratches, fibres and other linear phenomena introduced to the plate, circular haloes around bright stars due to internal reflections within the telescope and diffraction spikes near to bright stars. Appropriate techniques are developed for the detection of each of these. The methods are applied to the SSS data to develop a data set of spurious object detections, along with confidence measures, which can allow these unwanted data to be removed from consideration. These methods are general and can be adapted to other astronomical survey data.
Generalised Propagation for Fast Fourier Transforms with Partial or Missing Data

Discrete Fourier transforms and other related Fourier methods have been practically implementable due to the fast Fourier transform (FFT). However there are many situations where doing fast Fourier transforms without complete data would be desirable. In this paper it is recognised that formulating the FFT algorithm as a belief network allows suitable priors to be set for the Fourier coefficients. Furthermore efficient generalised belief propagation methods between clusters of four nodes enable the Fourier coefficients to be inferred and the missing data to be estimated in near to O(n log n) time, where n is the total of the given and missing data points. This method is compared with a number of common approaches such as setting missing data to zero or to interpolation. It is tested on generated data and for a Fourier analysis of a damaged audio signal.