nils-herrmann/clean_scirepeval huggingface dataset:
The Leave-One-Out Kernel

Recently, several attempts have been made for deriving data-dependent kernels from distribution estimates with parametric models (e.g. the Fisher kernel). In this paper, we propose a new kernel derived from any distribution estimators, parametric or nonparametric. This kernel is called the Leave-one-out kernel (i.e. LOO kernel), because the leave-one-out process plays an important role to compute this kernel. We will show that, when applied to a parametric model, the LOO kernel converges to the Fisher kernel asymptotically as the number of samples goes to infinity.
An introduction to kernel-based learning algorithms

This paper provides an introduction to support vector machines, kernel Fisher discriminant analysis, and kernel principal component analysis, as examples for successful kernel-based learning methods. We first give a short background about Vapnik-Chervonenkis theory and kernel feature spaces and then proceed to kernel based learning in supervised and unsupervised scenarios including practical and algorithmic considerations. We illustrate the usefulness of kernel algorithms by discussing applications such as optical character recognition and DNA analysis.
Pairwise coupling for machine recognition of hand-printed Japanese characters

Machine recognition of hand-printed Japanese characters has been an area of great interest for many years. A major problem of this classification task is the huge number of different characters. Applying standard "state-of-the-art" techniques, such as SVM, to multi-class problems of this kind imposes severe problems of both a conceptual and technical nature: (i) separating one class from all others may be an unnecessarily hard problem; and (ii) solving these subproblems can impose unacceptably high computational costs. In this paper, a new approach to Japanese character recognition is presented that successfully overcomes these shortcomings. It is based on a pairwise coupling procedure for probabilistic two-class kernel classifiers. Experimental results for Hiragana recognition effectively demonstrate that our method attains an excellent level of prediction accuracy while imposing very low computational costs.
Subspace Information Criterion for Sparse Regressors

Non-quadratic regularizers, in particular the ` 1 norm regularizer can yield sparse solutions that generalize well. In this work we propose the Generalized Subspace Information Criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the ` 1 norm regularizer as we compare with the Network Information Criterion and cross-validation in relatively large sample cases. However in the small sample cases, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high dimensional data and few samples.
Learning to Predict the Leave-One-Out Error of Kernel Based Classifiers

We propose an algorithm to predict the leave-one-out (LOO) error for kernel based classifiers. To achieve this goal with computational efficiency, we cast the LOO error approximation task into a classification problem. This means that we need to learn a classification of whether or not a given training sample - if left out of the data set - would be misclassified. For this learning task, simple data dependent features are proposed, inspired by geometrical intuition. Our approach allows to reliably select a good model as demonstrated in simulations on Support Vector and Linear Programming Machines. Comparisons to existing learning theoretical bounds, e.g. the span bound, are given for various model selection scenarios.
The subspace method in Hilbert space

The subspace method has usually been applied to a multidimensional space (i.e., feature space) which uses features as its basis. A subspace method can also be applied to a functional space, since the subspace can be defined by an arbitrary linear space. This paper proposes the mapping of a feature space onto the Hilbert subspace so that pattern recognition can be performed by the subspace method. The proposed method has been experimentally applied to the recognition of the Japanese hiragana syllabary and has achieved a higher recognition rate than the conventional method. This improvement is based on the fact that the dimension of the common part in all of the classes of the subspaces in the Hilbert function space becomes zero.
Large margin classifier via semiparametric inference

In this paper, we construct a learning method of stochastic perceptron based on semiparametric inference, and show that this method produces large margin solutions. In semiparametric inference, the parameters are divided into structural parameters which are to be estimated and nuisance parameters in which we do not have any interest. Here, the weight vector of perceptron is defined as structural parameters and the steepness of transfer function is defined as a nuisance parameter. Usually, rough estimate is substituted to nuisance parameters and only structural parameters are estimated. To compensate the estimation error caused by rough estimate, an additional term is added to the derivative of likelihood. We will show that this additional term is related to the regularization term which causes large margin solutions. This work suggests that the success of large margin classifiers can be attributed to semiparametric inference.
Optimal Hyperplane Classier with Adaptive Norm

The conventional optimal hyperplane classi er is de ned on Euclidean space where the norm is given a priori. In this paper, we propose a new optimal hyperplane classi er in which the norm also adapts for learning. For practical implementation, the norm is restricted to a weighted Euclidean norm and the weights are controlled in learning. The statistical properties of this classi er is analyzed via the generalization bound characterized by entropy numbers. An iterative training algorithm is designed to minimize the generalization bound while keeping the complete separation of training samples. As a result of online character recognition experiments, the optimal hyperplane classi er with adaptive norm outperformed the conventional one.
Support Vector Classi er with Asymmetric Kernel Functions

In support vector classi er, asymmetric kernel functions are not used so far, although they are frequently used in other kernel classi ers. The applicable kernels are limited to symmetric semipositive de nite ones because of Mercer's theorem. In this paper, SVM is extended to be applicable to asymmetric kernel functions. It is proven that, when a positive de nite kernel is given, the extended SVM is identical with the conventional SVM. In the 3D object recognition experiment, the extended SVM with asymmetric kernels performed better than the conventional SVM.
Pattern Recognition Method for Metric Space by Four Points Embedding

In solving pattern recognition problem in the Euclidean space, prototypes representing classes are de ned. On the other hand in the metric space, Nearest Neighbor method and K-Nearest Neighbor method are frequently used without de ning any prototypes. In this paper, we propose a new pattern recognition method for the metric space that can use prototypes which are the centroid of any three patterns in a class. This method is based on the theorem that four points of any metric space can be embedded into the Euclidean space by an appropriate metric transform.
Sequential fuzzy cluster extraction and its robustness against noise

Partitional clustering methods such as C-Means classify all samples into clusters. Even a noise sample that is distant from any cluster is assigned to one of the clusters. Noise samples included in clusters bias the clustering result and tend to produce meaningless clusters. Our clustering method repeats to extract mutually close samples as a cluster and leave isolated noises unclustered. Thus, the produced clusters are less a ected by noises than those of C-Means. Because clusters can be obtained analytically by our method, repeated trials to avoid local minima are not necessary. The method is shown to be e ective for extracting straight lines from images in the experiments.
Sequential fuzzy cluster extraction and its robustness against noise

Partitional clustering methods such as C-Means classify all samples into clusters. Even a noise sample that is distant from any cluster is assigned to one of the clusters. Noise samples included in clusters bias the clustering result and tend to produce meaningless clusters. Our clustering method repeatedly extracts mutually close samples as a cluster and leaves isolated noises unclustered. Thus, the produced clusters are less affected by noises than those of C-Means. Because clusters can be obtained analytically by our method, repeated trials to avoid local minima are not necessary. The method is shown to be effective for extracting straight lines from images in the experiments.
A nonparametric density model for classification in a high dimensional space

When the dimensionality of the feature space increases and takes beyond a certain point, the classification performance of a parametric classifier begins to deteriorate. This is because the number of parameters of the classifier depends on the dimensionality and gets too large in a high dimensional space. To obtain the density value at the point of interest without deriving the parameters, we propose a nonparametric Gaussian density model, where the sum of the log-density values at two points is described, without any parameter, by a function of the distance between the two points. The density value at the point of interest is estimated from the distance between the point and each of the training sample points using this model. We will empirically show that the density values estimated by the nonparametric model are more accurate than those by the parametric model in a high dimensional space. In a character recognition experiment, our nonparametric classifier achieved higher classification accuracy in comparison with the parametric classifier.
Clustering OCR-ed texts for browsing document image database

Document clustering is a powerful tool for browsing throughout a document database. Similar documents are gathered into several clusters and a representative document of each cluster is shown to users. To make users infer the content of the database from several representatives, the documents must be separated into tight clusters, in which documents are connected with high similarities. At the same time, clustering must be fast for user interaction. We propose an O(n/sup 2/) time, O(n) space cluster extraction method. It is faster than the ordinal clustering methods, and its clusters compare favorably with those produced by Complete Link for tightness. When we deal with OCR-ed documents, term loss caused by recognition faults can change similarities between documents. We also examined the effect of recognition faults to the performance of document clustering.
First measurement of the clustering evolution of photometrically classified quasars

We present new measurements of the quasar angular autocorrelation function from a sample of ~80,000 photometrically classified quasars taken from the First Data Release of the Sloan Digital Sky Survey. We find a best-fit model of ω(θ) = (0.066)θ-(0.98±0.15) for the angular correlation function, consistent with estimates of the slope from spectroscopic quasar surveys. We show that only models with little or no evolution in the clustering of quasars in comoving coordinates since a median redshift of z ~ 1.4 can recover a scale length consistent with local galaxies and active galactic nuclei (AGNs). A model with little evolution of quasar clustering in comoving coordinates is best explained in the current cosmological paradigm by rapid evolution in quasar bias. We show that quasar biasing must have changed from bQ ~ 3 at a (photometric) redshift of phot = 2.2 to bQ ~ 1.2-1.3 by phot = 0.75. Such a rapid increase with redshift in biasing implies that quasars at z ~ 2 cannot be the progenitors of modern L* objects; rather they must now reside in dense environments, such as clusters. Similarly, the duration of the UVX (ultraviolet-excess) quasar phase must be short enough to explain why local UVX quasars reside in essentially unbiased structures. Our estimates of bQ are in good agreement with recent spectroscopic results (Croom et al. 2005), which demonstrate that the implied evolution in bQ is consistent with quasars inhabiting halos of similar mass at every redshift. Treating quasar clustering as a bivariate function of both redshift and luminosity, we find no evidence for luminosity dependence in quasar clustering, and that redshift evolution thus affects quasar clustering more than changes in quasars' luminosity. Our results are robust against a range of systematic uncertainties. We provide a new method for quantifying stellar contamination in photometrically classified quasar catalogs via the correlation function.
Dual-Tree Fast Gauss Transforms

In previous work we presented an efficient approach to computing kernel summations which arise in many machine learning methods such as kernel density estimation. This approach, dual-tree recursion with finite-difference approximation, generalized existing methods for similar problems arising in computational physics in two ways appropriate for statistical problems: toward distribution sensitivity and general dimension, partly by avoiding series expansions. While this proved to be the fastest practical method for multivariate kernel density estimation at the optimal bandwidth, it is much less efficient at larger-than-optimal bandwidths. In this work, we explore the extent to which the dual-tree approach can be integrated with multipole-like Hermite expansions in order to achieve reasonable efficiency across all bandwidth scales, though only for low dimensionalities. In the process, we derive and demonstrate the first truly hierarchical fast Gauss transforms, effectively combining the best tools from discrete algorithms and continuous approximation theory.
An Application Performance Comparison of HPCx to EPCC's Blue Gene/L Service

A slug of solution, comprising a linear polymer such as polysulfone, polyethylene terephthalate or polyimide dissolved in a "good" solvent, such as morpholine, dimethylformamide m-cresol or dimethylsulfoxide, respectively, is emplaced in the near-wellbore region of an unconsolidated sand reservoir. A second slug comprising a poor solvent for the polymer, such as water or water+2-methanol is pumped into the region to contact the first slug. The linear polymer is precipitated and forms a three-dimensional network of interconnected strands extending through the fluid flow channels between the sand grains. The network functions to consolidate the sand without significantly damaging permeability. Petroleum and other fluids can then be produced without loose sand being entrained in the fluids. If necessary, this permeable network of plastic threads can be removed by re-injecting a slug of the good solvent to re-dissolve the plastic and reform the original linear polymer solution.
Multi-Tree Methods for Statistics on Very Large Datasets in Astronomy

Many fundamental statistical methods have become critical tools for scientific data analysis yet do not scale tractably to modern large datasets. This paper will describe very recent algorithms based on computational geometry which have dramatically reduced the computational complexity of 1) kernel density estimation (which also extends to nonparametric regression, classification, and clustering), and 2) the n-point correlation function for arbitrary n. These new multi-tree methods typically yield orders of magnitude in speedup over the previous state of the art for similar accuracy, making millions of data points tractable on desktop workstations for the first time.
Fast kernel matrix-vector multiplication with application to Gaussian process learning

A number of core computational problems in machine learning, both old and new, can be cast as a matrixvector multiplication between a kernel matrix or class-probability matrix and a vector of weights. This arises prominently, for example, in the kernel estimation methods of nonparametric statistics, many common probabilistic graphical models, and the more recent kernel machines. After highlighting the existence of this computational problem in several well-known machine learning methods, we focus on a solution for one specific example for clarity, Gaussian process (GP) prediction one whose applicability has been particularly hindered by this computational barrier. We demonstrate the application of a recent JV-body approach developed specifically for statistical problems, employing adaptive computational geometry and finite-difference approximation. This core algorithm reduces the O(N) matrix-vector multiplications within GP learning to O(N), making the resulting overall learning algorithm O(N). GP learning for N = 1 million points is demonstrated. 1 Kernel Matrix-Vector Multiplications in Learning A kernel matrix $ contains the kernel interaction of each point x^ in a query (test) dataset 2LQ (having size NQ) with each point from a reference (training) dataset X_n (having size iV^), where the kernel function K() often has some scale parameter a (the 'bandwidth'). Often the 'kernel function' is actually a probability density function, such as the Gaussian. In such cases $ is typically a class probability matrix. The query and reference set can be the same set. Often the core computational cost of a statistical method boils down to a multiplcation of this matrix $ with some vector of weights w. For example, in the weighted form of kernel density estimation, the density estimate at the q test point x_q
An Investigation of Practical Approximate Nearest Neighbor Algorithms

This paper concerns approximate nearest neighbor searching algorithms, which have become increasingly important, especially in high dimensional perception areas such as computer vision, with dozens of publications in recent years. Much of this enthusiasm is due to a successful new approximate nearest neighbor approach called Locality Sensitive Hashing (LSH). In this paper we ask the question: can earlier spatial data structure approaches to exact nearest neighbor, such as metric trees, be altered to provide approximate answers to proximity queries and if so, how? We introduce a new kind of metric tree that allows overlap: certain datapoints may appear in both the children of a parent. We also introduce new approximate k-NN search algorithms on this structure. We show why these structures should be able to exploit the same random-projection-based approximations that LSH enjoys, but with a simpler algorithm and perhaps with greater efficiency. We then provide a detailed empirical evaluation on five large, high dimensional datasets which show up to 31-fold accelerations over LSH. This result holds true throughout the spectrum of approximation levels.
Efficient photometric selection of quasars from the sloan digital sky survey: 100,000 z < 3 quasars from data release one

We present a catalog of 1,172,157 quasar candidates selected from the photometric imaging data of the Sloan Digital Sky Survey (SDSS). The objects are all point sources to a limiting magnitude of i = 21.3 from 8417 deg2 of imaging from SDSS Data Release 6 (DR6). This sample extends our previous catalog by using the latest SDSS public release data and probing both ultraviolet (UV)-excess and high-redshift quasars. While the addition of high-redshift candidates reduces the overall efficiency (quasars:quasar candidates) of the catalog to ~80%, it is expected to contain no fewer than 850,000 bona fide quasars, which is ~8 times the number of our previous sample and ~10 times the size of the largest spectroscopic quasar catalog. Cross-matching between our photometric catalog and spectroscopic quasar catalogs from both the SDSS and 2dF survey yields 88,879 spectroscopically confirmed quasars. For judicious selection of the most robust UV-excess sources (~500, 000 objects in all), the efficiency is nearly 97%—more than sufficient for detailed statistical analyses. The catalog's completeness to type 1 (broad-line) quasars is expected to be no worse than 70%, with most missing objects occurring at z < 0.7 and 2.5 < z < 3.0. In addition to classification information, we provide photometric redshift estimates (typically good to Δz ± 0.3 [2σ]) and cross-matching with radio, X-ray, and proper-motion catalogs. Finally, we consider the catalog's utility for determining the optical luminosity function of quasars and are able to confirm the flattening of the bright-end slope of the quasar luminosity function at z ~ 4 as compared to z ~ 2.
Efficient Photometric Selection of Quasars from the Sloan Digital Sky Survey: 100,000
 z
 < 3 Quasars from Data Release One

We present a catalog of 1,172,157 quasar candidates selected from the photometric imaging data of the Sloan Digital Sky Survey (SDSS). The objects are all point sources to a limiting magnitude of i=21.3 from 8417 sq. deg. of imaging from SDSS Data Release 6 (DR6). This sample extends our previous catalog by using the latest SDSS public release data and probing both UV-excess and high-redshift quasars. While the addition of high-redshift candidates reduces the overall efficiency (quasars:quasar candidates) of the catalog to ~80%, it is expected to contain no fewer than 850,000 bona fide quasars -- ~8 times the number of our previous sample, and ~10 times the size of the largest spectroscopic quasar catalog. Cross-matching between our photometric catalog and spectroscopic quasar catalogs from both the SDSS and 2dF Surveys, yields 88,879 spectroscopically confirmed quasars. For judicious selection of the most robust UV-excess sources (~500,000 objects in all), the efficiency is nearly 97% -- more than sufficient for detailed statistical analyses. The catalog's completeness to type 1 (broad-line) quasars is expected to be no worse than 70%, with most missing objects occurring at z<0.7 and 2.5<z<3.0. In addition to classification information, we provide photometric redshift estimates (typically good to Delta z +/- 0.3 [2 sigma]) and cross-matching with radio, X-ray, and proper motion catalogs. Finally, we consider the catalog's utility for determining the optical luminosity function of quasars and are able to confirm the flattening of the bright-end slope of the quasar luminosity function at z~4 as compared to z~2.