nils-herrmann/clean_scirepeval huggingface dataset:
Part of Speech Tagging Using a Network of Linear Separators

We present an architecture and an on-line learning algorithm and apply it to the problem of part-of-speech tagging. The architecture presented, SNOW, is a network of linear separators in the feature space, utilizing the Winnow update algorithm.Multiplicative weight-update algorithms such as Winnow have been shown to have exceptionally good behavior when applied to very high dimensional problems, and especially when the target concepts depend on only a small subset of the features in the feature space. In this paper we describe an architecture that utilizes this mistake-driven algorithm for multi-class prediction-selecting the part of speech of a word. The experimental analysis presented here provides more evidence to that these algorithms are suitable for natural language problems.The algorithm used is an on-line algorithm: every example is used by the algorithm only once, and is then discarded. This has significance in terms of efficiency, as well as quick adaptation to new contexts.We present an extensive experimental study of our algorithm under various conditions; in particular, it is shown that the algorithm performs comparably to the best known algorithms for POS.
Mistake-Driven Learning in Text Categorization

Learning problems in the text processing domain often map the text to a space whose dimensions are the measured features of the text, e.g., its words. Three characteristic properties of this domain are (a) very high dimensionality, (b) both the learned concepts and the instances reside very sparsely in the feature space, and (c) a high variation in the number of active features in an instance. In this work we study three mistake-driven learning algorithms for a typical task of this nature -- text categorization. We argue that these algorithms -- which categorize documents by learning a linear separator in the feature space -- have a few properties that make them ideal for this domain. We then show that a quantum leap in performance is achieved when we further modify the algorithms to better address some of the specific characteristics of the domain. In particular, we demonstrate (1) how variation in document length can be tolerated by either normalizing feature weights or by using negative weights, (2) the positive effect of applying a threshold range in training, (3) alternatives in considering feature frequency, and (4) the benefits of discarding features while training. Overall, we present an algorithm, a variation of Littlestone's Winnow, which performs significantly better than any other algorithm tested on this task using a similar feature set.
Linear Concepts and Hidden Variables: An Empirical Study

Some learning techniques for classification tasks work indirectly, by first trying to fit a full probabilistic model to the observed data. Whether this is a good idea or not depends on the robustness with respect to deviations from the postulated model. We study this question experimentally in a restricted, yet non-trivial and interesting case: we consider a conditionally independent attribute (CIA) model which postulates a single binary-valued hidden variable z on which all other attributes (i.e., the target and the observables) depend. In this model, finding the most likely value of anyone variable (given known values for the others) reduces to testing a linear function of the observed values. 
 
We learn CIA with two techniques: the standard EM algorithm, and a new algorithm we develop based on covariances. We compare these, in a controlled fashion, against an algorithm (a version of Winnow) that attempts to find a good linear classifier directly. Our conclusions help delimit the fragility of using the CIA model for classification: once the data departs from this model, performance quickly degrades and drops below that of the directly-learned linear classifier.
A Connectionist Framework for Reasoning: Reasoning with Examples

We present a connectionist architecture that supports almost instantaneous deductive and abductive reasoning. The deduction algorithm responds in few steps for single rule queries and in general, takes time that is linear with the number of rules in the query. The abduction algorithm produces an explanation in few steps and the best explanation in time linear with the size of the assumption set. The size of the network is polynomially related to the size of other representations of the domain, and may even be smaller. 
 
We base our connectionist model on Valiant's Neuroidal model (Val94) and thus make minimal assumptions about the computing elements, which are assumed to be classical threshold elements with states. Within this model we develop a reasoning framework that utilizes a model-based approach to reasoning (KKS93; KR94b). In particular, we suggest to interpret the connectionist architecture as encoding examples of the domain we reason about and show how to perform various reasoning tasks with this interpretation. We then show that the representations used can be acquired efficiently from interactions with the environment and discuss how this learning process influences the reasoning performance of the network.
Applying Winnow to Context-Sensitive Spelling Correction

Multiplicative weight-updating algorithms such as Winnow have been studied extensively in the COLT literature, but only recently have people started to use them in applications. In this paper, we apply a Winnow-based algorithm to a task in natural language: context-sensitive spelling correction. This is the task of fixing spelling errors that happen to result in valid words, such as substituting {\it to\/} for {\it too}, {\it casual\/} for {\it causal}, and so on. Previous approaches to this problem have been statistics-based; we compare Winnow to one of the more successful such approaches, which uses Bayesian classifiers. We find that: (1)~When the standard (heavily-pruned) set of features is used to describe problem instances, Winnow performs comparably to the Bayesian method; (2)~When the full (unpruned) set of features is used, Winnow is able to exploit the new features and convincingly outperform Bayes; and (3)~When a test set is encountered that is dissimilar to the training set, Winnow is better than Bayes at adapting to the unfamiliar test set, using a strategy we will present for combining learning on the training set with unsupervised learning on the (noisy) test set.
On Learning Visual Concepts and DNF Formulae

We consider the problem of learning DNF formulae in the mistake-bound and the PAC models. We develop a new approach, which is called polynomial explainability, that is shown to be useful for learning some new subclasses of DNF (and CNF) formulae that were not known to be learnable before. Unlike previous learnability results for DNF (and CNF) formulae, these subclasses are not limited in the number of terms or in the number of variables per term; yet, they contain the subclasses of κ-DNF and κ-term-DNF (and the corresponding classes of CNF) as special cases. We apply our DNF results to the problem of learning visual concepts and obtain learning algorithms for several natural subclasses of visual concepts that appear to have no natural boolean counterpart. On the other hand, we show that learning some other natural subclasses of visual concepts is as hard as learning the class of all DNF formulae. We also consider the robustness of these results under various types of noise.
Default-Reasoning with Models

Reasoning with model-based representations is an intuitive paradigm, which has been shown to be theoretically sound and to possess some computational advantages over reasoning with formula-based representations of knowledge. In this paper we present more evidence to the utility of such representations. 
 
In real life situations, one normally completes a lot of missing "context" information when answering queries. We model this situation by augmenting the available knowledge about the world with context-specific information; we show that reasoning with model-based representations can be done efficiently in the presence of varying context information. We then consider the task of default reasoning. We show that default reasoning is a generalization of reasoning within context, in which the reasoner has many "context" rules, which may be conflicting. We characterize the cases in which model-based reasoning supports efficient default reasoning and develop algorithms that handle efficiently fragments of Reiter's default logic. In particular, this includes cases in which performing the default reasoning task with the traditional, formula-based, representation is intractable. 
 
Further, we argue that these results support an incremental view of reasoning in a natural way.
Knowledge Representation for Semantic Entailment and Question-Answering

Semantic entailment is the problem of determining if the meaning of a given sentence entails that of another.Question-answering can be reduced to this problem by rephrasing the question as a statement that is entailed by correct answers. In [Brazet al., ] we present a principled approach to semantic entailment that builds on inducing re-representations of text snippets into a hierarchical knowledge representation along with an optimization-based inferential mechanism that makes use of it to prove semantic entailment. This paper provides details and analysis of the knowledge representation and knowledge resources issues in the above approach. We analyze our system’s behavior on a collection of question-answer pairs and use it to motivate and explain some of the design decisions in our hierarchical knowledge representation, that is centered around a predicateargument type abstract representation of text.
Learning to Reason: The Non-Monotonic Case

We suggest a new approach for the study of the non monotonicity of human commonsense reasoning. The two main premises that underlie this work are that commonsense reasoning is an inductive phenomenon and that missing information in the interaction of the agent with the environment may be as informative for future interactions as observed information. This intuition is normalized and the problem of reasoning from incomplete information is presented as a problem of learning attribute functions over a generalized domain. 
 
We consider examples that illustrate various aspects of the non monotonic reasoning phenomena which have been used over the years as bench marks for various formalisms and translate them into Learning to Reason problems. We demonstrate that these have concise representations over the generalized domain and prove that these representations can be learned efficiently. 
 
The framework developed suggests an operational approach to studying reasoning that is nevertheless rigorous and amenable to analysis. We show that this approach efficiently supports reasoning with incomplete information and at the same lime matches our expectations of plausible patterns of reasoning in cases where other theories do not. 
 
This work continues previous works in the Learning to Reason framework and supports the thesis that in order to develop a computational account for commonsense reasoning one should study the phenomena of learning and reasoning together.
A Probabilistic Interpretation of SVMs with an Application to Unbalanced Classification

In this paper, we show that the hinge loss can be interpreted as the neg-log-likelihood of a semi-parametric model of posterior probabilities. From this point of view, SVMs represent the parametric component of a semi-parametric model fitted by a maximum a posteriori estimation procedure. This connection enables to derive a mapping from SVM scores to estimated posterior probabilities. Unlike previous proposals, the suggested mapping is interval-valued, providing a set of posterior probabilities compatible with each SVM score. This framework offers a new way to adapt the SVM optimization problem to unbalanced classification, when decisions result in unequal (asymmetric) losses. Experiments show improvements over state-of-the-art procedures.
Model selection via penalization in the additive Cox model

The Cox proportional hazards model is the most popular model for the analysis of survival data. It allows estimating the relationship between covariates and a possibly censored failure time. The corresponding partial likelihood estimators are used for the estimation and prediction of relative risk of failure. However, if the explanatory variables are highly correlated or if the number of failures is not much greater than the number of covariates of interest, then partial likelihood estimators are unstable and have large variance. Penalization is extensively used to address these difficulties. It decreases the predictor variability to improve the accuracy of prediction. Ridge regression (l2-penalization) is one of the main penalization procedures. It has been generalized to the nonparametric setting to reduce the possibility of overfitting with high dimensional models. Thus, smoothing splines are used to estimate flexibly covariate effects in the additive Cox model. Tibshirani's lasso (l1-penalization) has also been applied to the Cox model, providing an alternative to quadratic penalization. An attractive feature of the l1-penalization is that it shrinks coefficients and sets some of them to zero, performing parameter estimation and variable selection simultaneously. We propose a new algorithm for variable selection and function estimation in the additive Cox model. The method is based on a generalization of the lasso to the nonparametric setting. Our proposal maximizes a penalized partial likelihood that includes a double penalty: on the l1-norm of linear components and on the (generalized) l1-norm of nonlinear components of spline coefficients. Because of their nature, these penalties shrink linear and nonlinear compounds, some of them reducing exactly to zero. Hence they give parsimonious models, select significant variables, and reveal nonlinearities in the effects of predictors. Our approach is compared to standard methods by simulations and an example. Different techniques to choose the penalty parameters are also tested.
Discrimination par modèles additifs parcimonieux

We propose a nonparametric classification method designed to support the inter-pretability of the prediction. On the one hand, the use of generalized additive models makes it possible to represent the effect of each input variable on the output variable graphically. On the other hand, parameters of this model are estimated via penalized likelihood, where the term of regularization generalizes the l 1 -penalization to the splines functions. This penalization favors parsimonious solutions selecting one part of the set of input variables, while allowing a flexible modeling of the dependence on the selected variables. We study the adaptation of various analytical model selection criteria to these models, and we evaluate them on two real data sets.
Recycling experiments for sludge monitoring in waste water treatment

Belt filter presses are widely used to remove water from residual sludge generated in wastewater treatment plants. The texture (fine or coarse) of the conditioned sludge is a relevant piece of information used by operators to control the dewatering equipment. A software sensor, which takes digital images as input and outputs a signal that is correlated to sludge dry content, was developed. This signal can be used as an input to an automated control system. To characterize the textures, features were computed by the use of a bank of Gabor filters. A model that ranks images according to their corresponding dry contents was calibrated from a database composed of digital images and dry content analysis results. A control system using this information makes it possible to reduce human intervention which is, till today, essential to assure proper operation of belt filter presses.
Monitoring of sludge dewatering equipment by image classification

Belt filter presses represent an economical means to dewater the residual sludge generated in wastewater treatment plants. In order to assure maximal water removal, the raw sludge is mixed with a chemical conditioner prior to being fed into the belt filter press. When the conditioner is properly dosed, the sludge acquires a coarse texture, with space between flocs. This information was exploited for the development of a software sensor, where digital images are the input signal, and the output is a numeric value proportional to the dewatered sludge dry content. Three families of features were used to characterize the textures. Gabor filtering, wavelet decomposition and co-occurrence matrix computation were the techniques used. A database of images, ordered by their corresponding dry contents, was used to calibrate the model that calculates the sensor output. The images were separated in groups that correspond to single experimental sessions. With the calibrated model, all images were correctly ranked within an experiment session. The results were very similar regardless of the family of features used. The output can be fed to a control system, or, in the case of fixed experiment conditions, it can be used to directly estimate the dewatered sludge dry content.
Bagging Equalizes Influence

Bagging constructs an estimator by averaging predictors trained on bootstrap samples. Bagged estimates almost consistently improve on the original predictor. It is thus important to understand the reasons for this success, and also for the occasional failures. It is widely believed that bagging is effective thanks to the variance reduction stemming from averaging predictors. However, seven years from its introduction, bagging is still not fully understood. This paper provides experimental evidence supporting the hypothesis that bagging stabilizes prediction by equalizing the influence of training examples. This effect is detailed in two different frameworks: estimation on the real line and regression. Bagging’s improvements/deteriorations are explained by the goodness/badness of highly influential examples, in situations where the usual variance reduction argument is at best questionable. Finally, reasons for the equalization effect are advanced. They support that other resampling strategies such as half-sampling should provide qualitatively identical effects while being computationally less demanding than bootstrap sampling.
Généralisation du lasso aux modèles additifs

Nous presentons une nouvelle methode d'estimation fonctionnelle dans le cadre des modeles additifs non parametriques ajustes par des splines cubiques de lissage. Cette methode generalise le "lasso" propose par Tibshirani (1995) pour les modeles lineaires. Comme dans le cas lineaire, certains coefficients sont retrecis, alors que les autres sont annules exactement, aboutissant ainsi a des modeles parcimonieux, selectionnant les variables jugees significatives. Les solutions sont calculees par un algorithme de point fixe, dans lequel une decomposition en valeurs singulieres permet de reduire considerablement le nombre de calculs. Notre approche est validee dans une etude experimentale comparant les performances du lasso a la selection de variables pas a pas.
One step ahead forecasting using Multilayered perceptron St

When dealing with time series the one step ahead forecasting problem based on experimental data is the problem of estimating the autoregression function of the underlying process When minimizing the expected forecast ing error is the main goal the exible approach has to be used to be able to adjust the complexity of the model to the complexity of the data Multilay ered perceptrons are a popular example of such a exible approach but not the only one Other methods such as kernel approximator e g Naradaya Watson regressor regression spline or wavelet regressor can also be used But whatever exible approach is the main issue remains the control of the complexity of the exible approximator Noise injection in the inputs is an e cient technique to do so The complexity of the regessor is then adjusted thanks to the quantity variance of injected noise This quantity is tuned using a bootstrap estimation of the forecasting error One unexpected e ect of this approach is the possibility to prove the consistency of the estimator under some assumptions about the underlying process who generates the time series The two main assumptions are the process is varying signi cantly and the process is bounded Furthermore the boundary conditions imply to eliminate the tendency for the remaining process to be bounded This theoret ical result permits to design a methodology for using multilayered perceptrons to forecast The whole approach was applied successfully to forecast the daily water consumption in the south of Paris
Adaptive Scaling for Feature Selection in SVMs

This paper introduces an algorithm for the automatic relevance determination of input variables in kernelized Support Vector Machines. Relevance is measured by scale factors defining the input space metric, and feature selection is performed by assigning zero weights to irrelevant variables. The metric is automatically tuned by the minimization of the standard SVM empirical risk, where scale factors are added to the usual set of parameters defining the classifier. Feature selection is achieved by constraints encouraging the sparsity of scale factors. The resulting algorithm compares favorably to state-of-the-art feature selection procedures and demonstrates its effectiveness on a demanding facial expression recognition problem.
Bagging improves uncertainty representation in evidential pattern classification

Uncertainty representation is a major issue in pattern recognition when the outputs of a classifier do not lead directly to a final decision, but are used in combination with other systems, or as input to an interactive decision process. In such contexts, it may be advantageous to resort to rich and flexible formalisms for representing and manipulating uncertain information, such as the Dempster-Shafer theory of Evidence. In this paper, it is shown that the quality and reliability of the outputs from an evidence-theoretic classifier may be improved using an adaptation from a resample-and-combine approach introduced by Breiman and known as "bagging". This approach is explained and studied experimentally using simulated data. In particular, results show that bagging improves classification accuracy and limits the influence of outliers and ambiguous training patterns.
Semi-supervised MarginBoost

In many discrimination problems a large amount of data is available but only a few of them are labeled. This provides a strong motivation to improve or develop methods for semi-supervised learning. In this paper, boosting is generalized to this task within the optimization framework of MarginBoost. We extend the margin definition to unlabeled data and develop the gradient descent algorithm that corresponds to the resulting margin cost function. This meta-learning scheme can be applied to any base classifier able to benefit from unlabeled data. We propose here to apply it to mixture models trained with an Expectation-Maximization algorithm. Promising results are presented on benchmarks with different rates of labeled data.
Bagging Can Stabilize without Reducing Variance

Bagging is a procedure averaging estimators trained on bootstrap samples. Numerous experiments have shown that bagged estimates almost consistently yield better results than the original predictor. It is thus important to understand the reasons for this success, and also for the occasional failures. Several arguments have been given to explain the effectiveness of bagging, among which the original "bagging reduces variance by averaging" is widely accepted. This paper provides experimental evidence supporting another explanation, based on the stabilization provided by spreading the influence of examples. With this viewpoint, bagging is interpreted as a case-weight perturbation technique, and its behavior can be explained when other arguments fail.