nils-herrmann/clean_scirepeval huggingface dataset:
1 Lifted First-Order Probabilistic Inference

Most probabilistic inference algorithms are specified and processed on a proposi-tional level, even though many domains are better represented by first-order specifications that compactly stand for a class of propositional instantiations. In the last fifteen years, many algorithms accepting first-order specifications have been proposed. However, these algorithms still perform inference on a mostly propositional model, generated by the instantiation of first-order constructs. When this is done, the rich and useful first-order structure is not explicit anymore. This first-order representation and structure allow us to perform lifted inference, that is, inference on the first-order representation directly, manipulating not only individuals but also groups of individuals. This has the potential of greatly speeding up inference. We precisely define the problem and present an algorithm that generalizes variable elimination and manipulates first-order representations in order to perform lifted inference. Probabilistic inference algorithms are widely employed in artificial intelligence. Among those, graphical models such as Bayesian and Markov networks (BNs and MNs respectively) (?)) are among the most popular. These models are specified by a set of conditional probabilities (for BNs) or factors, also called potential functions (for MNs). Both conditional probabilities and factors are defined over particular subsets of the available random variables, and map assignments of those random variables to positive real numbers (called potentials in MNs). For our purposes, it will be helpful to think of graphical models in general and simply consider conditional probabilities as a type of factor. For example, in an application for document subject classification, one can specify a dependence between the random variables subject apple, word mac (which
Inference with classifiers: a study of structured output problems in natural language processing

A large number of problems in natural language processing (NLP) involve outputs with complex structure. Conceptually in such problems, the task is to assign values to multiple variables which represent the outputs of several interdependent components. A natural approach to this task is to formulate it as a two-stage process. In the first stage, the variables are assigned initial values using machine learning based programs. In the second, an inference procedure uses the outcomes of the first stage classifiers along with domain specific constraints in order to infer a globally consistent final prediction. 
This dissertation introduces a framework, inference with classifiers, to study such problems. The framework is applied to two important and fundamental NLP problems that involve complex structured outputs, shallow parsing and semantic role labeling. In shallow parsing, the goal is to identify syntactic phrases in sentences, which has been found useful in a variety of large-scale NLP applications. Semantic role labeling is the task of identifying predicate-argument structure in sentences, a crucial step toward a deeper understanding of natural language. In both tasks, we develop state-of-the-art systems which have been used in practice. 
In this framework, we have shown the significance of incorporating constraints into the inference stage as a way to correct and improve the decisions of the stand alone classifiers. Although it is clear that incorporating constraints into inference necessarily improves global coherency, there is no guarantee of the improvement in the performance measured in terms of the accuracy of the local predictions---the metric that is of interest for most applications. We develop a better theoretic understanding of this issue. Under a reasonable assumption, we prove a sufficient condition to guarantee that using constraints cannot degrade the performance with respect to Hamming loss. In addition, we provide an experimental study suggesting that constraints can improve performance even when the sufficient conditions are not fully satisfied.
A study of the bipartite ranking problem in machine learning

The problem of ranking, in which the goal is to learn a real-valued ranking function that induces a ranking or ordering over an instance space, has recently gained attention in machine learning. A particular setting of interest is the bipartite ranking problem, in which instances come from two categories, positive and negative; the learner is given examples of instances labeled as positive or negative, and the goal is to learn from these examples a ranking function that ranks future positive instances higher than negative ones. This thesis makes four important contributions to the understanding of the bipartite ranking problem. 
First, we derive large deviation and uniform convergence bounds for the bipartite ranking error, a quantity used to measure the quality of a bipartite ranking function. These bounds serve to bound the expected error of a ranking function in terms of its empirical error on a finite sample. The uniform convergence bound is expressed in terms of a new set of combinatorial parameters that we term the bipartite rank-shatter coefficients. 
Second, we define a model of learnability for bipartite ranking functions, and derive both upper and lower bounds on the sample complexity of learning ranking functions in this model. The upper bound is expressed in terms of the bipartite rank-shatter coefficients. The lower bound is expressed in terms of another new combinatorial parameter that we term the rank dimension . 
Third, we derive generalization bounds for bipartite ranking algorithms based on the notion of algorithmic stability. Unlike bounds based on uniform convergence, these bounds can be applied also to algorithms that search function classes of unbounded complexity. In particular, we are able to apply our results to obtain generalization bounds for kernel-based ranking algorithms, to which bounds based on uniform convergence are often not applicable. 
Finally, we demonstrate a practical application of bipartite ranking to a problem in bioinformatics, namely the problem of identifying genes related to a given disease based on microarray data. Our studies on leukemia and colon cancer data sets show very promising results, including the identification of some exciting candidate genes as potential targets for drug development.
Emotions from Text: Machine Learning for Text-based Emotion Prediction

In addition to information, text contains attitudinal, and more specifically, emotional content. This paper explores the text-based emotion prediction problem empirically, using supervised machine learning with the SNoW learning architecture. The goal is to classify the emotional affinity of sentences in the narrative domain of children's fairy tales, for subsequent usage in appropriate expressive rendering of text-to-speech synthesis. Initial experiments on a preliminary data set of 22 fairy tales show encouraging results over a naive baseline and BOW approach for classification of emotional versus non-emotional contents, with some dependency on parameter tuning. We also discuss results for a tripartite model which covers emotional valence, as well as feature set alternations. In addition, we present plans for a more cognitively sound sequential model, taking into consideration a larger set of basic emotions.
Toward concept-based text understanding and mining

There is a huge amount of text information in the world, written in natural languages. Understanding and effectively utilizing the text requires the ability to disambiguate text fragments at several levels, syntactically and semantically, abstracting away details and using background knowledge in a variety of ways. One promising direction of understanding text in the real semantical sense, which is a natural ability of human beings, and that of supporting intelligent access to textual information is to implement concept-based text understanding and mining. That is, a mechanism of organizing, indexing, accessing textual information and discovering knowledge, centered around real-world concepts and entities. Unfortunately, due to the difficulty caused by language ambiguity, most current text-related techniques still directly deal with syntactic fragments and individual mentions of concepts, without considering a concept as a whole. 
A critical problem with these techniques is the lack of the capability to resolve the concept ambiguity in text. A given entity---representing a person, a location or an organization---may be mentioned in text in multiple, ambiguous ways. Supporting concept-based natural language understanding requires resolving conceptual ambiguity through Entity Reference Identification , and in particular, identifying entities from their mentions, within and across documents, and mapping mentions to them, with the hope to discover and organize information around identified entities. 
This thesis systematically studies this fundamental problem towards concept-based text understanding and mining. We develop several machine learning techniques to address different aspects of it, including (1) a discriminative approach of learning similar metrics to capture the appearance similarity between names; (2) a new supervised discriminative clustering framework, that can partition a set of names through some global optimization and incorporate learning into clustering, guided by supervision; and (3) a generative probabilistic model, at the heart of which is a view on how documents are generated and how names (of different entity types) are "sprinkled" into them. We show that all of these approaches perform very accurately, in the range of 90%--95% F1 measure for different entity types, better than baselines and previous approaches to (some aspects of) this problem. Our work also exhibits that, as more domain-specific knowledge is discovered and incorporated into the entity identification, the learning techniques developed accordingly, can achieve better performance. 
In addition to entity identification, we also extend the generative probabilistic model to address a significant application that are related to concept-based text understanding and mining---semantic integration between text and databases, based on entity identification and tracking.
Context Sensitive Paraphrasing

Given a sentence and a specic word or phrase in that sentence, is it possible to replace that word or phrase with another and have the sentence keep the same meaning? Most paraphrasing work develops patterns and templates that can replace other patterns or templates in some context. But what about a specic given context, how do we know if a paraphrase rule applies? Ideally if one knew the correct sense to every word in the sentence, then one would also know what other words or phrases can replace a given word and have the sentence keep its same or entailed meaning. As it is there is no reliable way to tag the sense of every word, and with current sense repositories it is difcult (and perhaps impossible) to determine what senses can replace another for every word and every domain. We will present a new task and formalism that rests somewhere between these two approaches. Our system will allow us to say if A can replace B in a context sensitive way, even on unseen A and B.
Discriminative Training of Clustering Functions: Theory and Experiments with Entity Identification

Clustering is an optimization procedure that partitions a set of elements to optimize some criteria, based on a fixed distance metric defined between the elements. Clustering approaches have been widely applied in natural language processing and it has been shown repeatedly that their success depends on defining a good distance metric, one that is appropriate for the task and the clustering algorithm used. This paper develops a framework in which clustering is viewed as a learning task, and proposes a way to train a distance metric that is appropriate for the chosen clustering algorithm in the context of the given task. Experiments in the context of the entity identification problem exhibit significant performance improvements over state-of-the-art clustering approaches developed for this problem.
Token-level Disambiguation of VerbNet classes

The automatic disambiguation of verbs in domain independent text becomes more and more important for applications such as Machine Translation, Text Summarization, and Question Answering, mainly because verbs play a key factor in the syntactic and semantic interpretation of sentences. In this paper we present a system for the automatic classification of token verbs in context based on VerbNet classes. A supervised machine learning classifier is trained and tested on a portion of PropBank using a set of lexical and syntactic features.
Demonstrating an Interactive Semantic Role Labeling System

Semantic Role Labeling (SRL) is the task of performing a shallow semantic analysis of text (i.e., Who did What to Whom, When, Where, How). This is a crucial step toward deeper understanding of text and has many immediate applications. Preprocessed information on text, mostly syntactic, has been shown to be important for SRL. Current research focuses on improving the performance assuming that this lower level information is given without any attention to the overall efficiency of the final system, although minimizing execution time is a necessity in order to support real world applications. The goal of our demonstration is to present an interactive SRL system that can be used both as a research and an educational tool. Its architecture is based on the state-of-the-art system (the top system in the 2005 CoNLL shared task), modified to process raw text through the addition of lower level processors, while achieving effective real time performance.
Lifted First-Order Probabilistic Inference

Most probabilistic inference algorithms are specified and processed on a propositional level. In the last decade, many proposals for algorithms accepting first-order specifications have been presented, but in the inference stage they still operate on a mostly propositional representation level.  2003] presented a method to perform inference directly on the first-order level, but this method is limited to special cases. In this paperwe present the first exact inference algorithm that operates directly on a first-order level, and that can be applied to any first-order model (specified in a language that generalizes undirected graphical models). Our experiments show superior performance in comparison with propositional exact inference.
Learnability of Bipartite Ranking Functions

The problem of ranking, in which the goal is to learn a real-valued ranking function that induces a ranking or ordering over an instance space, has recently gained attention in machine learning. We define a model of learnability for ranking functions in a particular setting of the ranking problem known as the bipartite ranking problem, and derive a number of results in this model. Our first main result provides a sufficient condition for the learnability of a class of ranking functions ${\mathcal F}$: we show that ${\mathcal F}$ is learnable if its bipartite rank-shatter coefficients, which measure the richness of a ranking function class in the same way as do the standard VC-dimension related shatter coefficients (growth function) for classes of classification functions, do not grow too quickly. Our second main result gives a necessary condition for learnability: we define a new combinatorial parameter for a class of ranking functions ${\mathcal F}$ that we term the rank dimension of ${\mathcal F}$, and show that ${\mathcal F}$ is learnable only if its rank dimension is finite. Finally, we investigate questions of the computational complexity of learning ranking functions.
Learning question classifiers: the role of semantic information

To respond correctly to a free form factual question given a large collection of text data, one needs to understand the question to a level that allows determining some of the constraints the question imposes on a possible answer. These constraints may include a semantic classification of the sought after answer and may even suggest using different strategies when looking for and verifying a candidate answer. This work presents a machine learning approach to question classification. Guided by a layered semantic hierarchy of answer types, we develop a hierarchical classifier that classifies questions into fine-grained classes. This work also performs a systematic study of the use of semantic information sources in natural language classification tasks. It is shown that, in the context of question classification, augmenting the input of the classifier with appropriate semantic category information results in significant improvements to classification accuracy. We show accurate results on a large collection of free-form questions used in TREC 10 and 11.
Robust Reading: Identification and Tracing of Ambiguous Names

A given entity, representing a person, a location or an organization, may be mentioned in text in multiple, ambiguous ways. Understanding natural language requires identifying whether different mentions of a name, within and across documents, represent the same entity. We develop an unsupervised learning approach that is shown to resolve accurately the name identification and tracing problem. At the heart of our approach is a generative model of how documents are generated and how names are sprinkled into them. In its most general form, our model assumes: (1) a joint distribution over entities, (2) an author model, that assumes that at least one mention of an entity in a document is easily identifiable, and then generates other mentions via (3) an appearance model, governing how mentions are transformed from the representative mention. We show how to estimate the model and do inference with it and how this resolves several aspects of the problem from the perspective of applications such as questions answering.
Identification and Tracing of Ambiguous Names: Discriminative and Generative Approaches

A given entity - representing a person, a location or an organization - may be mentioned in text in multiple, ambiguous ways. Understanding natural language requires identifying whether different mentions of a name, within and across documents, represent the same entity. 
 
We present two machine learning approaches to this problem, which we call the "Robust Reading" problem. Our first approach is a discriminative approach, trained in a supervised way. Our second approach is a generative model, at the heart of which is a view on how documents are generated and how names (of different entity types) are "sprinkled" into them. In its most general form, our model assumes: (1) a joint distribution over entities (e.g., a document that mentions "President Kennedy" is more likely to mention "Oswald" or "White House" than "Roger Clemens"), (2) an "author" model, that assumes that at least one mention of an entity in a document is easily identifiable, and then generates other mentions via (3) an appearance model, governing how mentions are transformed from tile "representative" mention. 
 
We show that both approaches perform very accurately, in the range of 90% - 95% F1 measure for different entity types, much better than previous approaches to (some aspects of) this problem. Our extensive experiments exhibit the contribution of relational and structural features and, somewhat surprisingly, that the assumptions made within our generative model are strong enough to yield a very powerful approach, that performs better than a supervised approach with limited supervised information.
A Unified Framework to Integrate Supervision and Metric Learning into Clustering

In this paper, we propose a unified framework for applying supervision to discriminative clustering, which: (1) explicitly formalizes the problem of training a partition function as a supervised metric-learning process; (2) learns a partition function that can optimize a supervised clustering error; and (3) flexibly learns a distance metric with regard to any clustering algorithm. Moreover, we develop a general gradient-descent learning algorithm that trains a distance metric under this framework. The convergence of this algorithm is guaranteed for some restricted cases. Our experimental study shows the significant performance improvement after integrating supervision and distance metric learning in clustering, trained in our new framework.
Inference & Learning with Linear Constraints

We present a discriminatory learning framework for the prob lem of assigning globally optimal values to a set of variables with complex an d expressive dependencies among them. The problem is modeled as an integer l inea program (ILP) where the cost values associated with the variables ar represented and trained as linear classifiers. The framework unifies and exte nds several existing discriminatory approaches; most importantly, it su pports more complex dependencies among variables than existing ones. This pres entation concentrates on the benefits of the additional expressivity and on c omparing different training paradigms – with and without global feedback – in th e context of semantic role labeling.
Bimodal HCI-related affect recognition

Perhaps the most fundamental application of affective computing will be Human-Computer Interaction (HCI) in which the computer should have the ability to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90% for our bimodal fusion.
Iterative Labeling for Semi-Supervised Learning

We propose a unified perspective of a large family of semi-supervised learning algorithms, which select and label unlabeled data in an iterative process. We discuss existing approaches that label examples based on the confidence of the current hypothesis, and propose an alternative approach that labels examples based on empirical risk. This new approach is shown to be statistically reasonable, allows for worst-case performance guarantees and, as we show, significantly outperforms confidence-based approaches in experiments.