nils-herrmann/clean_scirepeval huggingface dataset:
Learning Analogies and Semantic Relations

We present an algorithm for learning from unlabeled text, based on the 
Vector Space Model (VSM) of information retrieval, that can solve verbal 
analogy questions of the kind found in the Scholastic Aptitude Test (SAT). 
A verbal analogy has the form A:B::C:D, meaning "A is to B as C is to D"; 
for example, mason:stone::carpenter:wood. SAT analogy questions provide 
a word pair, A:B, and the problem is to select the most analogous word 
pair, C:D, from a set of five choices. The VSM algorithm correctly
answers 47% of a collection of 374 college-level analogy questions 
(random guessing would yield 20% correct). We motivate this research by 
relating it to work in cognitive science and linguistics, and by applying 
it to a difficult problem in natural language processing, determining 
semantic relations in noun-modifier pairs. The problem is to classify a 
noun-modifier pair, such as "laser printer", according to the semantic 
relation between the noun (printer) and the modifier (laser). We use a 
supervised nearest-neighbour algorithm that assigns a class to a given 
noun-modifier pair by finding the most analogous noun-modifier pair in 
the training data. With 30 classes of semantic relations, on a collection 
of 600 labeled noun-modifier pairs, the learning algorithm attains an F 
value of 26.5% (random guessing: 3.3%). With 5 classes of semantic
relations, the F value is 43.2% (random: 20%). The performance is 
state-of-the-art for these challenging problems.
IPC-4 Probabilistic Planning Track : FAQ 1 . 01 November 1 , 2003

The 2004 International Planning Competition, IPC-4, will include a probabilistic planning track for the first time. This document provides some of the high level decisions that have been made concerning how the competition will be run. The 2004 International Planning Competition, IPC-4, will include a probabilistic planning track for the first time. This document lays out some of the high level decisions that have been made concerning how the competition will be run. The details are still in flux at the time of this writing, and we hope to get feedback from the participants to spelled them out more precisely during the fall. The overriding goal of the probabilistic planning track is to bring together two communities converging on a similar set of research issues and aid them in creating comparable tools and approaches. One community consists of Markov decision process (MDP) researchers interested in developing algorithms that apply to powerfully expressive representations of environments. The other consists of planning researchers incorporating probabilistic and decision theoretic concepts into their planning algorithms. Cross fertilization has begun, but the probabilistic planning track promises a set of shared benchmarks and evaluation metrics that could crystallize efforts in this domain of study. This document represents a snapshot of the ongoing development of the IPC-4 probabilistic track. For the latest developments, please visit: http://www.cs.rutgers.edu/ ∼mlittman/topics/
Measuring praise and criticism: Inference of semantic orientation from association

The evaluative character of a word is called its semantic orientation. Positive semantic orientation indicates praise (e.g., "honest", "intrepid") and negative semantic orientation indicates criticism (e.g., "disturbing", "superfluous"). Semantic orientation varies in both direction (positive or negative) and degree (mild to strong). An automated system for measuring semantic orientation would have application in text classification, text filtering, tracking opinions in online discussions, analysis of survey responses, and automated chat systems (chatbots). This article introduces a method for inferring the semantic orientation of a word from its statistical association with a set of positive and negative paradigm words. Two instances of this approach are evaluated, based on two different statistical measures of word association: pointwise mutual information (PMI) and latent semantic analysis (LSA). The method is experimentally tested with 3,596 words (including adjectives, adverbs, nouns, and verbs) that have been manually labeled positive (1,614 words) and negative (1,982 words). The method attains an accuracy of 82.8% on the full test set, but the accuracy rises above 95% when the algorithm is allowed to abstain from classifying mild words.
IPC-4 Probabilistic Planning Track : FAQ 0 . 5 September 13 , 2003

The 2004 International Planning Competition, IPC-4, will include a probabilistic planning track for the first time. This document provides some of the high level decisions that have been made concerning how the competition will be run. The 2004 International Planning Competition, IPC-4, will include a probabilistic planning track for the first time. This document lays out some of the high level decisions that have been made concerning how the competition will be run. The details are still in flux at the time of this writing, and we hope to get feedback from the participants to spelled them out more precisely during the fall. The overriding goal of the probabilistic planning track is to bring together two communities converging on a similar set of research issues and aid them in creating comparable tools and approaches. One community consists of Markov decision process (MDP) researchers interested in developing algorithms that apply to powerfully expressive representations of environments. The other consists of planning researchers incorporating probabilistic and decision theoretic concepts into their planning algorithms. Cross fertilization has begun, but the probabilistic planning track promises a set of shared benchmarks and evaluation metrics that could crystallize efforts in this domain of study. This document represents a snapshot of the ongoing development of the IPC-4 probabilistic track. For the latest developments, please visit: http://www.cs.rutgers.edu/ ∼mlittman/topics/
Representation and Learning in Computational Game Theory

Game theory has emerged as the key tool for understanding and designing complex multiagent environments such as the Internet, systems of autonomous agents, and electronic communities or economies. To support these relatively recent uses of game theory — as well as for more ambitious modeling in some of the traditional application areas — there is a growing need for a computational theory that is largely absent from classical game theory research. Such a computational theory needs to provide a rich and flexible collection of models and representations for complex game-theoretic problems; powerful and efficient algorithms for manipulating and learning these models; and a deep understanding of the algorithmic and resource issues arising in all aspects of game theory. The overarching goal of the proposed work is to “scale up” the applicability of game theory, in much the same way that Bayesian networks and associated advances made complex, highdimensional probabilistic modeling possible in a wide set of applications in computer science and beyond. Two of the most important topics that have materialized to date — and the primary emphases of the current proposal — are the representation and efficient manipulation of large and complex games, and new approaches to learning in game-theoretic settings. On the topic of representation, the proposal includes the development of methods to model structured interaction in large-population games; the intersection of social network theory and game theory; new representations in repeated games; and representational issues for a variety of equilibria types. On the topic of learning, it includes the development of online multiplicative update methods for large and structured games; modeling cooperation in learning; applications of game theory to the analysis of machine learning methods; learning for games that change over time; and the relationship between game theory and reinforcement learning. It also covers many interesting topics in the intersection of representational and learning issues. The work proposed here will have a number of broader impacts. It will make it possible to study interactions in social and ecological systems in drastically larger systems using the new game-theoretic representations and algorithms, allowing researchers in other scientific disciplines to apply game theory to a broader array of problems. Through highly visible symposia, workshops, and tutorials at international conferences and interdisciplinary classes on computational game theory, the new discoveries will be disseminated to interested researchers and students. In addition, new programs in game theory will be created for two established outreach programs based at Penn’s Institute for Research in Cognitive Science. One of these is a summer school for advanced undergraduates, while the other is for minority and female high school students in the Philadelphia area.
Combining independent modules in lexical multiple-choice problems

Existing statistical approaches to natural language problems are very
coarse approximations to the true complexity of language processing.
As such, no single technique will be best for all problem
instances. Many researchers are examining ensemble methods that
combine the output of multiple modules to
create more accurate solutions. This paper examines three merging
rules for combining probability distributions: the familiar mixture
rule, the logarithmic rule, and a novel product rule.
These rules were applied with state-of-the-art results to two
problems used to assess human mastery of lexical
semantics -- synonym questions and analogy questions. All three
merging rules result in ensembles that are more accurate than any of
their component modules. The differences among the three rules are not statistically
significant, but it is suggestive that the popular mixture rule
is not the best rule for either of the two problems.
AAAI-2002 Fall Symposium Series

The AAAI-2002 Fall Symposium Series was held Friday through Sunday, 15 to 17 November 2002 at the Sea Crest Conference Center in North Falmouth, Massachusetts. The five symposia in the 2002 Fall Symposia Series were (1) Chance Discovery: The Discovery and Management of Chance Events; (2) Etiquette for Human-Computer Work; (3) Human-Robot Interaction; (4) Intent Inference for Users, Teams, and Adversaries; and (5) Personalized Agents. The highlights of each symposium were presented at a special plenary session. Association for the Advancement of Artificial Intelligence (AAAI) technical reports of most of the symposia will be made available to AAAI members.
Confidence Bands for Roc Curves

We address the problem of comparing the performance of classifiers. In this paper we study techniques for generating and evaluating bands on ROC curves. Historically this has been done using one-dimensional confidence intervals by freezing one variable - false-positiverate, or threshold on the classification scoring function. Weadapt two prior methods and introduce a new radial sweepmethod to generate confidence bands. We show, throughempirical studies, that the bands are too tight and introducea general optimization methodology for creatingbands that better fit the data, as well as methods for evaluatingconfidence bands. We show empirically that theoptimized confidence bands fit much better and that, usingour new evaluation method, it is possible to gauge therelative fit of different confidence bands.
A polynomial-time nash equilibrium algorithm for repeated games

With the increasing reliance on game theory as a foundation for auctions and electronic commerce, efficient algorithms for computing equilibria in multiplayer general-sum games are of great theoretical and practical interest. The computational complexity of finding a Nash equilibrium for a one-shot bimatrix game is a well known open problem. This paper treats a closely related problem, that of finding a Nash equilibrium for an average-payoff phrepeated bimatrix game, and presents a polynomial-time algorithm. Our approach draws on the "folk theorem" from game theory and shows how finite-state equilibrium strategies can be found efficiently and expressed succinctly.
Combining Independent Modules to Solve Multiple-choice Synonym and Analogy Problems

Existing statistical approaches to natural language problems are very coarse approximations to the true complexity of language processing. As such, no single technique will be best for all problem instances. Many researchers are examining ensemble methods that combine the output of successful, separately developed modules to create more accurate solutions. This paper examines three merging rules for combining probability distributions: the well known mixture rule, the logarithmic rule, and a novel product rule. These rules were applied with state-of-the-art results to two problems commonly used to assess human mastery of lexical semantics|synonym questions and analogy questions. All three merging rules result in ensembles that are more accurate than any of their component modules. The dierences among the three rules are not statistically signicant, but it is suggestive that the popular mixture rule is not the best rule for either of the two problems.
Randomized strategic demand reduction: getting more by asking for less

Auctions are an area of great academic and commercial interest; however, many important auctions are extremely complicated, involving simultaneous auction of hundreds of non-interchangeable goods with value dependencies. This makes it difficult or impossible for humans to grasp all of the nuances of their strategic decisions. Computer aid could ease the burden of efficiently competing in these auctions; indeed, very simple autonomous agents have started to appear as bidders in some auctions (e.g. on eBay). Although agents that function in more complex experimental scenarios have been deployed [5], studies of autonomous bidding agents and their interactions are relatively few and recent. Our work explores agent bidding strategies in a large-scale and realistic auction scenario, namely the FCC Spectrum Auction Simulator, orFAucS . That paper introducedknapsackagents that optimized the set of goods they bid on given a budget constraint, but without taking into account the needs and strategies of other agents. We extend this work by examining the possibilities of agent cooperation and uncertain knowledge. In this paper, we present punishing randomized strategic demand reduction (PRSDR), a strategy by which cooperative agents can significantly outperform the Knapsack agents despite having highly uncertain knowledge regarding each others’ goals and without any explicit inter-agent communication. The strategy is self-enforcing in that agents cannot benefit by defecting back to the Knapsack strategy, and serves as an example of a general strategy for efficient bidding in many realistic simultaneous multiple-round (SMR) auctions.
Unsupervised Learning of Semantic Orientation from a Hundred-Billion-Word Corpus

The evaluative character of a word is called its semantic orientation. A positive semantic orientation implies desirability (e.g., "honest", "intrepid") and a negative semantic orientation implies undesirability (e.g., "disturbing", "superfluous"). This paper introduces a simple algorithm for unsupervised learning of semantic orientation from extremely large corpora. The method involves issuing queries to a Web search engine and using pointwise mutual information to analyse the results. The algorithm is empirically evaluated using a training corpus of approximately one hundred billion words  the subset of the Web that is indexed by the chosen search engine. Tested with 3,596 words (1,614 positive and 1,982 negative), the algorithm attains an accuracy of 80%. The 3,596 test words include adjectives, adverbs, nouns, and verbs. The accuracy is comparable with the results achieved by Hatzivassiloglou and McKeown (1997), using a complex four-stage supervised learning algorithm that is restricted to determining the semantic orientation of adjectives.
Least-Squares Methods in Reinforcement Learning for Control

Least-squares methods have been successfully used for prediction problems in the context of reinforcement learning, but little has been done in extending these methods to control problems. This paper presents an overview of our research efforts in using least-squares techniques for control. In our early attempts, we considered a direct extension of the Least-Squares Temporal Difference (LSTD) algorithm in the spirit of Q-learning. Later, an effort to remedy some limitations of this algorithm (approximation bias, poor sample utilization) led to the Least-Squares Policy Iteration (LSPI) algorithm, which is a form of model-free approximate policy iteration and makes efficient use of training samples collected in any arbitrary manner. The algorithms are demonstrated on a variety of learning domains, including algorithm selection, inverted pendulum balancing, bicycle balancing and riding, multiagent learning in factored domains, and, recently, on two-player zero-sum Markov games and the game of Tetris.
Self-Enforcing Strategic Demand Reduction

Auctions are an area of great academic and commercial interest, from tiny auctions for toys on eBay to multi-billion-dollar auctions held by governments for resources or contracts. Although there has been significant research on auction theory, especially from the perspective of auction mechanisms, studies of autonomous bidding agents and their interactions are relatively few and recent. This paper examines several autonomous agent bidding strategies in the context of FAucS, a faithful simulation of a complex FCC spectrum auction. We introduce punishing randomized strategic demand reduction (PRSDR), a novel bidding strategy by which bidders can partition available goods in a mutually beneficial way without explicit inter-agent communication. When all use PRSDR, bidders obtain significantly better results than when using a reasonable baseline approach. The strategy automatically detects and punishes non-cooperating bidders to achieve robustness in the face of agent defection, and performs well under alternative conditions. The PRSDR strategy is fully implemented and we present detailed empirical results.
Implicit Negotiation in Repeated Games

In business-related interactions such as the on-going high-stakes FCC spectrum auctions, explicit communication among participants is regarded as collusion, and is therefore illegal. In this paper, we consider the possibility of autonomous agents engaging in implicit negotiation via their tacit interactions. In repeated general-sum games, our testbed for studying this type of interaction, an agent using a "best response" strategy maximizes its own payoff assuming its behavior has no effect on its opponent. This notion of best response requires some degree of learning to determine the fixed opponent behavior. Against an unchanging opponent, the best-response agent performs optimally, and can be thought of as a "follower," since it adapts to its opponent. However, pairing two best-response agents in a repeated game can result in suboptimal behavior. We demonstrate this suboptimality in several different games using variants of Q-learning as an example of a best-response strategy.We then examine two "leader" strategies that induce better performance from opponent followers via stubbornness and threats. These tactics are forms of implicit negotiation in that they aim to achieve a mutually beneficial outcome without using explicit communication outside of the game.
Selecting the Right Algorithm

Computer scientists always strive to find better and faster algorithms for any computational problem. It is usually true that programmers and/or users come across a plethora of different algorithms when looking to solve a particular problem efficiently. Each one of these algorithms might offer different guarantees and properties, but it is unlikely that a single one of them is the best (fastest) in all possible cases. So, the question that the programmer/user typically faces is: “Which algorithm should I select?” This question is largely due to the uncertainty in the input space, the inner workings of the algorithm (especially true for randomized algorithms), and the hardware characteristics. It’s hard to know in advance what kind of inputs will be provided, how exactly the computation will proceed, or even how efficiently the underlying hardware will support the needs of the different algorithms. Sometimes, a careful study can reveal that committing to a particular algorithm is better than committing to any of the other algorithms, but is this the best we can do? What if uncertainty is explicitly taken into account and the right decision is made dynamically on an instance-by-instance basis? To make the discussion more concrete, consider, for example, the problem of sorting. Why would you ever choose MergeSort or InsertionSort when you know that QuickSort is in general the fastest algorithm for sorting? That might be true to some extent, but it seems that if you allow for collaboration of these three “competitors”, the outcome can be beneficial. Think of this algorithm selection problem(Rice 1976) as a decision problem: “Which algorithm should I run whenever a new instance is presented?”The fact that two of these algorithms are recursive makes the problem even more interesting. Every time a recursive call is made, you can ask the same question: “Which algorithm should I choose for the current subproblem?”. Nothing really dictates that you have to use the same algorithm again and again throughout the recursion. How, then, can you optimize this sequence (or better, tree) of decisions? On what ground is each decision based?
Approximate Dimension Reduction at NTCIR

We carried out a comparison of cross-language retrieval methods on the NTCIR-1 data based on dimension reduction (latent semantic indexing). These methods all use a collection parallel documents (translations or approximate translations) and very little, if any, linguistic knowledge. In NTCIR-1, we compared latent semantic indexing, local LSI, and approximate dimensional equalization (ADE). We found that local LSI and ADE performed the best on this collection and were comparable to the best performing systems reported elsewhere. We also ran ADE on the NTCIR-2 and found it fared considerably less well.
PAC Generalization Bounds for Co-training

The rule-based bootstrapping introduced by Yarowsky, and its co-training variant by Blum and Mitchell, have met with considerable empirical success. Earlier work on the theory of co-training has been only loosely related to empirically useful co-training algorithms. Here we give a new PAC-style bound on generalization error which justifies both the use of confidences — partial rules and partial labeling of the unlabeled data — and the use of an agreement-based objective function as suggested by Collins and Singer. Our bounds apply to the multiclass case, i.e., where instances are to be assigned one of labels for k ≥ 2.
Review: Computer Language Games

In language games, word meanings play a central role. This means that closed world assumptions that are so powerful in many games cannot be depended on. This paper describes implemented systems that play the language games of American-style crosswords, cryptic crosswords, Trivial PursuitTM, and punning riddles. These early efforts illustrate the challenge of language games, demonstrate some emerging ideas for tackling language games by computer, and indicate fruitful areas for future research.