OpenRaiser/Intern-Atlas huggingface dataset:
An Unsupervised Approach to Anomaly Detection in Music Datasets
Authors: Yen-Cheng Lu, Chih-Wei Wu, Chang-Tien Lu, Alexander Lerch
Venue: SIGIR
Year: 2016

This paper presents an unsupervised method for systematically identifying anomalies in music datasets. The model integrates categorical regression and robust estimation techniques to infer anomalous scores in music clips. When applied to a music genre recognition dataset, the new method is able to detect corrupted, distorted, or mislabeled audio samples based on commonly used features in music information retrieval. The evaluation results show that the algorithm outperforms other anomaly detection methods and is capable of finding problematic samples identified by human experts. The proposed method introduces a preliminary framework for anomaly detection in music data that can serve as a useful tool to improve data integrity in the future.
Exploiting CPU SIMD Extensions to Speed-up Document Scoring with Tree Ensembles
Authors: Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, Rossano Venturini
Venue: SIGIR
Year: 2016

Scoring documents with learning-to-rank (LtR) models based on large ensembles of regression trees is currently deemed one of the best solutions to effectively rank query results to be returned by large scale Information Retrieval systems. This paper investigates the opportunities given by SIMD capabilities of modern CPUs to the end of efficiently evaluating regression trees ensembles. We propose V-QuickScorer (vQS), which exploits SIMD extensions to vectorize the document scoring, i.e., to perform the ensemble traversal by evaluating multiple documents simultaneously. We provide a comprehensive evaluation of vQS against the state of the art on three publicly available datasets. Experiments show that vQS provides speed-ups up to a factor of 3.2x.
Modeling User Feedback in Dynamic Search and Browsing
Authors: Jiyun Luo
Venue: SIGIR
Year: 2016

Nowadays searching for complicated information needs becomes more and more common. These complicated needs usually require the users to reform different queries and conduct multiple retrievals in a search session. There are a lot of technologies are developed to help session searches. Riccho, pseudo relevance feedback, and etc. can help finding relevant documents. xQuAD, RxQuAD, and etc. can help the user to explore. However none of these approaches alone works well in session searches, because they don't treat a search session as a whole. They can't answer questions like when to explore and when to exploit. In this work, we model session searches as Partially Observable Markov Decision Processes (POMDP). We model user's implicit feedbacks, such as query reformulation and user clickthrough data into the POMDP framework. Further we extend the forms of user feedbacks. We implement a new search interface which allows us to capture more explicit feedbacks from users, such as passage level relevance judgments, irrelevant judgments, duplicate judgment, and etc. We propose algorithms to effectively model these feedback signals into the POMDP framework and improve session search performance. Our algorithm is able to automatically balance users' needs of exploration and exploitation.
Principles for the Design of Online A/B Metrics
Authors: Widad Machmouchi, Georg Buscher
Venue: SIGIR
Year: 2016

In this paper, we describe principles for designing metrics in the context of A/B experiments. We share some issues that comes up in designing such experiments and provide solutions to avoid such pitfalls.
Search-based Evaluation from Truth Transcripts for Voice Search Applications
Authors: François Mairesse, Paul Raccuglia, Shiv Vitaladevuni
Venue: SIGIR
Year: 2016

Voice search applications are typically evaluated by comparing the predicted query to a reference human transcript, regardless of the search results returned by the query. While we find that an exact transcript match is highly indicative of user satisfaction, a transcript which does not match the reference still produces satisfactory search results a significant fraction of the time. This paper therefore proposes an evaluation method that compares the search results of the speech recognition hypotheses with the search results produced by a human transcript. Compared with a strict sentence match, a human evaluation shows that search result overlap is a better predictor of (a) user satisfaction and (b) search result click-through. Finally, we propose a model predicting the Expected Search Satisfaction Rate (ESSR), conditioned on search overlap outcomes. On a held out set of 1036 voice search queries, our model predicted an ESSR within 0.9% (relative) of the ground truth satisfaction averaged over 3 human judges.
On Information-Theoretic Document-Person Associations for Expert Search in Academia
Authors: Vítor Mangaravite, Rodrygo L. T. Santos
Venue: SIGIR
Year: 2016

State-of-the-art expert search approaches rely on document-person associations to infer the expertise of a candidate person for a given query. Such associations have traditionally been modeled as boolean variables, indicating whether or not a candidate authored a document, and further normalized to penalize prolific authorships. In this paper, we address expert search in academia, where the authorship of a document can be determined with reasonable certainty. In contrast to traditional approaches, we propose to model associations as non-boolean variables, reflecting the probability that a document is informative of the expertise of a candidate. Moreover, we introduce an alternative normalization scheme that measures how discriminative a particular document-person association is in light of all associations involving either the document or the person. Through a large-scale user study with academic experts from several areas of knowledge, we demonstrate the suitability of the proposed association and normalization schemes to improve the effectiveness of a state-of-the-art expert search approach.
The LExR Collection for Expertise Retrieval in Academia
Authors: Vítor Mangaravite, Rodrygo L. T. Santos, Isac S. Ribeiro, Marcos André Gonçalves, Alberto H. F. Laender
Venue: SIGIR
Year: 2016

Expertise retrieval has been the subject of intense research over the past decade, particularly with the public availability of benchmark test collections for expertise retrieval in enterprises. Another domain which has seen comparatively less research on expertise retrieval is academic search. In this paper, we describe the Lattes Expertise Retrieval (LExR) test collection for research on academic expertise retrieval. LExR has been designed to provide a large-scale benchmark for two complementary expertise retrieval tasks, namely, expert profiling and expert finding. Unlike currently available test collections, which fully support only one of these tasks, LExR provides graded relevance judgments performed by expert judges separately for each task. In addition, LExR is both cross-organization and cross-area, encompassing candidate experts from all areas of knowledge working in research institutions all over Brazil. As a result, it constitutes a valuable resource for fostering new research directions on expertise retrieval in an academic setting.
Understanding Human Language: Can NLP and Deep Learning Help?
Authors: Christopher D. Manning
Venue: SIGIR
Year: 2016

There is a lot of overlap between the core problems of information retrieval (IR) and natural language processing (NLP). An IR system gains from understanding a user need and from understanding documents, and hence being able to determine whether a document has information that satisfies the user need. Much of NLP is about the same thing: Natural language understanding aims to understand the meaning of questions and documents and meaning relationships. The exciting recent application of deep learning approaches in NLP has brought new tools for effectively understanding language semantics. In principle, there should be a lot of synergy, though in practice the concerns of IR on large systems and macro-scale understanding have tended to contrast with the emphasis in NLP on language structure and micro-scale understanding. My talk will emphasize the two topics of how NLP can contribute to understanding textual relationships and how deep learning approaches substantially aid in this goal. One basic -- and very successful tool -- has been the new generation of distributed word representations: neural word embeddings. However, beyond just word meanings, we need to understand how to compose the meanings of larger pieces of text. Two requirements for that are good ways to understand the structure of human language utterances and ways to compose their meanings. Deep learning methods can help for both tasks. Finally, we need to understand relationships between pieces of text, to be able to do tasks such as Natural Language Inference (or Recognizing Textual Entailment) and Question Answering, and I will look at some of our recent work in these areas, both with and without the help of neural networks
Ordinal Text Quantification
Authors: Giovanni Da San Martino, Wei Gao, Fabrizio Sebastiani
Venue: SIGIR
Year: 2016

In recent years there has been a growing interest in text quantification, a supervised learning task where the goal is to accurately estimate, in an unlabelled set of items, the prevalence (or "relative frequency") of each class c in a predefined set C. Text quantification has several applications, and is a dominant concern in fields such as market research, the social sciences, political science, and epidemiology. In this paper we tackle, for the first time, the problem of ordinal text quantification, defined as the task of performing text quantification when a total order is defined on the set of classes; estimating the prevalence of "five stars" reviews in a set of reviews of a given product, and monitoring this prevalence across time, is an example application. We present OQT, a novel tree-based OQ algorithm, and discuss experimental results obtained on a dataset of tweets classified according to sentiment strength.
How Informative is a Term?: Dispersion as a measure of Term Specificity
Authors: Rodney McDonell, Justin Zobel, Bodo Billerbeck
Venue: SIGIR
Year: 2016

Similarity functions assign scores to documents in response to queries. These functions require as input statistics about the terms in the queries and documents, where the intention is that the statistics are estimates of the relative informativeness of the terms. Common measures of informativeness use the number of documents containing each term (the document frequency) as a key measure. We argue in this paper that the distribution of within-document frequencies across a collection is also pertinent to informativeness, a measure that has not been considered in prior work: the most informative words tend to be those whose frequency of occurrence has high variance. We propose use of relative standard deviation (RSD) as a measure of variability incorporating within-document frequencies, and show that RSD compares favourably with inverse document frequency (IDF), in both in-principle analysis and in practice in retrieval, with small but consistent gains.
InfoScout: An Interactive, Entity Centric, Person Search Tool
Authors: Sean McKeown, Martynas Buivys, Leif Azzopardi
Venue: SIGIR
Year: 2016

Individuals living in highly networked societies publish a large amount of personal, and potentially sensitive, information online. Web investigators can exploit such information for a variety of purposes, such as in background vetting and fraud detection. However, such investigations require a large number of expensive man hours and human eﬀort. This paper describes InfoScout, a search tool which is intended to reduce the time it takes to identify and gather subject centric information on the Web. InfoScout collects relevance feedback information from the investigator in order to rerank search results, allowing the intended information to be discovered more quickly. Users may still direct their search as they see ﬁt, issuing ad-hoc queries and ﬁltering existing results by keywords. Design choices are informed by prior work and industry collaboration.
PULP: A System for Exploratory Search of Scientific Literature
Authors: Alan Medlar, Kalle Ilves, Ping Wang, Wray L. Buntine, Dorota Glowacka
Venue: SIGIR
Year: 2016

Despite the growing importance of exploratory search, information retrieval (IR) systems tend to focus on lookup search. Lookup searches are well served by optimising the precision and recall of search results, however, for exploratory search this may be counterproductive if users are unable to formulate an appropriate search query. We present a system called PULP that supports exploratory search for scientific literature, though the system can be easily adapted to other types of literature. PULP uses reinforcement learning (RL) to avert the user from context traps resulting from poorly chosen search queries, trading off between exploration (presenting the user with diverse topics) and exploitation (moving towards more specific topics). Where other RL-based systems suffer from the "cold start" problem, requiring sufficient time to adjust to a user's information needs, PULP initially presents the user with an overview of the dataset using temporal topic models. Topic models are displayed in an interactive alluvial diagram, where topics are shown as ribbons that change thickness with a given topics relative prevalence over time. Interactive, exploratory search sessions can be initiated by selecting topics as a starting point.
Uncovering Task Based Behavioral Heterogeneities in Online Search Behavior
Authors: Rishabh Mehrotra, Prasanta Bhattacharya, Emine Yilmaz
Venue: SIGIR
Year: 2016

While a major share of prior work have considered search sessions as the focal unit of analysis for seeking behavioral insights, search tasks are emerging as a competing perspective in this space. In the current work, we quantify user search task behavior for both single- as well as multi-task search sessions and relate it to tasks and topics. Specifically, we analyze user-disposition, topic and user-interest level heterogeneities that are prevalent in search task behavior. Our results show that while search multi-tasking is a common phenomenon among the search engine users, the extent and choice of multi-tasking topics vary significantly across users. We find that not only do users have varying propensities to multi-task, they also search for distinct topics across single-task and multi-task sessions. To our knowledge, this is among the first studies to fully characterize online search tasks with a focus on user- and topic-level differences that are observable from search sessions.
Going back in Time: An Investigation of Social Media Re-finding
Authors: Florian Meier, David Elsweiler
Venue: SIGIR
Year: 2016

Social Media (SM) has become a valuable information source to many in diverse situations. In IR, research has focused on real-time aspects and as such little is known about how long SM content is of value to users, if and how often it is re-accessed, the strategies people employ to re-access and if difficulties are experienced while doing so. We present results from a 5 month-long naturalistic, log-based study of user interaction with Twitter, which suggest re-finding to be a regular activity and that Tweets can offer utility for longer than one might think. We shed light on re-finding strategies revealing that remembered people are used as a stepping stone to Tweets rather than searching for content directly. Bookmarking strategies reported in the literature are used infrequently as a means to re-access. Finally, we show that by using statistical modelling it is possible to predict if a Tweet has future utility and is likely to be re-found. Our findings have implications for the design of social media search systems and interfaces, in particular for Twitter, to better support users re-find previously seen content.
Extracting Information Seeking Intentions for Web Search Sessions
Authors: Matthew Mitsui, Chirag Shah, Nicholas J. Belkin
Venue: SIGIR
Year: 2016

We present a method for extracting the self-reported intentions of users engaged in an information seeking episode. We recruited participants to conduct search sessions and subsequently asked them to self-report their intentions. A total of 27 users participated in a lab study, during which they worked on two search tasks. After each search session, participants indicated their intentions during that session while viewing a video replay. Results indicate that the set of search intentions provided to participants was sufficient to account for intentions in four journalism-related information seeking tasks: a copy editing task, interview preparation task, relationships task, and story pitch task. The results also suggest regular patterns in intentions that can be exploited for identification of task type as well as potential applications to personalization and recommendation during a search episode.
Axiomatic Analysis for Improving the Log-Logistic Feedback Model
Authors: Ali Montazeralghaem, Hamed Zamani, Azadeh Shakery
Venue: SIGIR
Year: 2016

Pseudo-relevance feedback (PRF) has been proven to be an effective query expansion strategy to improve retrieval performance. Several PRF methods have so far been proposed for many retrieval models. Recent theoretical studies of PRF methods show that most of the PRF methods do not satisfy all necessary constraints. Among all, the log-logistic model has been shown to be an effective method that satisfies most of the PRF constraints. In this paper, we first introduce two new PRF constraints. We further analyze the log-logistic feedback model and show that it does not satisfy these two constraints as well as the previously proposed "relevance effect" constraint. We then modify the log-logistic formulation to satisfy all these constraints. Experiments on three TREC newswire and web collections demonstrate that the proposed modification significantly outperforms the original log-logistic model, in all collections.
Distributional Random Oversampling for Imbalanced Text Classification
Authors: Alejandro Moreo, Andrea Esuli, Fabrizio Sebastiani
Venue: SIGIR
Year: 2016

The accuracy of many classification algorithms is known to suffer when the data are imbalanced (i.e., when the distribution of the examples across the classes is severely skewed). Many applications of binary text classification are of this type, with the positive examples of the class of interest far outnumbered by the negative examples. Oversampling (i.e., generating synthetic training examples of the minority class) is an often used strategy to counter this problem. We present a new oversampling method specifically designed for classifying data (such as text) for which the distributional hypothesis holds, according to which the meaning of a feature is somehow determined by its distribution in large corpora of data. Our Distributional Random Oversampling method generates new random minority-class synthetic documents by exploiting the distributional properties of the terms in the collection. We discuss results we have obtained on the Reuters-21578, OHSUMED-S, and RCV1-v2 datasets.
SIGIR 2016 Workshop WebQA II: Web Question Answering Beyond Factoids
Authors: Alessandro Moschitti, Lluís Màrquez, Preslav Nakov, Eugene Agichtein, Charles L. A. Clarke, Idan Szpektor
Venue: SIGIR
Year: 2016

Web search engines have made great progress at answering factoid queries. However, they are not well-tailored for managing more complex questions, especially when they require explanation and/or description. The WebQA workshop series aims at exploring diverse approaches to answering questions on the Web. This year, particular emphasis will be given to Community Question Answering (CQA), where comments by the users engaged in the forum communities can be used to answer new questions. Questions posted on the Web can be short and ambiguous (similarly to Web queries to a search engine). These issues make the WebQA task more challenging than traditional QA, and finding the most effective approaches for it remains an open problem.
Identifying Careless Workers in Crowdsourcing Platforms: A Game Theory Approach
Authors: Yashar Moshfeghi, Alvaro Francisco Huertas-Rosero, Joemon M. Jose
Venue: SIGIR
Year: 2016

In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.
Beyond Topical Relevance: Studying Understandability and Reliability in Consumer Health Search
Authors: João R. M. Palotti
Venue: SIGIR
Year: 2016

Nowadays people rely on search engines to explore, understand and manage their health. A recent study from Pew Internet states that one in each three adult American Internet users have used the Internet as a diagnosis tool. Retrieving incorrect or unclear health information poses high risks as people may dismiss serious symptoms, use inappropriate treatments or escalate their health concerns about common symptomatology. A number of studies have shown that the average user experiences difficulty in understanding the content of a large portion of the results retrieved by current search engine technology. Other studies have examined how poor the quality of health information on the web can be. In the context of consumer (non-experts) health search, search engines should not only retrieve relevant information, but also promote information that is understandable by the user and that is reliable/trustable and verified. The focus of my Ph.D. is to go beyond topical relevance and study understandability and reliability as two important facets of relevance that must be incorporated into search systems to increase user satisfaction, especially in the context of consumer health search.