OpenRaiser/Intern-Atlas huggingface dataset:
Click-based Hot Fixes for Underperforming Torso Queries
Authors: Masrour Zoghi, Tomás Tunys, Lihong Li, Damien Jose, Junyan Chen, Chun Ming Chin, Maarten de Rijke
Venue: SIGIR
Year: 2016

Ranking documents using their historical click-through rate (CTR) can improve relevance for frequently occurring queries, i.e., so-called head queries. It is difficult to use such click signals on non-head queries as they receive fewer clicks. In this paper, we address the challenge of dealing with torso queries on which the production ranker is performing poorly. Torso queries are queries that occur frequently enough so that they are not considered as tail queries and yet not frequently enough to be head queries either. They comprise a large portion of most commercial search engines' traffic, so the presence of a large number of underperforming torso queries can harm the overall performance significantly. We propose a practical method for dealing with such cases, drawing inspiration from the literature on learning to rank (LTR). Our method requires relatively few clicks from users to derive a strong re-ranking signal by comparing document relevance between pairs of documents instead of using absolute numbers of clicks per document. By infusing a modest amount of exploration into the ranked lists produced by a production ranker and extracting preferences between documents, we obtain substantial improvements over the production ranker in terms of page-level online metrics. We use an exploration dataset consisting of real user clicks from a large-scale commercial search engine to demonstrate the effectiveness of the method. We conduct further experimentation on public benchmark data using simulated clicks to gain insight into the inner workings of the proposed method. Our results indicate a need for LTR methods that make more explicit use of the query and other contextual information.
Transfer Learning for Cross-Lingual Sentiment Classification with Weakly Shared Deep Neural Networks
Venue: SIGIR
Year: 2016

NOTE FROM ACM: It has been determined that this article plagiarized the contents of a previously published paper. Therefore ACM has shut off access to this paper.
From Design to Analysis: Conducting Controlled Laboratory Experiments with Users
Authors: Diane Kelly, Anita Crescenzi
Venue: SIGIR
Year: 2017

This full-day tutorial provides general instruction about the design of controlled laboratory experiments that are conducted in order to better understand human information interaction and retrieval. Different data collection methods and procedures are described, with an emphasis on self-report measures and scales. This tutorial also introduces the use of statistical power analysis for sample size estimation and introduces and demonstrate two data analysis procedures, Multilevel Modeling and Structural Equation Modeling, that allow for examination of the whole set of variables present in interactive information retrieval (IIR) experiments, along with their various effect sizes. The goals of the tutorial are to increase participants' (1) understanding of the uses of controlled laboratory experiments with human participants; (2) understanding of the technical vocabulary and procedures associated with such experiments and (3) confidence in conducting and evaluating IIR experiments. Ultimately, we hope our tutorial will increase research capacity and research quality in IR by providing instruction about best practices to those contemplating interactive IR experiments.
Can Deep Effectiveness Metrics Be Evaluated Using Shallow Judgment Pools?
Authors: Xiaolu Lu, Alistair Moffat, J. Shane Culpepper
Venue: SIGIR
Year: 2017

Increasing test collection sizes and limited judgment budgets create measurement challenges for IR batch evaluations, challenges that are greater when using deep effectiveness metrics than when using shallow metrics, because of the increased likelihood that unjudged documents will be encountered. Here we study the problem of metric score adjustment, with the goal of accurately estimating system performance when using deep metrics and limited judgment sets, assuming that dynamic score adjustment is required per topic due to the variability in the number of relevant documents. We seek to induce system orderings that are as close as is possible to the orderings that would arise if full judgments were available. Starting with depth-based pooling, and no prior knowledge of sampling probabilities, the first phase of our two-stage process computes a background gain for each document based on rank-level statistics. The second stage then accounts for the distributional variance of relevant documents. We also exploit the frequency statistics of pooled relevant documents in order to determine a threshold for dynamically determining the set of topics to be adjusted. Taken together, our results show that: (i) better score estimates can be achieved when compared to previous work; (ii) by setting a global threshold, we are able to adapt our methods to different collections; and (iii) the proposed estimation methods reliably approximate the system orderings achieved when many more relevance judgments are available. We also consider pools generated by a two-strata sampling approach.
Understanding and Modeling Success in Email Search
Authors: Jin Young Kim, Nick Craswell, Susan T. Dumais, Filip Radlinski, Fang Liu
Venue: SIGIR
Year: 2017

Email has been a dominant form of communication for many years, and email search is an important problem. In contrast to other search setting, such as web search, there have been few studies of user behavior and models of email search success. Research in email search is challenging for many reasons including the personal and private nature of the collection. Third party judges can not look at email search queries or email message content requiring new modeling techniques. In this study, we built an opt-in client application which monitors a user's email search activity and then pops up an in-situ survey when a search session is finished. We then merged the survey data with server-side behavioral logs. This approach allows us to study the relationship between session-level outcome and user behavior, and then build a model to predict success for email search based on behavioral interaction patterns. Our results show that generative models (MarkovChain) of success can predict the session-level success of email search better than baseline heuristics and discriminative models (RandomForest). The success model makes use of email-specific log activities such as reply, forward and move, as well as generic signals such as click with long dwell time. The learned model is highly interpretable, and reusable in that it can be applied to unlabeled interaction logs in the future.
Characterizing and Predicting Enterprise Email Reply Behavior
Authors: Liu Yang, Susan T. Dumais, Paul N. Bennett, Ahmed Hassan Awadallah
Venue: SIGIR
Year: 2017

Email is still among the most popular online activities. People spend a significant amount of time sending, reading and responding to email in order to communicate with others, manage tasks and archive personal information. Most previous research on email is based on either relatively small data samples from user surveys and interviews, or on consumer email accounts such as those from Yahoo! Mail or Gmail. Much less has been published on how people interact with enterprise email even though it contains less automatically generated commercial email and involves more organizational behavior than is evident in personal accounts. In this paper, we extend previous work on predicting email reply behavior by looking at enterprise settings and considering more than dyadic communications. We characterize the influence of various factors such as email content and metadata, historical interaction features and temporal features on email reply behavior. We also develop models to predict whether a recipient will reply to an email and how long it will take to do so. Experiments with the publicly-available Avocado email collection show that our methods outperform all baselines with large gains. We also analyze the importance of different features on reply behavior predictions. Our findings provide new insights about how people interact with enterprise email and have implications for the design of the next generation of email clients.
Autonomous Crowdsourcing through Human-Machine Collaborative Learning
Authors: Azad Abad, Moin Nabi, Alessandro Moschitti
Venue: SIGIR
Year: 2017

In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: the classifier (i.e., the machine) selects the highest quality examples for training the crowdsource workers (i.e., the humans). Then, the latter annotate the lower quality examples such that the classifier can be re-trained with more accurate examples. This process can be iterated several times. We tested our approach on two different tasks, Relation Extraction and Community Question Answering, which are also in two different languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. At the same time, our method approach the performance than state-of-the-art methods using expensive Gold Standard for training workers
Non-negative Matrix Factorization Meets Word Embedding
Authors: Melissa Ailem, Aghiles Salah, Mohamed Nadif
Venue: SIGIR
Year: 2017

Document clustering is central in modern information retrieval applications. Among existing models, non-negative-matrix factorization (NMF) approaches have proven effective for this task. However, NMF approaches, like other models in this context, exhibit a major drawback, namely they use the bag-of-word representation and, thus, do not account for the sequential order in which words occur in documents. This is an important issue since it may result in a significant loss of semantics. In this paper, we aim to address the above issue and propose a new model which successfully integrates a word embedding model, word2vec, into an NMF framework so as to leverage the semantic relationships between words. Empirical results, on several real-world datasets, demonstrate the benefits of our model in terms of text document clustering as well as document/word embedding.
A Cross-Platform Collection for Contextual Suggestion
Authors: Mohammad Aliannejadi, Ida Mele, Fabio Crestani
Venue: SIGIR
Year: 2017

Suggesting personalized venues helps users to find interesting places on location-based social networks (LBSNs). Although there are many LBSNs online, none of them is known to have thorough information about all venues. The Contextual Suggestion track at TREC aimed at providing a collection consisting of places as well as user context to enable researchers to examine and compare different approaches, under the same evaluation setting. However, the officially released collection of the track did not meet many participants' needs related to venue content, online reviews, and user context. That is why almost all successful systems chose to crawl information from different LBSNs. For example, one of the best proposed systems in the TREC 2016 Contextual Suggestion track crawled data from multiple LBSNs and enriched it with venue-context appropriateness ratings, collected using a crowdsourcing platform. Such collection enabled the system to better predict a venue's appropriateness to a given user's context. In this paper, we release both collections that were used by the system above. We believe that these datasets give other researchers the opportunity to compare their approaches with the top systems in the track. Also, it provides the opportunity to explore different methods to predicting contextually appropriate venues.
Axiomatic Thinking for Information Retrieval: And Related Tasks
Authors: Enrique Amigó, Hui Fang, Stefano Mizzaro, ChengXiang Zhai
Venue: SIGIR
Year: 2017

This is the first workshop on the emerging interdisciplinary research area of applying axiomatic thinking to information retrieval (IR) and related tasks. The workshop aims to help foster collaboration of researchers working on different perspectives of axiomatic thinking and encourage discussion and research on general methodological issues related to applying axiomatic thinking to IR and related tasks.
EvALL: Open Access Evaluation for Information Access Systems
Authors: Enrique Amigó, Jorge Carrillo de Albornoz, Mario Almagro-Cádiz, Julio Gonzalo, Javier Rodríguez-Vidal, Felisa Verdejo
Venue: SIGIR
Year: 2017

The EvALL online evaluation service aims to provide a unified evaluation framework for Information Access systems that makes results completely comparable and publicly available for the whole research community. For researchers working on a given test collection, the framework allows to: (i) evaluate results in a way compliant with measurement theory and with state-of-the-art evaluation practices in the field; (ii) quantitatively and qualitatively compare their results with the state of the art; (iii) provide their results as reusable data to the scientific community; (iv) automatically generate evaluation figures and (low-level) interpretation of the results, both as a pdf report and as a latex source. For researchers running a challenge (a comparative evaluation campaign on shared data), the framework helps them to manage, store and evaluate submissions, and to preserve ground truth and system output data for future use by the research community. EvALL can be tested at http://evall.uned.es.
Improving Retrieval Performance for Verbose Queries via Axiomatic Analysis of Term Discrimination Heuristic
Authors: Mozhdeh Ariannezhad, Ali Montazeralghaem, Hamed Zamani, Azadeh Shakery
Venue: SIGIR
Year: 2017

Number of terms in a query is a query-specific constant that is typically ignored in retrieval functions. However, previous studies have shown that the performance of retrieval models varies for different query lengths, and it usually degrades when query length increases. A possible reason for this issue can be the extraneous terms in longer queries that makes it a challenge for the retrieval models to distinguish between the key and complementary concepts of the query. As a signal to understand the importance of a term, inverse document frequency (IDF) can be used to discriminate query terms. In this paper, we propose a constraint to model the interaction between query length and IDF. Our theoretical analysis shows that current state-of-the-art retrieval models, such as BM25, do not satisfy the proposed constraint. We further analyze the BM25 model and suggest a modification to adapt BM25 so that it adheres to the new constraint. Our experiments on three TREC collections demonstrate that the proposed modification outperforms the baselines, especially for verbose queries.
Search Without a Query: Powering Job Recommendations via Search Index at LinkedIn
Authors: Dhruv Arya, Ganesh Venkataraman
Venue: SIGIR
Year: 2017

The mission of LinkedIn is to connect the world's professionals to make them more productive and successful. LinkedIn operates the world's largest professional network on the Internet with more than 500 Million members in over 200 countries. Core to realizing the mission is to help people find jobs. In this paper, we describe how the jobs recommendations is powered by a search index and some practical challenges involved in scaling such a system.
Candidate Selection for Large Scale Personalized Search and Recommender Systems
Authors: Dhruv Arya, Ganesh Venkataraman, Aman Grover, Krishnaram Kenthapadi
Venue: SIGIR
Year: 2017

Modern day social media search and recommender systems require complex query formulation that incorporates both user context and their explicit search queries. Users expect these systems to be fast and provide relevant results to their query and context. With millions of documents to choose from, these systems utilize a multi-pass scoring function to narrow the results and provide the most relevant ones to users. Candidate selection is required to sift through all the documents in the index and select a relevant few to be ranked by subsequent scoring functions. It becomes crucial to narrow down the document set while maintaining relevant ones in resulting set. In this tutorial we survey various candidate selection techniques and deep dive into case studies on a large scale social media platform. In the later half we provide hands-on tutorial where we explore building these candidate selection models on a real world dataset and see how to balance the tradeoff between relevance and latency.
The Lucene for Information Access and Retrieval Research (LIARR) Workshop at SIGIR 2017
Authors: Leif Azzopardi, Matt Crane, Hui Fang, Grant Ingersoll, Jimmy Lin, Yashar Moshfeghi, Harrisen Scells, Peilin Yang, Guido Zuccon
Venue: SIGIR
Year: 2017

As an empirical discipline, information access and retrieval research requires substantial software infrastructure to index and search large collections. This workshop is motivated by the desire to better align information retrieval research with the practice of building search applications from the perspective of open-source information retrieval systems. Our goal is to promote the use of Lucene for information access and retrieval research.
Semantic Query Understanding
Authors: Ricardo Baeza-Yates
Venue: SIGIR
Year: 2017

Queries are often ambiguous and can be interpreted in many ways, even by humans. Hence, semantic query understanding's primary objective is to understand the intention behind the query. This implies first predicting the language used to express the query. Second, parsing the query according to that language. Third, extracting the entities and concepts mentioned in the query. Finally, based on all this information, we predict one or more possible intentions with a certain probability, which is particularly important for ambiguous queries. These scores will be one of the inputs for the final semantic ranking. For example, given the query "bond", possible results for query understanding are a financial instrument, the movie character, a chemical reaction, or a term for endearment. Semantic ranking refers to ranking search results using semantic information. In a standard search engine, a rank is computed by using signals or features coming from the search query, from the documents in the collection being searched and from the search context, such as the language and device being used. Using semantic processing, we also add semantic features that come from concepts present in the knowledge base that appear in the query and semantically match documents in the collection. To do this efficiently, all documents are preprocessed semantically to build an index that includes semantic annotations. To accomplish semantic ranking, we use machine learning in several stages. The first stage selects the data sources that we should use to answer the query. In the second stage, each data source generates a set of answers using "learning to rank." The third and final stage ranks these data sources, selecting and ordering the intentions as well as the answers inside each intention (e.g., news) that will appear in the final composite answer. All these techniques are language independent, but may use language dependent features.
Building Bridges across Social Platforms: Answering Twitter Questions with Yahoo! Answers
Authors: Mossaab Bagdouri, Douglas W. Oard
Venue: SIGIR
Year: 2017

This paper investigates techniques for answering microblog questions by searching in a large community question answering website. Some question transformations are considered, some proprieties of the answering platform are examined, how to select among the various available configurations in a learning-to-rank framework is studied.
Detecting Positive Medical History Mentions
Authors: Bing Bai, Pierre-François Laquerre, Richard G. Jackson, Robert Stewart
Venue: SIGIR
Year: 2017

In medical practice, knowing the medical history of a patient is crucial for diagnosis and treatment suggestion. However, such information is often recorded in unstructured notes from doctors, potentially mixed with the medical history of family members and mentions of disorders for other reasons (e.g. as potential side-effects). In this work we designed a scheme to automatically extract the medical history of patients from a large healthcare database. More specifically, we first extracted medical conditions mentioned using a rule-based system and a medical gazetteer, then we classified whether such mentions reflected the patient's history or not. We designed our method to be simple and with little human intervention. Our results are very encouraging, supporting the potential for efficient and effective deployment in clinical practice.
Embedding-based Query Expansion for Weighted Sequential Dependence Retrieval Model
Authors: Saeid Balaneshinkordan, Alexander Kotov
Venue: SIGIR
Year: 2017

Although information retrieval models based on Markov Random Fields (MRF), such as Sequential Dependence Model and Weighted Sequential Dependence Model (WSDM), have been shown to outperform bag-of-words probabilistic and language modeling retrieval models by taking into account term dependencies, it is not known how to effectively account for term dependencies in query expansion methods based on pseudo-relevance feedback (PRF) for retrieval models of this type. In this paper, we propose Semantic Weighted Dependence Model (SWDM), a PRF based query expansion method for WSDM, which utilizes distributed low-dimensional word representations (i.e., word embeddings). Our method finds the closest unigrams to each query term in the embedding space and top retrieved documents and directly incorporates them into the retrieval function of WSDM. Experiments on TREC datasets indicate statistically significant improvement of SWDM over state-of-the-art MRF retrieval models, PRF methods for MRF retrieval models and embedding based query expansion methods for bag-of-words retrieval models.
Multitask Learning for Fine-Grained Twitter Sentiment Analysis
Authors: Georgios Balikas, Simon Moura, Massih-Reza Amini
Venue: SIGIR
Year: 2017

Traditional sentiment analysis approaches tackle problems like ternary (3-category) and fine-grained (5-category) classification by learning the tasks separately. We argue that such classification tasks are correlated and we propose a multitask approach based on a recurrent neural network that benefits by jointly learning them. Our study demonstrates the potential of multitask models on this type of problems and improves the state-of-the-art results in the fine-grained sentiment classification problem.
On the Use of an Intermediate Class in Boolean Crowdsourced Relevance Annotations for Learning to Rank Comments
Authors: Alberto Barrón-Cedeño, Giovanni Da San Martino, Simone Filice, Alessandro Moschitti
Venue: SIGIR
Year: 2017

In many Information Retrieval tasks, the boundary between classes is not well defined, and assigning a document to a specific class may be complicated, even for humans. For instance, a document which is not directly related to the user's query may still contain relevant information. In this scenario, an option is to define an intermediate class collecting ambiguous instances. Yet some natural questions arise. Is this annotation strategy convenient? how should the intermediate class be treated? To answer these questions, we explored two community question answering datasets whose comments were originally annotated with three classes. We re-annotated a subset of instances considering a binary good vs bad setting. Our main contribution is to show empirically that the inclusion of an intermediate class to assess Boolean relevance is not useful. Moreover, in case the data is already annotated with a 3-class strategy, the instances from the intermediate class can be safely removed at training time.