OpenRaiser/Intern-Atlas huggingface dataset:
A Study of SVM Kernel Functions for Sensitivity Classification Ensembles with POS Sequences
Authors: Graham McDonald, Nicolás García-Pedrajas, Craig Macdonald, Iadh Ounis
Venue: SIGIR
Year: 2017

Freedom of Information (FOI) laws legislate that government documents should be opened to the public. However, many government documents contain sensitive information, such as confidential information, that is exempt from release. Therefore, government documents must be sensitivity reviewed prior to release, to identify and close any sensitive information. With the adoption of born-digital documents, such as email, there is a need for automatic sensitivity classification to assist digital sensitivity review. SVM classifiers and Part-of-Speech sequences have separately been shown to be promising for sensitivity classification. However, sequence classification methodologies, and specifically SVM kernel functions, have not been fully investigated for sensitivity classification. Therefore, in this work, we present an evaluation of five SVM kernel functions for sensitivity classification using POS sequences. Moreover, we show that an ensemble classifier that combines POS sequence classification with text classification can significantly improve sensitivity classification effectiveness (+6.09% F2) compared with a text classification baseline, according to McNemar's test of significance.
Extracting Hierarchies of Search Tasks & Subtasks via a Bayesian Nonparametric Approach
Authors: Rishabh Mehrotra, Emine Yilmaz
Venue: SIGIR
Year: 2017

A significant amount of search queries originate from some real world information need or tasks [13]. In order to improve the search experience of the end users, it is important to have accurate representations of tasks. As a result, significant amount of research has been devoted to extracting proper representations of tasks in order to enable search systems to help users complete their tasks, as well as providing the end user with better query suggestions [9], for better recommendations [41], for satisfaction prediction [36] and for improved personalization in terms of tasks [24, 38]. Most existing task extraction methodologies focus on representing tasks as flat structures. However, tasks often tend to have multiple subtasks associated with them and a more naturalistic representation of tasks would be in terms of a hierarchy, where each task can be composed of multiple (sub)tasks. To this end, we propose an efficient Bayesian nonparametric model for extracting hierarchies of such tasks & subtasks. We evaluate our method based on real world query log data both through quantitative and crowdsourced experiments and highlight the importance of considering task/subtask hierarchies.
User Interaction Sequences for Search Satisfaction Prediction
Authors: Rishabh Mehrotra, Imed Zitouni, Ahmed Hassan Awadallah, Ahmed El Kholy, Madian Khabsa
Venue: SIGIR
Year: 2017

Detecting and understanding implicit measures of user satisfaction are essential for meaningful experimentation aimed at enhancing web search quality. While most existing studies on satisfaction prediction rely on users' click activity and query reformulation behavior, often such signals are not available for all search sessions and as a result, not useful in predicting satisfaction. On the other hand, user interaction data (such as mouse cursor movement) is far richer than just click data and can provide useful signals for predicting user satisfaction. In this work, we focus on considering holistic view of user interaction with the search engine result page (SERP) and construct detailed universal interaction sequences of their activity. We propose novel ways of leveraging the universal interaction sequences to automatically extract informative, interpretable subsequences. In addition to extracting frequent, discriminatory and interleaved subsequences, we propose a Hawkes process model to incorporate temporal aspects of user interaction. Through extensive experimentation we show that encoding the extracted subsequences as features enables us to achieve significant improvements in predicting user satisfaction. We additionally present an analysis of the correlation between various subsequences and user satisfaction. Finally, we demonstrate the usefulness of the proposed approach in covering abandonment cases. Our findings provide a valuable tool for fine-grained analysis of user interaction behavior for metric development.
Luandri: A Clean Lua Interface to the Indri Search Engine
Authors: Bhaskar Mitra, Fernando Diaz, Nick Craswell
Venue: SIGIR
Year: 2017

In recent years, the information retrieval (IR) community has witnessed the first successful applications of deep neural network models to short-text matching and ad-hoc retrieval tasks. However, the two communities - focused on deep neural networks and on IR - have less in common when it comes to the choice of programming languages. Indri, an indexing framework popularly used by the IR community, is written in C++, while Torch, a popular machine learning library for deep learning, is written in the light-weight scripting language Lua. To bridge this gap, we introduce Luandri (pronounced "laundry"), a simple interface for exposing the search capabilities of Indri to Torch models implemented in Lua.
Predicting Information Seeking Intentions from Search Behaviors
Authors: Matthew Mitsui, Jiqun Liu, Nicholas J. Belkin, Chirag Shah
Venue: SIGIR
Year: 2017

It has been shown that people attempt to accomplish a variety of intentions during the course of an information seeking session, and there is reason to believe that these different information seeking intentions can benefit from system support tailored to each such intention. We address the problem of predicting the presence of such intentions during an information seeking session, through analysis of observable user search behaviors. We present results of a study of 40 participants, each working on two different journalism tasks, which investigated how their search behaviors could indicate their intentions. Using 725 query-segments captured from this study, we demonstrate that information seeking intentions can be predicted with a simple classification model using a linear combination of search behavior features that can be logged with a browser plug-in.
Term Proximity Constraints for Pseudo-Relevance Feedback
Authors: Ali Montazeralghaem, Hamed Zamani, Azadeh Shakery
Venue: SIGIR
Year: 2017

Pseudo-relevance feedback (PRF) refers to a query expansion strategy based on top-retrieved documents, which has been shown to be highly effective in many retrieval models. Previous work has introduced a set of constraints (axioms) that should be satisfied by any PRF model. In this paper, we propose three additional constraints based on the proximity of feedback terms to the query terms in the feedback documents. As a case study, we consider the log-logistic model, a state-of-the-art PRF model that has been proven to be a successful method in satisfying the existing PRF constraints, and show that it does not satisfy the proposed constraints. We further modify the log-logistic model based on the proposed proximity-based constraints. Experiments on four TREC collections demonstrate the effectiveness of the proposed constraints. Our modification the log-logistic model leads to significant and substantial (up to 15%) improvements. Furthermore, we show that the proposed proximity-based function outperforms the well-known Gaussian kernel which does not satisfy all the proposed constraints.
Information Retrieval Model using Generalized Pareto Distribution and Its Application to Instance Search
Authors: Masaya Murata, Kaoru Hiramatsu, Shin&apos, ichi Satoh
Venue: SIGIR
Year: 2017

We adopt the generalized Pareto distribution for the information-based model and show that the parameters can be estimated based on the mean excess function. The proposed information retrieval model corresponds to the extension of the divergence from independence and is designed to be data-driven. The proposed model is then applied to the specific object search called the instance search and the effectiveness is experimentally confirmed.
Skill Translation Models in Expert Finding
Authors: Arash Dargahi Nobari, Sajad Sotudeh Gharebagh, Mahmood Neshati
Venue: SIGIR
Year: 2017

Finding talented users on Stackoverflow can be a challenging task due to term mismatch between queries and content published on it. In this paper, we propose two translation models to augment a given query with relevant words. The first model is based on a statistical approach and the second one is a word embedding model. Interestingly, the translations provided by these methods are not the same. Although the first model in most cases selects pieces of program codes as translations, the second model provides more semantically related words. Our experiments on a large dataset indicate the efficiency of proposed models.
Applying Information Extraction for Patent Structure Analysis
Authors: Masayuki Okamoto, Zifei Shan, Ryohei Orihara
Venue: SIGIR
Year: 2017

Patent engineers are spending significant time analyzing patent claim structures to grasp the range of technology covered or to compare similar patents in the same patent family. Though patent claims are the most important section in a patent, it is hard for a human to examine them. In this paper, we propose an information-extraction-based technique to grasp the patent claim structure. We confirmed that our approach is promising through empirical evaluation of entity mention extraction and the relation extraction method. We also built a preliminary interface to visualize patent structures, compare patents, and search similar patents.
Seeing Bot
Authors: Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, Tao Mei
Venue: SIGIR
Year: 2017

We demonstrate a video captioning bot, named Seeing Bot, which can generate a natural language description about what it is seeing in near real time. Specifically, given a live streaming video, Seeing Bot runs two pre-learned and complementary captioning modules in parallel - one for generating image-level caption for each sampled frame, and the other for generating video-level caption for each sampled video clip. In particular, both the image and video captioning modules are boosted by incorporating semantic attributes which can enrich the generated descriptions, leading to human-level caption generation. A visual-semantic embedding model is then exploited to rank and select the final caption from the two parallel modules by considering the semantic relevance between video content and the generated captions. The Seeing Bot finally converts the generated description to speech and sends the speech to an end user via an earphone. Our demonstration is conducted on any videos in the wild and supports live video captioning.
A Neural Language Model for Query Auto-Completion
Authors: Dae Hoon Park, Rikio Chiba
Venue: SIGIR
Year: 2017

Query auto-completion (QAC) systems suggest queries that complete a user's text as the user types each character. Such queries are typically selected among previously stored queries, based on specific attributes such as popularity. However, queries cannot be suggested if a user's text does not match any queries in the storage. In order to suggest queries for previously unseen text, we propose a neural language model that learns how to generate a query from a starting text, a prefix. Specifically, we employ a recurrent neural network to handle prefixes in variable length. We perform the first neural language model experiments for QAC, and we evaluate the proposed methods with a public data set. Empirical results show that the proposed methods are as effective as traditional methods for previously seen queries and are superior to the state-of-the-art QAC method for previously unseen queries.
Cross-Device User Linking: URL, Session, Visiting Time, and Device-log Embedding
Authors: Minh C. Phan, Aixin Sun, Yi Tay
Venue: SIGIR
Year: 2017

Cross-Device User Linking is the task of detecting same users given their browsing logs on different devices (e.g., tablet, mobile phone, PC, etc.). The problem was introduced in CIKM Cup 2016 together with a new dataset provided by Data-Centric Alliance (DCA). In this paper, we present insightful analysis on the dataset and propose a solution to link users based on their visited URLs, visiting time, and profile embeddings. We cast the problem as pairwise classification and use gradient boosting as the leaning-to-rank model. Our model works on a set of features exacted from URLs, titles, time and session data derived from user device-logs. The model outperforms the best solution in the CIKM Cup by a large margin.
Twicalli: An Earthquake Detection System Based on Citizen Sensors Used for Emergency Response in Chile
Authors: Barbara Poblete
Venue: SIGIR
Year: 2017

In this talk I will describe "Twicalli", a real-time earthquake detection system based on citizen sensors. This system is publicly available for over a year, at http://twicalli.cl, and is currently in use as a decision support tool by the National Seismology Office and by the Hydrographic and the Oceanographic Service in Chile. The novelty of our system relies on the fact that it has a very good precision and recall tradeoff for earthquakes of all magnitude ranges that were reported on Twitter. Our earthquake detection methodology is simple, efficient, unsupervised, and it can detect earthquakes reported globally in any language and any region. This complements existing approaches that are either: i) supervised and customized to a particular geographical region, which makes them very expensive to scale geographically and keep up-to-date, or ii) unsupervised with low earthquake recall.The evaluation of our system, performed during a 9-month period, shows that our solution is competitive to the best state-of-the-art methods, providing very good precision and recall performance for a wide range of earthquake magnitudes.
Online Learning to Rank for Cross-Language Information Retrieval
Authors: Razieh Rahimi, Azadeh Shakery
Venue: SIGIR
Year: 2017

Online learning to rank for information retrieval has shown great promise in optimization of Web search results based on user interactions. However, online learning to rank has been used only in the monolingual setting where queries and documents are in the same language. In this work, we present the first empirical study of optimizing a model for Cross-Language Information Retrieval (CLIR) based on implicit feedback inferred from user interactions. We show that ranking models for CLIR with acceptable performance can be learned in an online setting although ranking features are noisy because of the language mismatch.
Information Retrieval Meets Game Theory: The Ranking Competition Between Documents? Authors
Authors: Nimrod Raifer, Fiana Raiber, Moshe Tennenholtz, Oren Kurland
Venue: SIGIR
Year: 2017

In competitive search settings as the Web, there is an ongoing ranking competition between document authors (publishers) for certain queries. The goal is to have documents highly ranked, and the means is document manipulation applied in response to rankings. Existing retrieval models, and their theoretical underpinnings (e.g., the probability ranking principle), do not account for post-ranking corpus dynamics driven by this strategic behavior of publishers. However, the dynamics has major effect on retrieval effectiveness since it affects content availability in the corpus. Furthermore, while manipulation strategies observed over the Web were reported in past literature, they were not analyzed as ongoing, and changing, post-ranking response strategies, nor were they connected to the foundations of classical ad hoc retrieval models (e.g., content-based document-query surface level similarities and document relevance priors). We present a novel theoretical and empirical analysis of the strategic behavior of publishers using these foundations. Empirical analysis of controlled ranking competitions that we organized reveals a key strategy of publishers: making their documents (gradually) become similar to documents ranked the highest in previous rankings. Our theoretical analysis of the ranking competition as a repeated game, and its minmax regret equilibrium, yields a result that supports the merits of this publishing strategy. We further show that it can be predicted with high accuracy, and without explicit knowledge of the ranking function, whether documents will be promoted to the highest rank in our competitions. The prediction utilizes very few features which quantify changes of documents, specifically with respect to those previously ranked the highest.
Experiments with Convolutional Neural Network Models for Answer Selection
Authors: Jinfeng Rao, Hua He, Jimmy Lin
Venue: SIGIR
Year: 2017

In recent years, neural networks have been applied to many text processing problems. One example is learning a similarity function between pairs of text, which has applications to paraphrase extraction, plagiarism detection, question answering, and ad hoc retrieval. Within the information retrieval community, the convolutional neural network model proposed by Severyn and Moschitti in a SIGIR 2015 paper has gained prominence. This paper focuses on the problem of answer selection for question answering: we attempt to replicate the results of Severyn and Moschitti using their open-source code as well as to reproduce their results via a de novo (i.e., from scratch) implementation using a completely different deep learning toolkit. Our de novo implementation is instructive in ascertaining whether reported results generalize across toolkits, each of which have their idiosyncrasies. We were able to successfully replicate and reproduce the reported results of Severyn and Moschitti, albeit with minor differences in effectiveness, but affirming the overall design of their model. Additional ablation experiments break down the components of the model to show their contributions to overall effectiveness. Interestingly, we find that removing one component actually increases effectiveness and that a simplified model with only four word overlap features performs surprisingly well, even better than convolution feature maps alone.
Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model
Authors: Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Jun Ma, Maarten de Rijke
Venue: SIGIR
Year: 2017

As a framework for extractive summarization, sentence regression has achieved state-of-the-art performance in several widely-used practical systems. The most challenging task within the sentence regression framework is to identify discriminative features to encode a sentence into a feature vector. So far, sentence regression approaches have neglected to use features that capture contextual relations among sentences.
Forward to the Past: Notes towards a Pre-history of Web Search
Authors: Stephen Robertson
Venue: SIGIR
Year: 2017

To me, an awareness of history is a fundamental requirement for progress; and I believe that we in the field of information retrieval are currently ill-served in this domain, or at least not as aware as we should be. While it is true that a researcher in IR is expected to acquire some knowledge of what has gone before, this knowledge is typically fairly narrow in scope. there are some IR researchers who regard IR as a branch of computer science, for example, despite the fact that the field has a long and venerable history entirely outside the domain of computers, as well as a considerable current presence in academic departments well away from CSDs. Outside of our immediate community, there is a widespread belief that web search engines arose out of nothing, a totally new invention for the totally new world of the web. This is unfortunate. It is true that there were huge changes in the information retrieval world in the second half of the twentiethcentury. The ideas, methods and systems that were around in 2000 (particularly the early web search engines, up to and including Google) seem at first glance to be completely different from the ideas, methods and systems that were around in 1950. But these developments involved in large measure the usual process of evolution rather than revolution, and the usual mix of steps large or small, forwards or backwards or sideways. Some of these steps were taken in the world of IR research, and some in the domain of practical commercial systems.
Online In-Situ Interleaved Evaluation of Real-Time Push Notification Systems
Authors: Adam Roegiest, Luchen Tan, Jimmy Lin
Venue: SIGIR
Year: 2017

Real-time push notification systems monitor continuous document streams such as social media posts and alert users to relevant content directly on their mobile devices. We describe a user study of such systems in the context of the TREC 2016 Real-Time Summarization Track, where system updates are immediately delivered as push notifications to the mobile devices of a cohort of users. Our study represents, to our knowledge, the first deployment of an interleaved evaluation framework for prospective information needs, and also provides an opportunity to examine user behavior in a realistic setting. Results of our online in-situ evaluation are correlated against the results a more traditional post-hoc batch evaluation. We observe substantial correlations between many online and batch evaluation metrics, especially for those that share the same basic design (e.g., are utility-based). For some metrics, we observe little correlation, but are able to identify the volume of messages that a system pushes as one major source of differences.
Generalized Mixed Effect Models for Personalizing Job Search
Authors: Ankan Saha, Dhruv Arya
Venue: SIGIR
Year: 2017

Job Search is a core product at LinkedIn which makes it essential to generate highly relevant search results when a user searches for jobs on Linkedin. Historically job results were ranked using linear models consisting of a combination of user, job and query features. This paper talks about a new generalized mixed effect models introduced in the context of ranking candidate job results for a job search performed on LinkedIn. We build a per-query model which is populated with coefficients corresponding to job-features in addition to the existing global model features. We describe the details of the new method along with the challenges faced in launching such a model into production and making it efficient at a very large scale. Our experiments show improvement over previous baseline ranking models, in terms of offline metrics (both AUC and [email protected] metrics) as well as online metrics in production (Job Applies) which are of interest to us. The resulting method is more powerful and has also been adopted in other applications at LinkedIn successfully.