OpenRaiser/Intern-Atlas huggingface dataset:
Addressing News-Related Standing Information Needs
Authors: Kristine M. Rogers
Venue: SIGIR
Year: 2018

A user with a standing need for updates on current events uses a structured exploration process for finding and reviewing new documents, with the user comparing document information to her mental model. To avoid missing key changes on the topic, the user should see some documents on each of the subtopics available that day. This research includes a system and evaluation approach for this standing need use case.
CHEERS: CHeap & Engineered Evaluation of Retrieval Systems
Authors: Kevin Roitero
Venue: SIGIR
Year: 2018

In test collection based evaluation of retrieval effectiveness, many research investigated different directions for an economical and a semi-automatic evaluation of retrieval systems. Although several methods have been proposed and experimentally evaluated, their accuracy seems still limited. In this paper we present our proposal for a more engineered approach to information retrieval evaluation.
IRevalOO: An Object Oriented Framework for Retrieval Evaluation
Authors: Kevin Roitero, Eddy Maddalena, Yannick Ponte, Stefano Mizzaro
Venue: SIGIR
Year: 2018

We propose IRevalOO, a flexible Object Oriented framework that (i) can be used as-is as a replacement of the widely adopted trec\_eval software, and (ii) can be easily extended (or "instantiated'', in framework terminology) to implement different scenarios of test collection based retrieval evaluation. Instances of IRevalOO can provide a usable and convenient alternative to the state-of-the-art software commonly used by different initiatives (TREC, NTCIR, CLEF, FIRE, etc.). Also, those instances can be easily adapted to satisfy future customization needs of researchers, as: implementing and experimenting with new metrics, even based on new notions of relevance; using different formats for system output and "qrels''; and in general visualizing, comparing, and managing retrieval evaluation results.
Effectiveness Evaluation with a Subset of Topics: A Practical Approach
Authors: Kevin Roitero, Michael Soprano, Stefano Mizzaro
Venue: SIGIR
Year: 2018

Several researchers have proposed to reduce the number of topics used in TREC-like initiatives. One research direction that has been pursued is what is the optimal topic subset of a given cardinality that evaluates the systems/runs in the most accurate way. Such a research direction has been so far mainly theoretical, with almost no indication on how to select the few good topics in practice. We propose such a practical criterion for topic selection: we rely on the methods for automatic system evaluation without relevance judgments, and by running some experiments on several TREC collections we show that the topics selected on the basis of those evaluations are indeed more informative than random topics.
Identifying Sub-events and Summarizing Disaster-Related Information from Microblogs
Authors: Koustav Rudra, Pawan Goyal, Niloy Ganguly, Prasenjit Mitra, Muhammad Imran
Venue: SIGIR
Year: 2018

In recent times, humanitarian organizations increasingly rely on social media to search for information useful for disaster response. These organizations have varying information needs ranging from general situational awareness (i.e., to understand a bigger picture) to focused information needs e.g., about infrastructure damage, urgent needs of affected people. This research proposes a novel approach to help crisis responders fulfill their information needs at different levels of granularities. Specifically, the proposed approach presents simple algorithms to identify sub-events and generate summaries of big volume of messages around those events using an Integer Linear Programming (ILP) technique. Extensive evaluation on a large set of real world Twitter dataset shows (a). our algorithm can identify important sub-events with high recall (b). the summarization scheme shows (6---30%) higher accuracy of our system compared to many other state-of-the-art techniques. The simplicity of the algorithms ensures that the entire task is done in real time which is needed for practical deployment of the system.
Interactive Symptom Elicitation for Diagnostic Information Retrieval
Authors: Tuukka Ruotsalo, Antti Lipsanen
Venue: SIGIR
Year: 2018

Medical information retrieval suffers from a dual problem: users struggle in describing what they are experiencing from a medical perspective and the search engine is struggling in retrieving the information exactly matching what users are experiencing. We demonstrate interactive symptom elicitation for diagnostic information retrieval. Interactive symptom elicitation builds a model from the user's initial description of the symptoms and interactively elicitates new information about symptoms by posing questions of related, but uncertain, symptoms for the user. As a result, the system interactively learns the estimates of symptoms while controlling the uncertainties related to the diagnostic process. The learned model is then used to rank the associated diagnoses that the user might be experiencing. Our preliminary experimental results show that interactive symptom elicitation can significantly improve user's capability to describe their symptoms, increase the confidence of the model, and enable effective diagnostic information retrieval.
Conducting Laboratory Experiments Properly with Statistical Tools: An Easy Hands-on Tutorial
Authors: Tetsuya Sakai
Venue: SIGIR
Year: 2018

This hands-on half-day tutorial consists of two 90-minute sessions. Part I covers the following topics: paired and two-sample t -tests, confidence intervals (with Excel and R); familywise error rate, multiple comparison procedures; ANOVA (with Excel and R); Tukey's HSD test, simultaneous confidence intervals (with R). Part II covers the following topics: randomised Tukey HSD test (with Discpower); what's wrong with statistical significance tests?; effect sizes, statistical power; topic set size design (with Excel); power analysis (with R); summary: how to report your results. Participants should have some prior knowledge about the very basics of statistical significance testing and are strongly encouraged to bring a laptop with R already installed. The tutorial participants will be able to design and conduct statistical significance tests for comparing the mean effectiveness scores of two or more systems appropriately, and to report on the test results in an informative manner.
A Living Lab Study of Query Amendment in Job Search
Authors: Bahar Salehi, Damiano Spina, Alistair Moffat, Sargol Sadeghi, Falk Scholer, Timothy Baldwin, Lawrence Cavedon, Mark Sanderson, Wilson Wong, Justin Zobel
Venue: SIGIR
Year: 2018

Errors in formulation of queries made by users can lead to poor search results pages. We performed a living lab study using online A/B testing to measure the degree of improvement achieved with a query amendment technique when applied to a commercial job search engine. Of particular interest in this case study is a clear 'success' signal, namely, the number of job applications lodged by a user as a result of querying the service. A set of 276 queries was identified for amendment in four different categories through the use of word embeddings, with large gains in conversion rates being attained in all four of those categories. Our analysis of query reformulations also provides a better understanding of user satisfaction in the case of problematic queries (ones with fewer results than fill a single page) by observing that users tend to reformulate rewritten queries less.
LessonWare: Mining Student Notes to Provide Personalized Feedback
Authors: Perry Samson, Charles Bassam
Venue: SIGIR
Year: 2018

A new educational service has been prototyped by Echo360 that uses natural language processing to analyze students notes and provide personalized recommendations on how to both improve note-taking and scaffold learning. The LessonWare middleware system uses computer-generated transcriptions from class captures and available student notes to identify key terms mentioned during class sessions. The combination of analyzed key terms and corresponding timestamps allows contextual linkages to be created between educational resources. Student notes are automatically augmented with corresponding moments in class captures, specific pages in the course eTextbook or open education resources or specific adaptive learning assets.
A Test Collection for Coreferent Mention Retrieval
Authors: Rashmi Sankepally, Tongfei Chen, Benjamin Van Durme, Douglas W. Oard
Venue: SIGIR
Year: 2018

This paper introduces the coreferent mention retrieval task, in which the goal is to retrieve sentences that mention a specific entity based on a query by example in which one sentence mentioning that entity is provided. The development of a coreferent mention retrieval test collection is then described. Results are presented for five coreferent mention retrieval systems, both to illustrate the use of the collection and to specify the results that were pooled on which human coreference judgments were performed. The new test collection is built from content that is available from the Linguistic Data Consortium; the partitioning and human annotations used to create the test collection atop that content are being made freely available.
Improving Systematic Review Creation With Information Retrieval
Authors: Harrisen Scells
Venue: SIGIR
Year: 2018

Systematic reviews, in particular medical systematic reviews, are time consuming and costly to produce but are of value for clinical decision making, policy, and regulations. The largest contributing factors to the time and monetary costs are the searching (including the formulation of queries) and screening processes. These initial processes involve researchers reading the abstracts of thousands and sometimes hundreds of thousands of research articles to determine if the retrieved articles should be included or excluded from the systematic review. This research explores automatic methodologies to reduce the workload relating to the searching and initial screening processes. The objective of this research is to use Information Retrieval techniques to improve the retrieval of literature for medical systematic reviews.
An Information Retrieval Experiment Framework for Domain Specific Applications
Authors: Harrisen Scells, Daniel Locke, Guido Zuccon
Venue: SIGIR
Year: 2018

We present a framework for constructing and executing information retrieval experiment pipelines. The framework as a whole is built primarily for domain specific applications such as medical literature search for systematic reviews, or finding factually or legally applicable case law in the legal domain; however it can also be used for more general tasks. There are a number of pre-implemented components that enable common information retrieval experiments such as ad-hoc retrieval or query analysis through query performance predictors. In addition, this collection of tools seeks to be user friendly, well documented, and easily extendible. Finally, the entire pipeline can be distributed as a single binary with no dependencies, ready to use with a simple domain specific language (DSL) for constructing pipelines.
Killing Two Birds With One Stone: Concurrent Ranking of Tags and Comments of Social Images
Authors: Boon-Siew Seah, Aixin Sun, Sourav S. Bhowmick
Venue: SIGIR
Year: 2018

User-generated comments and tags can reveal important visual concepts associated with an image in Flickr. However, due to the inherent noisiness of the metadata, not all user tags are necessarily descriptive of the image. Likewise, comments may contain spam or chatter that are irrelevant to the image. Hence, identifying and ranking relevant tags and comments can boost applications such as tag-based image search, tag recommendation, etc. In this paper, we present a lightweight visual signature-based model to concurrently generate ranked lists of comments and tags of a social image based on their joint relevance to the visual features, user comments, and user tags. The proposed model is based on sparse reconstruction of the visual content of an image using its tags and comments. Through empirical study on Flickr dataset, we demonstrate the effectiveness and superiority of the proposed technique against state-of-the-art tag ranking and refinement techniques.
Measuring Influence on Instagram: A Network-Oblivious Approach
Authors: Noam Segev, Noam Avigdor, Eytan Avigdor
Venue: SIGIR
Year: 2018

This paper focuses on the problem of scoring and ranking influential users of Instagram, a visual content sharing online social network (OSN). Instagram is the second largest OSN in the world with 700 million active Instagram accounts, 32% of all worldwide Internet users. Among the millions of users, photos shared by more influential users are viewed by more users than posts shared by less influential counterparts. This raises the question of how to identify those influential Instagram users. In our work, we present and discuss the lack of relevant tools and insufficient metrics for influence measurement, focusing on a network oblivious approach and show that the graph-based approach used in other OSNs is a poor fit for Instagram. In our study, we consider user statistics, some of which are more intuitive than others, and several regression models to measure users' influence.
An Information Retrieval Framework for Contextual Suggestion Based on Heterogeneous Information Network Embeddings
Authors: Dominic Seyler, Praveen Chandar, Matthew Davis
Venue: SIGIR
Year: 2018

We present an Information Retrieval framework that leverages Heterogeneous Information Network (HIN) embeddings for contextual suggestion. Our method represents users, documents and other context-related documents as heterogeneous objects in a HIN. Using meta-paths, selected based on domain knowledge, we create graph embeddings from this network, thereby learning a representation of users and objects in the same semantic vector space. This allows inferences of user interest on unseen objects based on distance in the embedding space. These object distances are then incorporated as features in a well-established learning to rank (LTR) framework. We make use of the 2016 TREC Contextual Suggestion (TRECCS) dataset, which contains user profiles in the form of relevance-rated documents, and demonstrate the competitiveness of our approach by comparing our system to the best performing systems of the TRECCS task.
Combined Regression and Tripletwise Learning for Conversion Rate Prediction in Real-Time Bidding Advertising
Authors: Lili Shan, Lei Lin, Chengjie Sun
Venue: SIGIR
Year: 2018

In real-time bidding advertising (RTB), the buyers bid for individual advertisement impressions provided by publishers in real time. The final goal of the buyers is to maximize the return on their investment. To gain higher returns, buyers prefer to first purchase more conversion impressions than click-only ones and then purchase more click-only impressions prior to non-click ones. Simultaneously, to reduce the expense, they need to accurately estimate a reasonable bid price, the predicted precision of which depends on the precision of the predicted conversion rate (CVR) or predicted click-through rate (CTR). Therefore, the predicted CVR or predicted CTR must provide not only good ranking values but also correct regression estimations. This paper is focused on the CVR estimation problem for buy-sides in RTB and a combined regression and tripletwise ranking method (CRT) is proposed that jointly considers regression loss and tripletwise ranking loss to estimate the CVR. This method attempts to rank conversion impressions above click-only ones and simultaneously rank click-only impressions above non-click ones. Meanwhile, through simultaneously utilizing the historical conversion and click information to alleviate sparsity, the CRT method is also aimed to achieve a good two category-ranking performance, as well as a good regression performance for predicting the CVR.
Utilizing Inter-Passage Similarities for Focused Retrieval
Authors: Eilon Sheetrit
Venue: SIGIR
Year: 2018

Our main goal is studying the merits of using inter-passage similarities for the task of focused retrieval; i.e., ranking passages in documents by their relevance to an information need expressed by a query. As an initial research direction we study the cluster hypothesis for passage (focused) retrieval. We propose a novel suite of cluster hypothesis tests that employ inter-passage similarities and demonstrate that the cluster hypothesis holds for passages. In addition, we present several future directions we intend to pursue.
Testing the Cluster Hypothesis with Focused and Graded Relevance Judgments
Authors: Eilon Sheetrit, Anna Shtok, Oren Kurland, Igal Shprincis
Venue: SIGIR
Year: 2018

The cluster hypothesis is a fundamental concept in ad hoc retrieval. Heretofore, cluster hypothesis tests were applied to documents using binary relevance judgments. We present novel tests that utilize graded and focused relevance judgments; the latter are markups of relevant text in relevant documents. Empirical exploration reveals that the cluster hypothesis holds not only for documents, but also for passages, as measured by the proposed tests. Furthermore, the hypothesis holds to a higher extent for highly relevant documents and for those that contain a high fraction of relevant text.
Ontology Evaluation with Path-based Text-aware Entropy Computation
Authors: Ying Shen, Daoyuan Chen, Min Yang, Yaliang Li, Nan Du, Kai Lei
Venue: SIGIR
Year: 2018

With the rising importance of knowledge exchange, ontologies have become a key technology in the development of shared knowledge models for semantic-driven applications, such as knowledge interchange and semantic integration. Significant progress has been made in the use of entropy to measure the predictability and redundancy of knowledge bases, particularly ontologies. However, the current entropy applications used to evaluate ontologies consider only single-point connectivity rather than path connectivity, assign equal weights to each entity and path, and assume that vertices are static. To address these deficiencies, the present study proposes a Path-based Text-aware Entropy Computation method, PTEC, by considering the path information between different vertices and the textual information within the path to calculate the connectivity path of the whole network and the different weights between various nodes. Information obtained from structure-based embedding and text-based embedding is multiplied by the connectivity matrix of the entropy computation. An experimental evaluation of three real-world ontologies is performed based on ontology statistical information (data quantity), entropy evaluation (data quality), and a case study (ontology structure and text visualization). These aspects mutually demonstrate the reliability of our method. Experimental results demonstrate that PTEC can effectively evaluate ontologies, particularly those in the medical field.
Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs
Authors: Ying Shen, Yang Deng, Min Yang, Yaliang Li, Nan Du, Wei Fan, Kai Lei
Venue: SIGIR
Year: 2018

Ranking question answer pairs has attracted increasing attention recently due to its broad applications such as information retrieval and question answering (QA). Significant progresses have been made by deep neural networks. However, background information and hidden relations beyond the context, which play crucial roles in human text comprehension, have received little attention in recent deep neural networks that achieve the state of the art in ranking QA pairs. In the paper, we propose KABLSTM, a Knowledge-aware Attentive Bidirectional Long Short-Term Memory, which leverages external knowledge from knowledge graphs (KG) to enrich the representational learning of QA sentences. Specifically, we develop a context-knowledge interactive learning architecture, in which a context-guided attentive convolutional neural network (CNN) is designed to integrate knowledge embeddings into sentence representations. Besides, a knowledge-aware attention mechanism is presented to attend interrelations between each segments of QA pairs. KABLSTM is evaluated on two widely-used benchmark QA datasets: WikiQA and TREC QA. Experiment results demonstrate that KABLSTM has robust superiority over competitors and sets state-of-the-art.
Dynamic Composition of Question Answering Pipelines with FRANKENSTEIN
Authors: Kuldeep Singh, Ioanna Lytra, Arun Sethupat Radhakrishna, Akhilesh Vyas, Maria-Esther Vidal
Venue: SIGIR
Year: 2018

Question answering (QA) systems provide user-friendly interfaces for retrieving answers from structured and unstructured data given natural language questions. Several QA systems, as well as related components, have been contributed by the industry and research community in recent years. However, most of these efforts have been performed independently from each other and with different focuses, and their synergies in the scope of QA have not been addressed adequately. FRANKENSTEIN is a novel framework for developing QA systems over knowledge bases by integrating existing state-of-the-art QA components performing different tasks. It incorporates several reusable QA components, employs machine learning techniques to predict best performing components and QA pipelines for a given question, and generates static and dynamic executable QA pipelines. In this paper, we illustrate different functionalities of FRANKENSTEIN for performing independent QA component execution, QA component prediction, given an input question as well as the static and dynamic composition of different QA pipelines.
A Personal Privacy Preserving Framework: I Let You Know Who Can See What
Authors: Xuemeng Song, Xiang Wang, Liqiang Nie, Xiangnan He, Zhumin Chen, Wei Liu
Venue: SIGIR
Year: 2018

The booming of social networks has given rise to a large volume of user-generated contents (UGCs), most of which are free and publicly available. A lot of users' personal aspects can be extracted from these UGCs to facilitate personalized applications as validated by many previous studies. Despite their value, UGCs can place users at high privacy risks, which thus far remains largely untapped. Privacy is defined as the individual's ability to control what information is disclosed, to whom, when and under what circumstances. As people and information both play significant roles, privacy has been elaborated as a boundary regulation process, where individuals regulate interaction with others by altering the openness degree of themselves to others. In this paper, we aim to reduce users' privacy risks on social networks by answering the question of Who Can See What. Towards this goal, we present a novel scheme, comprising of descriptive, predictive and prescriptive components. In particular, we first collect a set of posts and extract a group of privacy-oriented features to describe the posts. We then propose a novel taxonomy-guided multi-task learning model to predict which personal aspects are uncovered by the posts. Lastly, we construct standard guidelines by the user study with 400 users to regularize users' actions for preventing their privacy leakage. Extensive experiments on a real-world dataset well verified our scheme.