OpenRaiser/Intern-Atlas huggingface dataset:
Reranking for Efficient Transformer-based Answer Selection
Authors: Yoshitomo Matsubara, Thuy Vu, Alessandro Moschitti
Venue: SIGIR
Year: 2020

IR-based Question Answering (QA) systems typically use a sentence selector to extract the answer from retrieved documents. Recent studies have shown that powerful neural models based on the Transformer can provide an accurate solution to Answer Sentence Selection (AS2). Unfortunately, their computation cost prevents their use in real-world applications. In this paper, we show that standard and efficient neural rerankers can be used to reduce the amount of sentence candidates fed to Transformer models without hurting Accuracy, thus improving efficiency up to four times. This is an important finding as the internal representation of shallower neural models is dramatically different from the one used by a Transformer model, e.g., word vs. contextual embeddings.
Active Learning Stopping Strategies for Technology-Assisted Sensitivity Review
Authors: Graham McDonald, Craig Macdonald, Iadh Ounis
Venue: SIGIR
Year: 2020

Active learning strategies are often deployed in technology-assisted review tasks, such as e-discovery and sensitivity review, to learn a classifier that can assist the reviewers with their task. In particular, an active learning strategy selects the documents that are expected to be the most useful for learning an effective classifier, so that these documents can be reviewed before the less useful ones. However, when reviewing for sensitivity, the order in which the documents are reviewed can impact on the reviewers' ability to perform the review. Therefore, when deploying active learning in technology-assisted sensitivity review, we want to know when a sufficiently effective classifier has been learned, such that the active learning can stop and the reviewing order of the documents can be selected by the reviewer instead of the classifier. In this work, we propose two active learning stopping strategies for technology-assisted sensitivity review. We evaluate the effectiveness of our proposed approaches in comparison with three state-of-the-art stopping strategies from the literature. We show that our best performing approach results in a significantly more effective sensitivity classifier (+6.6% F2) than the best performing stopping strategy from the literature (McNemar's test, p<0.05).
Agreement and Disagreement between True and False-Positive Metrics in Recommender Systems Evaluation
Authors: Elisa Mena-Maldonado, Rocío Cañamares, Pablo Castells, Yongli Ren, Mark Sanderson
Venue: SIGIR
Year: 2020

False-positive metrics can capture an important side of recommendation quality, focusing on the impact of suggestions that are disliked by users, as a complement of common metrics that only measure the amount of successful recommendations. In this paper we research the extent to which false-positive metrics agree or disagree with true-positive metrics in the offline evaluation of recommender systems. We discover a surprising degree of systematic disagreement that was occasionally noted but not explained in the literature by previous authors. We find an explanation for the discrepancy be-tween the metrics in the effect of popularity biases, which impact false and true-positive metrics in very different ways: instead of rewarding the recommendation of popular items, as with true-positive, false-positive metrics penalize the popular. We determine precise conditions and cases in the general trends, with a formal explanation for our findings, which we confirm and illustrate empirically in experiments with different datasets.
DukeNet: A Dual Knowledge Interaction Network for Knowledge-Grounded Conversation
Authors: Chuan Meng, Pengjie Ren, Zhumin Chen, Weiwei Sun 0001, Zhaochun Ren, Zhaopeng Tu, Maarten de Rijke
Venue: SIGIR
Year: 2020

Today's conversational agents often generate responses that not sufficiently informative. One way of making them more informative is through the use of of external knowledge sources with so-called Knowledge-Grounded Conversations (KGCs). In this paper, we target the Knowledge Selection (KS) task, a key ingredient in KGC, that is aimed at selecting the appropriate knowledge to be used in the next response. Existing approaches to Knowledge Selection (KS) based on learned representations of the conversation context, that is previous conversation turns, and use Maximum Likelihood Estimation (MLE) to optimize KS. Such approaches have two main limitations. First, they do not explicitly track what knowledge has been used in the conversation nor how topics have shifted during the conversation. Second, MLE often relies on a limited set of example conversations for training, from which it is hard to infer that facts retrieved from the knowledge source can be re-used in multiple conversation contexts, and vice versa.
A Reinforcement Learning Framework for Relevance Feedback
Authors: Ali Montazeralghaem, Hamed Zamani, James Allan 0001
Venue: SIGIR
Year: 2020

We present RML, the first known general reinforcement learning framework for relevance feedback that directly optimizes any desired retrieval metric, including precision-oriented, recall-oriented, and even diversity metrics: RML can be easily extended to directly optimize any arbitrary user satisfaction signal. Using the RML framework, we can select effective feedback terms and weight them appropriately, improving on past methods that fit parameters to feedback algorithms using heuristic approaches or methods that do not directly optimize for retrieval performance. Learning an effective relevance feedback model is not trivial since the true feedback distribution is unknown. Experiments on standard TREC collections compare RML to existing feedback algorithms, demonstrate the effectiveness of RML at optimizing for MAP and α-n DCG, and show the impact on related measures.
Controlling Fairness and Bias in Dynamic Learning-to-Rank
Authors: Marco Morik, Ashudeep Singh, Jessica Hong, Thorsten Joachims
Venue: SIGIR
Year: 2020

Rankings are the primary interface through which many online platforms match users to items (e.g. news, products, music, video). In these two-sided markets, not only the users draw utility from the rankings, but the rankings also determine the utility (e.g. exposure, revenue) for the item providers (e.g. publishers, sellers, artists, studios). It has already been noted that myopically optimizing utility to the users -- as done by virtually all learning-to-rank algorithms -- can be unfair to the item providers. We, therefore, present a learning-to-rank approach for explicitly enforcing merit-based fairness guarantees to groups of items (e.g. articles by the same publisher, tracks by the same artist). In particular, we propose a learning algorithm that ensures notions of amortized group fairness, while simultaneously learning the ranking function from implicit feedback data. The algorithm takes the form of a controller that integrates unbiased estimators for both fairness and utility, dynamically adapting both as more data becomes available. In addition to its rigorous theoretical foundation and convergence guarantees, we find empirically that the algorithm is highly practical and robust.
Read what you need: Controllable Aspect-based Opinion Summarization of Tourist Reviews
Authors: Rajdeep Mukherjee, Hari Chandana Peruri, Uppada Vishnu, Pawan Goyal 0002, Sourangshu Bhattacharya, Niloy Ganguly
Venue: SIGIR
Year: 2020

Manually extracting relevant aspects and opinions from large volumes of user-generated text is a time-consuming process. Summaries, on the other hand, help readers with limited time budgets to quickly consume the key ideas from the data. State-of-the-art approaches for multi-document summarization, however, do not consider user preferences while generating summaries. In this work, we argue the need and propose a solution for generating personalized aspect-based opinion summaries from large collections of online tourist reviews. We let our readers decide and control several attributes of the summary such as the length and specific aspects of interest among others. Specifically, we take an unsupervised approach to extract coherent aspects from tourist reviews posted onTripAdvisor. We then propose an Integer Linear Programming (ILP) based extractive technique to select an informative subset of opinions around the identified aspects while respecting the user-specified values for various control parameters. Finally, we evaluate and compare our summaries using crowdsourcing and ROUGE-based metrics and obtain competitive results.
Combining Contextualized and Non-contextualized Query Translations to Improve CLIR
Authors: Suraj Nair 0001, Petra Galuscáková, Douglas W. Oard
Venue: SIGIR
Year: 2020

In cross-language information retrieval using probabilistic structured queries (PSQ), translation probabilities from statistical machine translation act as a bridge between the query and document vocabulary. These translation probabilities are typically estimated from a sentence-aligned corpus on a word to word basis without taking into account the context. Neural methods, by contrast, can learn to translate using the context around the words, and this can be used as a basis for estimating context-dependent translation probabilities. However, sparsity limits the accuracy of context-specific translation probabilities for rare words, which can be important in retrieval applications. This paper presents evidence that combining such context-dependent translation probabilities with context-independent translation probabilities learned from the same parallel corpus can yield improvements in the effectiveness of cross-language ranked retrieval.
Receptor: A Platform for Exploring Latent Relations in Sensitive Documents
Authors: Hitarth Narvala, Graham McDonald, Iadh Ounis
Venue: SIGIR
Year: 2020

Many government and public organisations have a requirement to release their official documents to the public and therefore need to review such documents to identify and protect any sensitive information that they contain. When reviewing a document for sensitivity, reviewers often use information from other documents within the collection to assist in their decisions. It can be difficult for the reviewers to find related documents in large digital collections when they are performing sensitivity review. Receptor is a new solution that aims to provide sensitivity reviewers with the ability to explore a collection of documents to discover latent relations, between for example entities and events, that can be a reliable indicator of sensitive information. The system provides novel scalable graph search and exploration functionalities as well as interactive visualisations of the latent relations between related entities, events, and documents to enable users to identify hidden patterns of sensitivity.
FactCatch: Incremental Pay-as-You-Go Fact Checking with Minimal User Effort
Authors: Thanh Tam Nguyen, Matthias Weidlich 0001, Hongzhi Yin, Bolong Zheng, Quang Huy Nguyen, Quoc Viet Hung Nguyen
Venue: SIGIR
Year: 2020

The open nature of the Web enables users to produce and propagate any content without authentication, which has been exploited to spread thousands of unverified claims via millions of online documents. Maintenance of credible knowledge bases thus has to rely on fact checking that constructs a trusted set of facts through credibility assessment. Due to an inherent lack of ground truth information and language ambiguity, fact checking cannot be done in a purely automated manner without compromising accuracy. However, state-of-the-art fact checking services, rely mostly on human validation, which is costly, slow, and non-transparent. This paper presents FactCatch, a human-in-the-loop system to guide users in fact checking that aims at minimisation of the invested effort. It supports incremental quality estimation, mistake mitigation, and pay-as-you-go instantiation of a high-quality fact database.
Policy-Aware Unbiased Learning to Rank for Top-k Rankings
Authors: Harrie Oosterhuis, Maarten de Rijke
Venue: SIGIR
Year: 2020

Counterfactual Learning to Rank (LTR) methods optimize ranking systems using logged user interactions that contain interaction biases. Existing methods are only unbiased if users are presented with all relevant items in every ranking. There is currently no existing counterfactual unbiased LTR method for top-k rankings. We introduce a novel policy-aware counterfactual estimator for LTR metrics that can account for the effect of a stochastic logging policy. We prove that the policy-aware estimator is unbiased if every relevant item has a non-zero probability to appear in the top-k ranking. Our experimental results show that the performance of our estimator is not affected by the size of k: for any k, the policy-aware estimator reaches the same retrieval performance while learning from top-k feedback as when learning from feedback on the full ranking. Lastly, we introduce novel extensions of traditional LTR methods to perform counterfactual LTR and to optimize top-k metrics. Together, our contributions introduce the first policy-aware unbiased LTR approach that learns from top-k feedback and optimizes top-k metrics. As a result, counterfactual LTR is now applicable to the very prevalent top-k ranking setting in search and recommendation.
An Eye Tracking Study of Web Search by People With and Without Dyslexia
Authors: Srishti Palani, Adam Fourney, Shane Williams, Kevin Larson, Irina Spiridonova, Meredith Ringel Morris
Venue: SIGIR
Year: 2020

Web search is a key digital literacy skill that can be particularly challenging for people with dyslexia, a common learning disability that affects reading and spelling skills in about 15% of the English-speaking population. In this paper, we collected and analyzed eye-tracking, search log, and self-report data from 27 participants (14 with dyslexia) to confirm that searchers with dyslexia struggle with all stages of the search process and have markedly different gaze patterns and search behavior that reflect the strategies used and challenges faced. Based on these findings, we discuss design implications to improve the cognitive accessibility of web search.
An Intent-guided Collaborative Machine for Session-based Recommendation
Authors: Zhiqiang Pan, Fei Cai, Yanxiang Ling, Maarten de Rijke
Venue: SIGIR
Year: 2020

Session-based recommendation produces item predictions mainly based on anonymous sessions. Previous studies have leveraged collaborative information from neighbor sessions to boost the recommendation accuracy for a given ongoing session. Previous work often selects the most recent sessions as candidate neighbors, thereby failing to identify the most related neighbors to obtain an effective neighbor representation. In addition, few existing methods simultaneously consider the sequential signal and the most recent interest in an ongoing session. In this paper, we introduce an Intent-guided Collaborative Machine for Session-based Recommendation (ICM-SR). ICM-SR encodes an ongoing session by leveraging the prior sequential items and the last item to generate an accurate session representation, which is then used to produce initial item predictions as intent. After that, we design an intent-guided neighbor detector to locate the correct neighbor sessions. Finally, the representations of the current session and the neighbor sessions are adaptively combined by a gated fusion layer to produce the final item recommendations. Experiments conducted on two public benchmark datasets show that ICM-SR achieves a significant improvement in terms of Recall and MRR over the state-of-the-art baselines.
Rethinking Item Importance in Session-based Recommendation
Authors: Zhiqiang Pan, Fei Cai, Yanxiang Ling, Maarten de Rijke
Venue: SIGIR
Year: 2020

Session-based recommendation aims to predict a user's actions at the next timestamp based on anonymous sessions. Previous work mainly focuses on the transition relationship between items that the user interacted with during an ongoing session. They generally fail to pay enough attention to the importance of the items involved in these interactions in terms of their relevance to user's main intent. In this paper, we propose a Session-based Recommendation approach with an Importance Extraction Module, i.e., SR-IEM, that considers both a user's long-term and recent behavior in an ongoing session. We employ a modified self-attention mechanism to estimate item importance in a session, which is then used to predict user's long-term preference. Item recommendations are produced by combining the user's long-term preference and their current interest as conveyed by the last item they interacted with. Comprehensive experiments are conducted on two publicly available benchmark datasets. The proposed SR-IEM model outperforms start-of-the-art baselines in terms of Recall and MRR for the task of session-based recommendation. In addition, compared to state-of-the-art models, SR-IEM has a reduced computational complexity.
Large Scale Abstractive Multi-Review Summarization (LSARS) via Aspect Alignment
Authors: Haojie Pan, Rongqin Yang, Xin Zhou, Rui Wang 0005, Deng Cai 0001, Xiaozhong Liu 0001
Venue: SIGIR
Year: 2020

In an active e-commerce environment, customers process a large number of reviews when deciding on whether to buy a product or not. Abstractive Multi-Review Summarization aims to assist users to efficiently consume the reviews that are the most relevant to them. We propose the first large-scale abstractive multi-review summarization dataset that leverages more than 17.9 billion raw reviews and uses novel aspect-alignment techniques based on aspect annotations. Furthermore, we demonstrate that one can generate higher-quality review summaries by using a novel aspect-alignment-based model. Results from both automatic and human evaluation show that the proposed dataset plus the innovative aspect-alignment model can generate high-quality and trustful review summaries.
Experimaestro and Datamaestro: Experiment and Dataset Managers (for IR)
Authors: Benjamin Piwowarski
Venue: SIGIR
Year: 2020

Abstract Ensuring reproducibility is key to all scientific domains. As Information Retrieval (IR) experiments are often composed of several steps that can be shared between tested models, and rely on various resources, it is difficult to keep track of all the experimental settings and to ensure experiments can be reproduced easily. In this demo paper, we present two managers, Experimaestro and Datamaestro, and their add-ons for IR, designed to help to define and run experimental plans.
MHM: Multi-modal Clinical Data based Hierarchical Multi-label Diagnosis Prediction
Authors: Zhi Qiao, Zhen Zhang, Xian Wu 0001, Shen Ge, Wei Fan 0001
Venue: SIGIR
Year: 2020

Diagnosis prediction aims to forecast diseases that a patient might have in his next hospital visit, which is critical in Clinical Decision Supporting System (CDSS). Existing approaches mainly formulate diagnosis prediction as a multi-label classification problem and use discrete medical codes as major features. While the structural information among medical codes and time series data in clinical data are generally neglected. In this paper, we propose Multi-modal Clinical Data based Hierarchical Multi-label model (MHM) to integrate discrete medical codes, structural information and time series data into the same framework for diagnosis prediction task. Experimental results on two real world datasets demonstrate the superiority of proposed MHM over state-of-the-art approaches.
User Behavior Retrieval for Click-Through Rate Prediction
Authors: Jiarui Qin, Weinan Zhang 0001, Xin Wu, Jiarui Jin, Yuchen Fang, Yong Yu 0001
Venue: SIGIR
Year: 2020

Click-through rate (CTR) prediction plays a key role in modern online personalization services. In practice, it is necessary to capture user's drifting interests by modeling sequential user behaviors to build an accurate CTR prediction model. However, as the users accumulate more and more behavioral data on the platforms, it becomes non-trivial for the sequential models to make use of the whole behavior history of each user. First, directly feeding the long behavior sequence will make online inference time and system load infeasible. Second, there is much noise in such long histories to fail the sequential model learning. The current industrial solutions mainly truncate the sequences and just feed recent behaviors to the prediction model, which leads to a problem that sequential patterns such as periodicity or long-term dependency are not embedded in the recent several behaviors but in far back history. To tackle these issues, in this paper we consider it from the data perspective instead of just designing more sophisticated yet complicated models and propose User Behavior Retrieval for CTR prediction (UBR4CTR) framework. In UBR4CTR, the most relevant and appropriate user behaviors will be firstly retrieved from the entire user history sequence using a learnable search method. These retrieved behaviors are then fed into a deep model to make the final prediction instead of simply using the most recent ones. It is highly feasible to deploy UBR4CTR into industrial model pipeline with low cost. Experiments on three real-world large-scale datasets demonstrate the superiority and efficacy of our proposed framework and models.
GAG: Global Attributed Graph Neural Network for Streaming Session-based Recommendation
Authors: Ruihong Qiu, Hongzhi Yin, Zi Huang, Tong Chen 0005
Venue: SIGIR
Year: 2020

Streaming session-based recommendation (SSR) is a challenging task that requires the recommender system to do the session-based recommendation (SR) in the streaming scenario. In the real-world applications of e-commerce and social media, a sequence of user-item interactions generated within a certain period are grouped as a session, and these sessions consecutively arrive in the form of streams. Most of the recent SR research has focused on the static setting where the training data is first acquired and then used to train a session-based recommender model. They need several epochs of training over the whole dataset, which is infeasible in the streaming setting. Besides, they can hardly well capture long-term user interests because of the neglect or the simple usage of the user information. Although some streaming recommendation strategies have been proposed recently, they are designed for streams of individual interactions rather than streams of sessions. In this paper, we propose a G lobal A ttributed G raph (GAG) neural network model with a Wasserstein reservoir for the SSR problem. On one hand, when a new session arrives, a session graph with a global attribute is constructed based on the current session and its associate user. Thus, the GAG can take both the global attribute and the current session into consideration to learn more comprehensive representations of the session and the user, yielding a better performance in the recommendation. On the other hand, for the adaptation to the streaming session scenario, a Wasserstein reservoir is proposed to help preserve a representative sketch of the historical data. Extensive experiments on two real-world datasets have been conducted to verify the superiority of the GAG model compared with the state-of-the-art methods.