OpenRaiser/Intern-Atlas huggingface dataset:
Towards Trustworthiness in the Context of Explainable Search
Authors: Sayantan Polley, Rashmi Raju Koparde, Akshaya Bindu Gowri, Maneendra Perera, Andreas Nürnberger
Venue: SIGIR
Year: 2021

Explainable AI (XAI) is currently a vibrant research topic. However, the absence of ground truth explanations makes it difficult to evaluate XAI systems such as Explainable Search. We present an Explainable Search system with a focus on evaluating the XAI aspect of Trustworthiness along with the retrieval performance. We present SIMFIC 2.0 (Similarity in Fiction), an enhanced version of a recent explainable search system. The system retrieves books similar to a selected book in a query-by-example setting. The motivation is to explain the notion of similarity in fiction books. We extract hand-crafted interpretable features for fiction books and provide global explanations by fitting a linear regression and local explanations based on similarity measures. The Trustworthiness facet is evaluated using user studies, while the ranking performance is compared by analysis of user clicks. Eye tracking is used to investigate user attention to the explanation elements when interacting with the interface. Initial experiments show statistically significant results on the Trustworthiness of the system, paving way for interesting research directions that are being investigated.
The Information Retrieval Anthology
Authors: Martin Potthast, Sebastian Günther 0002, Janek Bevendorff, Jan Philipp Bittner, Alexander Bondarenko 0001, Maik Fröbe, Christian Kahmann, Andreas Niekler, Michael Völske, Benno Stein 0001, Matthias Hagen
Venue: SIGIR
Year: 2021

We present the IR Anthology, a corpus of information retrieval publications accessible via a metadata browser and a full-text search engine. Following the example of the well-known ACL Anthology, the IR Anthology serves as a hub for researchers interested in information retrieval. Our search engine ChatNoir indexes the publications' full texts, enabling a focused search and linking users to the respective publisher's site for personal access. Listing more than 40,000 publications at the time of writing, the IR Anthology can be freely accessed at https://IR.webis.de.
Learning Robust Dense Retrieval Models from Incomplete Relevance Labels
Authors: Prafull Prakash, Julian Killingback, Hamed Zamani
Venue: SIGIR
Year: 2021

Recent deployment of efficient billion-scale approximate nearest neighbor (ANN) search algorithms on GPUs has motivated information retrieval researchers to develop neural ranking models that learn low-dimensional dense representations for queries and documents and use ANN search for retrieval. However, optimizing these dense retrieval models poses several challenges including negative sampling for (pair-wise) training. A recent model, called ANCE, successfully uses dynamic negative sampling using ANN search. This paper improves upon ANCE by proposing a robust negative sampling strategy for scenarios where the training data lacks complete relevance annotations. This is of particular importance as obtaining large-scale training data with complete relevance judgment is extremely expensive. Our model uses a small validation set with complete relevance judgments to accurately estimate a negative sampling distribution for dense retrieval models. We also explore leveraging a lexical matching signal during training and pseudo-relevance feedback during evaluation for improved performance. Our experiments on the TREC Deep Learning Track benchmarks demonstrate the effectiveness of our solutions.
Hierarchical Multi-modal Contextual Attention Network for Fake News Detection
Authors: Shengsheng Qian, Jinguang Wang, Jun Hu 0016, Quan Fang, Changsheng Xu
Venue: SIGIR
Year: 2021

Nowadays, detecting fake news on social media platforms has become a top priority since the widespread dissemination of fake news may mislead readers and have negative effects. To date, many algorithms have been proposed to facilitate the detection of fake news from the hand-crafted feature extraction methods to deep learning approaches. However, these methods may suffer from the following limitations: (1) fail to utilize the multi-modal context information and extract high-order complementary information for each news to enhance the detection of fake news; (2) largely ignore the full hierarchical semantics of textual content to assist in learning a better news representation. To overcome these limitations, this paper proposes a novel hierarchical multi-modal contextual attention network (HMCAN) for fake news detection by jointly modeling the multi-modal context information and the hierarchical semantics of text in a unified deep model. Specifically, we employ BERT and ResNet to learn better representations for text and images, respectively. Then, we feed the obtained representations of images and text into a multi-modal contextual attention network to fuse both inter-modality and intra-modality relationships. Finally, we design a hierarchical encoding network to capture the rich hierarchical semantics for fake news detection. Extensive experiments on three public real datasets demonstrate that our proposed HMCAN achieves state-of-the-art performance.
PECAN: A Platform for Searching Chat Conversations
Authors: Kunpeng Qin, Harrisen Scells, Guido Zuccon
Venue: SIGIR
Year: 2021

Often, existing chat services that organisations and individuals use today provide a way to search through previously sent messages. However, many of these chat services provide far-limited search functionalities, typically exact matching on individual messages. In this paper, we introduce a new task for addressing this problem, called searching for conversations, whereby the aim is to retrieve and rank groups of related messages given a search query. We promote this task by providing a platform for research and development called PECAN. Our platform provides all the necessary functionality researchers need to conduct experiments on searching for conversations. Our system is also generic so as to support organisations and individuals who wish to search through their chat message archives.
Retrieving Skill-Based Teams from Collaboration Networks
Authors: Radin Hamidi Rad, Ebrahim Bagheri, Mehdi Kargar, Divesh Srivastava, Jaroslaw Szlichta
Venue: SIGIR
Year: 2021

Given a set of required skills, the objective of the team formation problem is to form a team of experts that cover the required skills. Most existing approaches are based on graph methods, such as minimum-cost spanning trees. These approaches, due to their limited view of the network, fail to capture complex interactions among experts and are computationally intractable. More recent approaches adopt neural architectures to learn a mapping between the skills and experts space. While they are more effective, these techniques face two main limitations: (1) they consider a fixed representation for both skills and experts, and (2) they overlook the significant amount of past collaboration network information. We learn dense representations for skills and experts based on previous collaborations and bootstrap the training process through transfer learning. We also propose to fine-tune the representation of skills and experts while learning the mapping function. Our experiments over the DBLP dataset verify that our proposed architecture is able to outperform the state-of-the-art graph and neural methods over both ranking and quality metrics.
Improving Neural Text Style Transfer by Introducing Loss Function Sequentiality
Authors: Chinmay Rane, Gaël Dias, Alexis Lechervy, Asif Ekbal
Venue: SIGIR
Year: 2021

Text style transfer is an important issue for conversational agents as it may adapt utterance production to specific dialogue situations. It consists in introducing a given style within a sentence while preserving its semantics. Within this scope, different strategies have been proposed that either rely on parallel data or take advantage of non-supervised techniques. In this paper, we follow the latter approach and show that the sequential introduction of different loss functions into the learning process can boost the performance of a standard model. We also evidence that combining different style classifiers that either focus on global or local textual information improves sentence generation. Experiments on the Yelp dataset show that our methodology strongly competes with the current state-of-the-art models across style accuracy, grammatical correctness, and content preservation.
Learning with Little Data: Industry Challenges and Innovations
Authors: Nikhil Rao 0001
Venue: SIGIR
Year: 2021

In e-commerce applications, customers search and discover one or more products using queries. Some of these queries are broad and diverse, with multiple intents. Therefore, relying purely on the anonymized and aggregated customer historical behavioral data is not sufficient to train machine learned models. For example, customers may click and purchase a galaxy charger for a "samsung galaxy s9" query. The item is not an exact match for the customer query. However, it serves as a complement to the original query and may be purchased. To address these potential mismatches from surfacing in search results, e-commerce systems rely on machine learned models trained on human- annotated data. There are two challenges in collecting human annotated data. First, the human annotation process does not scale and it is hard to obtain large volumes of annotations in multiple languages. Second, annotators must query existing systems to obtain samples for auditing, resulting in very few mismatched examples (data skewness) and counterfactual biases. In this talk, we address these challenges using two recent advances in deep learning. To address the data skewness, we generate hard negative examples using positive examples. The key idea here is to generate synthetic data using a Variational Encoder Decoder (VED) architecture. We show how a modified loss function with a novel combiner (to combine VED with the classifier) can avoid policy-based gradients and other heuristics. To address the sparsity of data in less popular languages, we combine data across all languages using language-agnostic representation learning. The side information we use aligns the items across languages in the same latent space. We show that our approaches significantly improve upon state of the art baselines, by over 25% in F1 score for the variational model, and over 20% in F1 score for the multilingual model.
Evaluating the Predictivity of IR Experiments
Authors: Lida Rashidi, Justin Zobel, Alistair Moffat
Venue: SIGIR
Year: 2021

Experimental evaluation is regarded as a critical element of any research activity in Information Retrieval, and is typically used to support assertions of the form "Technique A provides better retrieval effectiveness than does Technique B". Implicit in such claims are the characteristics of the data to which the results apply, in terms of both the queries used and the documents they were applied to. Here we explore the role of evaluation on a collection as a prediction of relative performance on collections that have different characteristics. In particular, by synthesizing new collections that vary from each other in a controlled way, we show that it is possible to explore the reliability of an IR evaluation pipeline, and to better understand the complex interrelationship between documents, queries, and metrics that is an important part of any experimental validation. Our results show that predictivity declines as the collection is varied, even in simple ways such as shifting in focus from one document source to another similar source.
Predicting User Demography and Device from News Comments
Authors: Ohad Rozen, Joel Oren, Ariel Raviv
Venue: SIGIR
Year: 2021

Demographics of online users such as age and gender play an important role in personalized web applications, particularly in the News domain. However, it is difficult to directly obtain the demographic information of online users. Past works have attempted to predict user demography based on reading patterns obtained from news browsing data. However, such data can be very limited. Luckily, in recent years, posts and comments have become much prevalent among online users, and the comments from users of different demographics exhibit differences in contents and writing styles. Thus, comments can provide additional clues for demographic prediction. In this paper, we study predicting users' demographics based on both news browsing data and the associated user generated comments. To this end, we make a novel use of a recently introduced BERT-based model to embed each comment in the context of its associated article. We experiment on real-world datasets, and explore the contribution of both browsing data and user generated data in the task of predicting three different user attributes: gender, location type (e.g., rural vs. urban), and mobile device. Finally we show that our approach can effectively improve the performance of such predictions and outperforms baseline methods.
On the Two-Sample Randomisation Test for IR Evaluation
Authors: Tetsuya Sakai
Venue: SIGIR
Year: 2021

While previous work in comparing statistical significance tests for IR system evaluation have focused on paired data tests (e.g., for evaluating two systems using a common test collection), two-sample tests must be used when the reproducibility of IR experiments across different test collections must be examined. Using real runs and a test collection from the NTCIR-15 WWW-3 Task, the present study compares the properties of three two-sample significance tests for comparing two systems: Student's t-test (i.e., the classical parametric test), the Wilcoxon rank sum test (i.e., the classical nonparametric test), and the randomisation test (i.e., a population-free method that utilises modern computational power). In terms of the false positive rate (i.e., the chance of detecting a statistical significance even though the two samples of evaluation measure scores come from the same system), the three tests behave similarly, although the Wilcoxon rank sum test appears to be slightly more robust than the other two for very small topic set sizes (e.g., 10 topics each) with a large significance level (e.g., α=0.10). On the other hand, the t-test and the Wilcoxon rank sum test are very similar to each other from the following two viewpoints: "How often do they both detect a nonexistent difference?" and "How often do they both overlook a true difference?" Compared to the two classical significance tests, the randomisation test behaves markedly differently in terms of the above two viewpoints. Hence, we suggest that researchers should at least be aware of the above properties of the three two-sample tests when choosing from them.
WWW3E8: 259, 000 Relevance Labels for Studying the Effect of Document Presentation Order for Relevance Assessors
Authors: Tetsuya Sakai, Sijie Tao, Zhaohao Zeng
Venue: SIGIR
Year: 2021

In IR evaluation based on depth-k pooling, there are several strategies to order the pooled documents for relevance assessors. Among them, the simplest approach is to completely randomise the order "so assessors cannot tell if a document was highly ranked by some system or how many systems (or which systems) retrieved the document." An approach that is in sharp contrast to the above is the prioritisation approach taken by NTCIRPOOL, a tool widely used at NTCIR. NTCIRPOOL sorts the pooled documents by "pseudorelevance," a statistic that reflects the popularity of each document within the depth-k pools. Although these two strategies have coexisted for over two decades, the IR research community has yet to reach a consensus as to what advantages each of these two strategies actually offer. To help researchers directly address this question using their favourite methods of analysis, we have released a large-scale data set called WWW3E8. It comprises eight independent sets of qrels for the 160 English topics of the NTCIR-15 WWW-3 task: four qrels files constructed using the randomisation approach, and another four constructed using the prioritisation approach of NTCIRPOOL. Each qrels file covers 32,375 topic-document pairs; hence, WWW3E8 contains a total of 259,000 relevance labels. Moreover, the data set contains the raw English subtask run files from the WWW-3 task, the randomised and prioritised pool files, and topic-by-run score matrices of the official measures used in the task. Hence, researchers interested in the above research question regarding document ordering can utilise WWW3E8 as a common ground to directly compare the two strategies.
Transfer Learning for the Multilingual and Multi-Domain Classification of Messages Relating to Crises
Authors: Cinthia Sánchez
Venue: SIGIR
Year: 2021

Social media plays an important role as a source of information during crisis events. It allows for more rapid dissemination of critical information than traditional news media, as its users can provide immediate information from the locations where events are unfolding. Several studies have addressed the automatic detection of crisis-related messages to contribute to disaster management and humanitarian assistance. However, most of them have focused on a particular language (usually English) or type of event, which limits their applicability to other contexts. The lack of labeled data in different languages and types of disasters poses a major obstacle to the application of supervised learning-based approaches to more diverse scenarios. To address this problem, this research aims to characterize messages related to diverse crisis domains in a language-agnostic manner in order to construct multilingual crisis detectors. To achieve this, we propose a comprehensive evaluation of transfer learning performance in terms of crisis domain and language, comparing different data representations and classification techniques.
Utility of Missing Concepts in Query-biased Summarization
Authors: Sheikh Muhammad Sarwar, Felipe Moraes, Jiepu Jiang, James Allan 0001
Venue: SIGIR
Year: 2021

Query-biased Summarization (QBS) aims to produce a query-dependent summary of a retrieved document to reduce the human effort for inspecting the full-text content. Typical summarization approaches extract document snippets that overlap with the query and show them to searchers. Such QBS methods show relevant information in a document but do not inform searchers what is missing. Our study focuses on reducing user effort in finding relevant documents by exposing the information in the query that is missing in the retrieved results. We use a classical approach, DSPApprox, to find terms or phrases relevant to a query. Then, we identify which terms or phrases are missing in a document, present them in a search interface, and ask crowd workers to judge document relevance based on snippets and missing information. Experimental results show both benefits and limitations of our method compared with traditional ones that only show relevant snippets.
Hyperbolic Online Time Stream Modeling
Authors: Ramit Sawhney, Shivam Agarwal, Megh Thakkar, Arnav Wadhwa, Rajiv Ratn Shah
Venue: SIGIR
Year: 2021

The rapidly rising ubiquity and dissemination of online information such as social media text and news improve user accessibility towards financial markets, however, modeling these vast streams of irregular, temporal data poses a challenge. Such temporal streams of information show power-law dynamics, scale-free characteristics, and time irregularities that sequential models are unable to accurately model. In this work, we propose the first Hierarchical Time-Aware Hyperbolic LSTM (HTLSTM), which leverages the Riemannian manifold for encoding the scale-free nature of a sequence of text in a time-aware fashion. Through experiments on three financial tasks: stock trading, equity price movement prediction, and financial risk prediction, we demonstrate HTLSTM's applicability for modeling temporal sequences of online information. On real-world data from four global stock markets and three stock indices spanning data in English and Chinese, we make a step towards time-aware text modeling via hyperbolic geometry.
Seer-Dock: A General-Purpose Dockerized Scholarly Document Collection and Management Framework
Authors: Dina Sayed, Mohamed Nour, Heiko Schuldt
Venue: SIGIR
Year: 2021

The harvesting, management, and analysis of thematic document collections is a major challenge in a wide variety of applications. While the criteria for compiling such collections are individual, the entire process is largely standardized. Therefore, it is not efficient to build new systems over and over again to take over these tasks. In this work, we introduce Seer-Dock, a novel and easy-to deploy general-purpose dockerized framework to build a scholarly document harvesting and management system. It is based on CiteSeerX, the most widely used scholarly search engine. Seer-Dock uses docker containers for all components and thus enables its users to rapidly deploy a full-fledged document collection and management system on any operating system platform and tailor it to the specific needs of an application domain. Moreover, it is easy to scale, orchestrate, maintain, and recover. In this resource paper, we introduce the architecture of Seer-Dock and its components. Like its kernel CiteSeerX, Seer-Dock is available under an Apache 2 open source license.
AI Based Information Retrieval System for Identifying Harmful Online Gaming Patterns
Authors: Deepanshi Seth, Rukma Talwadker, Tridib Mukherjee, Usama Chitapure, Nagesh Adiga, Avantika Gupta
Venue: SIGIR
Year: 2021

Games of skill are an excellent source of recreation and relaxation. Games are also the safest and readily accessible constructs for social interaction and community affairs which potentially opens up new avenues for realising personal worth, social acceptance, respect & recognition. However, when these games are played with real money, ensuring game prudence, whereby users play real-money skill games only for entertainment purposes, and do so well within their resourceful means, becomes necessary. It becomes paramount for the wellness of players and also to ensure online gaming is only available for sheer entertainment. In this proposal, we present an automated, data driven, AI powered, Responsible Game Play (RGP) framework cum tool which has been integrated in our online skill gaming platform. RGP pipeline is a combination of: a) a couple of anomaly detection Rule Based Engines; b) a Deep Learning Pipeline which models the game play characteristics of healthy and engaged players to identify potentially risky players, and c) a ML based Local Expert which leverages users' longitudinal behavioral patterns and constructs new features using the adjacent AI OPS and Signal Processing Domains. We integrate the psychometric assessment to nudge and coarse correct at-risk players proactively, ahead of time
DarkJargon.net: A Platform for Understanding Underground Conversation with Latent Meaning
Authors: Dominic Seyler, Wei Liu, Yunan Zhang 0001, XiaoFeng Wang 0001, ChengXiang Zhai
Venue: SIGIR
Year: 2021

An essential part of underground conversation are dark jargon terms. They are benign-looking, but have hidden, sometimes sinister meanings and are used by participants of underground forums for illicit behavior. For example, the dark term "rat" is often used in lieu of "Remote Access Trojan". We present a novel online platform that caters to the understating of underground conversation with latent meaning. Our system enables researchers, law enforcement agents and "white-hat" hackers to gain invaluable insights into underground communication by providing them with a tool to (1) look-up dark jargon terms in a dictionary; (2) explore the usage of dark jargon over time and interpret their meaning; (3) collaborate and contribute their own research findings. Furthermore, we introduce a novel dark jargon interpretation method that leverages masked language modeling of a transformer-based architecture.