OpenRaiser/Intern-Atlas huggingface dataset:
WCIS 2019: 1st Workshop on Conversational Interaction Systems
Authors: Abhinav Rastogi, Alexandros Papangelis, Rahul Goel, Chandra Khatri
Venue: SIGIR
Year: 2019

The first workshop on Conversational Interaction Systems is held in Paris, France on July 25th, 2019, co-located with the ACM Special Interest Group on Information Retrieval (SIGIR). The goal of the workshop is to bring together researchers from academia and industry to discuss the challenges and future of conversational agents and interactive systems. The workshop has an exciting program that spans a number of subareas including: multi-modal conversational interfaces, dialogue accessibility, and scaling such systems. The program includes eight invited talks, a lively panel discussion on emerging topics, and presentation of original research papers.
On Tradeoffs Between Document Signature Methods for a Legal Due Diligence Corpus
Authors: Adam Roegiest, Edward Lee
Venue: SIGIR
Year: 2019

While document signatures are a well established tool in IR, they have primarily been investigated in the context of web documents. Legal due diligence documents, by their nature, have more similar structure and language than we may expect out of standard web collections. Moreover, many due diligence systems strive to facilitate real-time interactions and so time from document ingestion to availability should be minimal. Such constraints further limit the possible solution space when identifying near duplicate documents. We present an examination of the tradeoffs that document signature methods face in the due diligence domain. In particular, we quantify the trade-off between signature length, time to compute, number of hash collisions, and number of nearest neighbours for a 90,000 document due diligence corpus.
Normalized Query Commitment Revisited
Authors: Haggai Roitman
Venue: SIGIR
Year: 2019

We revisit the Normalized Query Commitment (NQC) query performance prediction (QPP) method. To this end, we suggest a scaled extension to a discriminative QPP framework and use it to analyze NQC. Using this analysis allows us to redesign NQC and suggest several options for improvement.
An Open-Access Platform for Transparent and Reproducible Structured Searching
Authors: Tony Russell-Rose, Jon Chamberlain
Venue: SIGIR
Year: 2019

Knowledge workers such as patent agents, recruiters and legal researchers undertake work tasks in which search forms a core part of their duties. In these instances, the search task often involves formulation of complex queries expressed as Boolean strings. However, creating effective Boolean queries remains an ongoing challenge, often compromised by errors and inefficiencies. In this paper, we demonstrate a new approach to structured searching in which concepts are expressed as objects on a two-dimensional canvas. Interactive query suggestions are provided via an NLP services API, and support is offered for optimising, translating and sharing search strategies as executable artefacts. This eliminates many sources of error, makes the query semantics more transparent, and offers an open-access platform for sharing reproducible search strategies and best practices.
Which Diversity Evaluation Measures Are "Good"?
Authors: Tetsuya Sakai, Zhaohao Zeng
Venue: SIGIR
Year: 2019

This study evaluates 30 IR evaluation measures or their instances, of which nine are for adhoc IR and 21 are for diversified IR, primarily from the viewpoint of whether their preferences of one SERP (search engine result page) over another actually align with users' preferences. The gold preferences were contructed by hiring 15 assessors, who independently examined 1,127 SERP pairs and made preference assessments. Two sets of preference assessments were obtained: one based on a relevance question "Which SERP is more relevant?'' and the other based on a diversity question "Which SERP is likely to satisfy a higher number of users?'' To our knowledge, our study is the first to have collected diversity preference assessments in this way and evaluated diversity measures successfully. Our main results are that (a) Popular adhoc IR measures such as nDCG actually align quite well with the gold relevance preferences; and that (b) While the ♯-measures align well with the gold diversity preferences, intent-aware measures perform relatively poorly. Moreover, as by-products of our analysis of existing evaluation measures, we define new adhoc measures called iRBU (intentwise Rank-Biased Utility) and EBR (Expected Blended Ratio); we demonstrate that an instance of iRBU performs as well as nDCG when compared to the gold relevance preferences. On the other hand, the original RBU, a recently-proposed diversity measure, underperforms the best ♯-measures when compared to the gold diversity preferences.
Jointly Modeling Relevance and Sensitivity for Search Among Sensitive Content
Authors: Mahmoud F. Sayed, Douglas W. Oard
Venue: SIGIR
Year: 2019

Current search engines are designed to find what we want. But unprocessed archival collections can't be made available for search if they contain sensitive content that needs to be protected. Traditionally, content if first examined through a sensitivity review process, which becomes more difficult and time-consuming as content volumes increase. To mitigate these costs and delays, search technology should be capable of providing access to relevant content while protecting sensitive content. This paper proposes an approach that leverages learning to rank techniques. We use learning to rank to optimize a loss function that balances the value of finding relevant content with the imperative to protect sensitive content. In the experiments, a LETOR benchmark dataset, OHSUMED, is used with a subset of the MeSH labels representing the sensitive documents. Results show the efficacy of the proposed approach in comparison with some simpler baselines.
Automatic Understanding of the Visual World
Authors: Cordelia Schmid
Venue: SIGIR
Year: 2019

One of the central problems of artificial intelligence is machine perception, i.e., the ability to understand the visual world based on input from sensors such as cameras. In this talk, I will present recent progress of my team in this direction. I will start with presenting results on how to generate additional training data using weak annotations, motion information and synthetic data. Next, I will discuss our results for action recognition in videos, where human tubelets have shown to be successful. Our tubelet approach moves away from state-of-the-art frame based approaches and improves classification and localization by relying on joint information from several frames. We show how to extend this type of method to weakly supervised learning of actions, which allows us to scale to large amounts of data with sparse manual annotation. Finally, I will present recent work on grasping with a robot arm based on learning long-horizon manipulations with a hierarchy of RL and imitation-based skills.
Length-adaptive Neural Network for Answer Selection
Authors: Taihua Shao, Fei Cai, Honghui Chen, Maarten de Rijke
Venue: SIGIR
Year: 2019

Answer selection focuses on selecting the correct answer for a question. Most previous work on answer selection achieves good performance by employing an RNN, which processes all question and answer sentences with the same feature extractor regardless of the sentence length. These methods often encounter the problem of long-term dependencies. To address this issue, we propose a Length-adaptive Neural Network (LaNN) for answer selection that can auto-select a neural feature extractor according to the length of the input sentence. In particular, we propose a flexible neural structure that applies a BiLSTM-based feature extractor for short sentences and a Transformer-based feature extractor for long sentences. To the best of our knowledge, LaNN is the first neural network structure that can auto-select the feature extraction mechanism based on the input. We quantify the improvements of LaNN against several competitive baselines on the public WikiQA dataset, showing significant improvements over the state-of-the-art.
Privacy-aware Document Ranking with Neural Signals
Authors: Jinjin Shao, Shiyu Ji, Tao Yang
Venue: SIGIR
Year: 2019

The recent work on neural ranking has achieved solid relevance improvement, by exploring similarities between documents and queries using word embeddings. It is an open problem how to leverage such an advancement for privacy-aware ranking, which is important for top K document search on the cloud. Since neural ranking adds more complexity in score computation, it is difficult to prevent the server from discovering embedding-based semantic features and inferring privacy-sensitive information. This paper analyzes the critical leakages in interaction-based neural ranking and studies countermeasures to mitigate such a leakage. It proposes a privacy-aware neural ranking scheme that integrates tree ensembles with kernel value obfuscation and a soft match map based on adaptively-clustered term closures. The paper also presents an evaluation with two TREC datasets on the relevance of the proposed techniques and the trade-offs for privacy and storage efficiency.
Towards Context-Aware Evaluation for Image Search
Authors: Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Min Zhang, Shaoping Ma
Venue: SIGIR
Year: 2019

Compared to general web search, image search engines present results in a significantly different way, which leads to changes in user behavior patterns, and thus creates challenges for the existing evaluation mechanisms. In this paper, we pay attention to the context factor in the image search scenario. On the basis of a mean-variance analysis, we investigate the effects of context and find that evaluation metrics align with user satisfaction better when the returned image results have high variance. Furthermore, assuming that the image results a user has examined might affect her following judgments, we propose the Context-Aware Gain (CAG), a novel evaluation metric that incorporates the contextual effects within the well-known gain-discount framework. Our experiment results show that, with a proper combination of discount functions, the proposed context-aware evaluation metric can significantly improve the performances of offline metrics for image search evaluation, considering user satisfaction as the golden standard.
Supervised Hierarchical Cross-Modal Hashing
Authors: Changchang Sun, Xuemeng Song, Fuli Feng, Wayne Xin Zhao, Hao Zhang, Liqiang Nie
Venue: SIGIR
Year: 2019

Recently, due to the unprecedented growth of multimedia data, cross-modal hashing has gained increasing attention for the efficient cross-media retrieval. Typically, existing methods on cross-modal hashing treat labels of one instance independently but overlook the correlations among labels. Indeed, in many real-world scenarios, like the online fashion domain, instances (items) are labeled with a set of categories correlated by certain hierarchy. In this paper, we propose a new end-to-end solution for supervised cross-modal hashing, named HiCHNet, which explicitly exploits the hierarchical labels of instances. In particular, by the pre-established label hierarchy, we comprehensively characterize each modality of the instance with a set of layer-wise hash representations. In essence, hash codes are encouraged to not only preserve the layer-wise semantic similarities encoded by the label hierarchy, but also retain the hierarchical discriminative capabilities. Due to the lack of benchmark datasets, apart from adapting the existing dataset FashionVC from fashion domain, we create a dataset from the online fashion platform Ssense consisting of 15,696 image-text pairs labeled by 32 hierarchical categories. Extensive experiments on two real-world datasets demonstrate the superiority of our model over the state-of-the-art methods.
DivGraphPointer: A Graph Pointer Network for Extracting Diverse Keyphrases
Authors: Zhiqing Sun, Jian Tang, Pan Du, Zhi-Hong Deng, Jian-Yun Nie
Venue: SIGIR
Year: 2019

Keyphrase extraction from documents is useful to a variety of applications such as information retrieval and document summarization. This paper presents an end-to-end method called DivGraphPointer for extracting a set of diversified keyphrases from a document. DivGraphPointer combines the advantages of traditional graph-based ranking methods and recent neural network-based approaches. Specifically, given a document, a word graph is constructed from the document based on word proximity and is encoded with graph convolutional networks, which effectively capture document-level word salience by modeling long-range dependency between words in the document and aggregating multiple appearances of identical words into one node. Furthermore, we propose a diversified point network to generate a set of diverse keyphrases out of the word graph in the decoding process. Experimental results on five benchmark data sets show that our proposed method significantly outperforms the existing state-of-the-art approaches.
Event Tracker: A Text Analytics Platform for Use During Disasters
Authors: Charles Thomas, Richard McCreadie, Iadh Ounis
Venue: SIGIR
Year: 2019

Emergency management organisations currently rely on a wide range of disparate tools and technologies to support the monitoring and management of events during crisis situations. This has a number of disadvantages, in terms of training time for new staff members, reliance on external services, and a lack of integration (and hence poor transfer of information) between those services. On the other hand, Event Tracker is a new solution that aims to provide a unified view of an event, integrating information from emergency response officers, the public (via social media) and also volunteers from around the world. In particular, Event Tracker provides a series of novel functionalities to realise this unified view of the event, namely: real-time identification of critical information, automatic grouping of content by the information needs of response officers, as well as real-time volunteers management and communication. This is supported by an efficient and scalable back-end infrastructure designed to ingest and process high-volumes of real-time streaming data with low latency.
Improving Collaborative Metric Learning with Efficient Negative Sampling
Authors: Viet-Anh Tran, Romain Hennequin, Jimena Royo-Letelier, Manuel Moussallam
Venue: SIGIR
Year: 2019

Distance metric learning based on triplet loss has been applied with success in a wide range of applications such as face recognition, image retrieval, speaker change detection and recently recommendation with the Collaborative Metric Learning (CML) model. However, as we show in this article, CML requires large batches to work reasonably well because of a too simplistic uniform negative sampling strategy for selecting triplets. Due to memory limitations, this makes it difficult to scale in high-dimensional scenarios. To alleviate this problem, we propose here a 2-stage negative sampling strategy which finds triplets that are highly informative for learning. Our strategy allows CML to work effectively in terms of accuracy and popularity bias, even when the batch size is an order of magnitude smaller than what would be needed with the default uniform sampling. We demonstrate the suitability of the proposed strategy for recommendation and exhibit consistent positive results across various datasets.
ABCPRec: Adaptively Bridging Consumer and Producer Roles for User-Generated Content Recommendation
Authors: Kosetsu Tsukuda, Satoru Fukayama, Masataka Goto
Venue: SIGIR
Year: 2019

In Web services dealing with user-generated content (UGC), a user can have two roles: a role of a consumer and that of a producer. Since most item recommendation models have only considered the role of a user as a consumer, how to leverage the two roles to improve UGC recommendation accuracy has been underexplored. In this paper, based on the state-of-the-art UGC recommendation method called CPRec (consumer and producer based recommendation), we propose ABCPRec (adaptively bridging CPRec). Unlike CPRec, which assumes that the two roles of a user are always related to each other, ABCPRec adaptively bridges the two roles according to the similarity between her nature as a consumer and that as a producer. This enables the model to learn each user's characteristics as both a consumer and a producer and to recommend items to each user more accurately. By using two real-world datasets, we showed that our proposed method significantly outperformed comparative methods in terms of AUC.
Parrot: A Python-based Interactive Platform for Information Retrieval Research
Authors: Xinhui Tu, Jimmy X. Huang, Jing Luo, Runjie Zhu, Tingting He
Venue: SIGIR
Year: 2019

Open source softwares play an important role in information retrieval research. Most of the existing open source information retrieval systems are implemented in Java or C++ programming language. In this paper, we propose Parrot1, a Python-based interactive platform for information retrieval research. The proposed platform has mainly three advantages in comparison with the existing retrieval systems: (1) It is integrated with Jupyter Notebook, an interactive programming platform which has proved to be effective for data scientists to tackle big data and AI problems. As a result, users can interactively visualize and diagnose a retrieval model; (2) As an application written in Python, it can be easily used in combination with the popular deep learning frameworks such as Tersorflow and Pytorch; (3) It is designed especially for researchers. Less code is needed to create a new retrieval model or to modify an existing one. Our efforts have focused on three functionalists: good usability, interactive programming, and good interoperability with the popular deep learning frameworks. To confirm the performance of the proposed system, we conduct comparative experiments on a number of standard test collections. The experimental results show that the proposed system is both efficient and effective, providing a practical framework for researchers in information retrieval.
Challenges and Opportunities in Understanding Spoken Queries Directed at Modern Entertainment Platforms
Authors: Ferhan Türe, Jinfeng Rao, Raphael Tang, Jimmy Lin
Venue: SIGIR
Year: 2019

Modern in-home entertainment platforms---representing the evolution of the humble television of yesteryear---are packed with features and content: they offer a dizzying array of programs spanning hundreds of channels as well as a catalog of on-demand programs offering tens of thousands of options. Furthermore, the entertainment platform may serve as an in-home hub, providing capabilities ranging from playing music to controlling the home security system. At a high level, our goal is to provide natural speech-based access to these myriad features as an alternative to physical button entry on a remote control.
A New Perspective on Score Standardization
Authors: Julián Urbano, Harlley Lima, Alan Hanjalic
Venue: SIGIR
Year: 2019

In test collection based evaluation of IR systems, score standardization has been proposed to compare systems across collections and minimize the effect of outlier runs on specific topics. The underlying idea is to account for the difficulty of topics, so that systems are scored relative to it. Webber et al. first proposed standardization through a non-linear transformation with the standard normal distribution, and recently Sakai proposed a simple linear transformation. In this paper, we show that both approaches are actually special cases of a simple standardization which assumes specific distributions for the per-topic scores. From this viewpoint, we argue that a transformation based on the empirical distribution is the most appropriate choice for this kind of standardization. Through a series of experiments on TREC data, we show the benefits of our proposal in terms of score stability and statistical test behavior.
How to Deal with Scarce Annotations in Answer Selection
Authors: Emmanuel Vallée, Delphine Charlet, Gabriel Marzinotto, Fabrice Clérot, Frank Meyer
Venue: SIGIR
Year: 2019

Addressing Question Answering (QA) tasks with complex neural networks typically requires a large amount of annotated data to achieve a satisfactory accuracy of the models. In this work, we are interested in simple models that can potentially give good performance on datasets with no or few annotations. First, we propose new unsupervised baselines that leverage distributed word and sentence representations. Second, we compare the ability of our neural network architectures to learn from few annotated samples and we demonstrate how these methods can benefit from a pre-training on an external dataset. With a particular emphasis on the reproducibility of our results, we show that our simple models can approach or reach state-of-the-art performance on four common QA datasets.
Learning from Fact-checkers: Analysis and Generation of Fact-checking Language
Authors: Nguyen Vo, Kyumin Lee
Venue: SIGIR
Year: 2019

In fighting against fake news, many fact-checking systems comprised of human-based fact-checking sites (e.g., snopes.com and politifact.com) and automatic detection systems have been developed in recent years. However, online users still keep sharing fake news even when it has been debunked. It means that early fake news detection may be insufficient and we need another complementary approach to mitigate the spread of misinformation. In this paper, we introduce a novel application of text generation for combating fake news. In particular, we (1) leverage online users named fact-checkers, who cite fact-checking sites as credible evidences to fact-check information in public discourse; (2) analyze linguistic characteristics of fact-checking tweets; and (3) propose and build a deep learning framework to generate responses with fact-checking intention to increase the fact-checkers' engagement in fact-checking activities. Our analysis reveals that the fact-checkers tend to refute misinformation and use formal language (e.g. few swear words and Internet slangs). Our framework successfully generates relevant responses, and outperforms competing models by achieving up to 30% improvements. Our qualitative study also confirms that the superiority of our generated responses compared with responses generated from the existing models.
Neural Graph Collaborative Filtering
Authors: Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, Tat-Seng Chua
Venue: SIGIR
Year: 2019

Learning vector representations (aka. embeddings) of users and items lies at the core of modern recommender systems. Ranging from early matrix factorization to recently emerged deep learning based methods, existing efforts typically obtain a user's (or an item's) embedding by mapping from pre-existing features that describe the user (or the item), such as ID and attributes. We argue that an inherent drawback of such methods is that, the collaborative signal, which is latent in user-item interactions, is not encoded in the embedding process. As such, the resultant embeddings may not be sufficient to capture the collaborative filtering effect. In this work, we propose to integrate the user-item interactions - more specifically the bipartite graph structure - into the embedding process. We develop a new recommendation framework Neural Graph Collaborative Filtering (NGCF), which exploits the user-item graph structure by propagating embeddings on it. This leads to the expressive modeling of high-order connectivity in user-item graph, effectively injecting the collaborative signal into the embedding process in an explicit manner. We conduct extensive experiments on three public benchmarks, demonstrating significant improvements over several state-of-the-art models like HOP-Rec [39] and Collaborative Memory Network [5]. Further analysis verifies the importance of embedding propagation for learning better user and item representations, justifying the rationality and effectiveness of NGCF. Codes are available at https://github.com/xiangwang1223/neural_graph_collaborative_filtering.