OpenRaiser/Intern-Atlas huggingface dataset:
Conversational vs Traditional: Comparing Search Behavior and Outcome in Legal Case Retrieval
Authors: Bulou Liu, Yueyue Wu, Yiqun Liu 0001, Fan Zhang 0053, Yunqiu Shao, Chenliang Li 0005, Min Zhang 0006, Shaoping Ma
Venue: SIGIR
Year: 2021

In recent years, legal case retrieval has attracted much attention in the IR research community. It aims to retrieve supporting cases for a given query case and contributes to better legal systems. While using a legal case retrieval system, users always feel difficult to construct accurate queries to express their information need, especially when they lack sufficient domain knowledge. Since conversational search has been widely recognized to fulfill users' complex and exploratory information need, we investigate whether conversational search paradigm can be adopted to improve users' legal case retrieval experience. We design a laboratory-based study to collect users' interaction behaviors and explicit feedback signals while using traditional and agent-mediated conversational legal case retrieval systems. Based on the collected data, we compare search behavior and outcome of these two different kinds of interaction paradigms. Compared with the traditional one, experimental results show that users can achieve better retrieval performance with the conversational case retrieval system. Moreover, conversational system can also save users' efforts in formulating queries and examining results.
X-2ch: Quad-Channel Collaborative Graph Network over Knowledge-Embedded Edges
Authors: Kachun Lo, Tsukasa Ishigaki
Venue: SIGIR
Year: 2021

Carrying abundant side information, knowledge graph (KG) has shown its great potential in enriching the sparsity of collaborative filtering (CF) for recommendation. Although graph neural networks (GNNs) have been successfully employed to learn user preferences from KG and CF signals simultaneously, most models suffer from inferior performance due to their deficient designs, i.e., 1) formulating no distinction between users, items and KG entities, 2) confounding KG signals with CF signals and 3) completely neglecting the effects of edges, which is vital for graph information propagation. In this paper, we propose a quad-channel graph model (X-2ch) to tackle these problems. First, rather than lodging KG entities on graph as nodes, X-2ch distills KG information and embeds them as edge attributes in a bi-directional manner to model the natural user-item interaction process. Second, X-2ch introduces a novel quad-channel learning scheme, including a collaborative user-item update and a CF-KG attentive propagation, to holistically capture the interconnectivity of users and items while preserving their distinct properties. Experiments on two real-world benchmarks show substantial improvement over the state-of-the-art baselines.
Position Enhanced Mention Graph Attention Network for Dialogue Relation Extraction
Authors: Xinwei Long, Shuzi Niu, Yucheng Li 0002
Venue: SIGIR
Year: 2021

Dialogue Relation Extraction (DRE) is a new kind of relation extraction task from multi-turn dialogues. Different from the previous tasks, speaker specific relations are implicitly mixed together in both a local utterance window and a speaker context. To tackle both local and speaker dependency challenges, we explicitly construct a unified mention co-occurrence graph within a local utterance window or all utterances of a speaker from different entities. For each dialogue, a position enhanced graph attention network over this graph is proposed to obtain position aware mention representations in terms of both contexts. A gate function is utilized to help obtain a discriminative representation enough for each relation from original and position aware mention representations. For each entity pair in this dialogue, a pairwise attention mechanism is deployed to aggregate those discriminative mention representations as pair representation, which is fed into a standard multi-label classifier for relation label prediction. Experimental results on two benchmarks show the performance improvement of the proposed method is at least 1.6% and 3.2% compared with SOTA.
Predicting Patient Readmission Risk from Medical Text via Knowledge Graph Enhanced Multiview Graph Convolution
Authors: Qiuhao Lu, Thien Huu Nguyen, Dejing Dou
Venue: SIGIR
Year: 2021

Unplanned intensive care unit (ICU) readmission rate is an important metric for evaluating the quality of hospital care. Efficient and accurate prediction of ICU readmission risk can not only help prevent patients from inappropriate discharge and potential dangers, but also reduce associated costs of healthcare. In this paper, we propose a new method that uses medical text of Electronic Health Records (EHRs) for prediction, which provides an alternative perspective to previous studies that heavily depend on numerical and time-series features of patients. More specifically, we extract discharge summaries of patients from their EHRs, and represent them with multiview graphs enhanced by an external knowledge graph. Graph convolutional networks are then used for representation learning. Experimental results prove the effectiveness of our method, yielding state-of-the-art performance for this task.
Distant Supervision based Machine Reading Comprehension for Extractive Summarization in Customer Service
Authors: Bing Ma, Cao Liu, Jingyu Wang 0001, Shujie Hu, Fan Yang 0087, Xunliang Cai, Guanglu Wan, Jiansong Chen, Jianxin Liao
Venue: SIGIR
Year: 2021

Given a long text, the summarization system aims to obtain a shorter highlight while keeping important information on the original text. For customer service, the summaries of most dialogues between an agent and a user focus on several fixed key points, such as user's question, user's purpose, the agent's solution, and so on. Traditional extractive methods are difficult to extract all predefined key points exactly. Furthermore, there is a lack of large-scale and high-quality extractive summarization datasets containing key points. In order to solve the above challenges, we propose a Distant Supervision based Machine Reading Comprehension model for extractive Summarization (DSMRC-S). DSMRC-S transforms the summarization task into the machine reading comprehension problem, to fetch key points from the original text exactly according to the predefined questions. In addition, a distant supervision method is proposed to alleviate the lack of eligible extractive summarization datasets. We conduct experiments on a large-scale summarization dataset collected in customer service scenarios, and the results show that the proposed DSMRC-S outperforms the strong baseline methods by 4 points on ROUGE-L.
Enhanced Representation Learning for Examination Papers with Hierarchical Document Structure
Authors: Yixiao Ma, Shiwei Tong, Ye Liu 0011, Likang Wu, Qi Liu 0003, Enhong Chen, Wei Tong, Zi Yan
Venue: SIGIR
Year: 2021

Representation learning of examination papers is the cornerstone of the Examination Paper Analysis (EPA) in education area including Paper Difficulty Prediction (PDR) and Finding Similar Papers (FSP). Previous works mainly focus on the representation learning of each test item, but few works notice the hierarchical document structure in examination papers. To this end, in this paper, we propose a novel Examination Organization Encoder (EOE) to learn a robust representation of the examination paper with the hierarchical document structure. Specifically, we first propose a syntax parser to recover the hierarchical document structure and convert an examination paper to an Examination Organization Tree (EOT), where the test items are the leaf nodes and the internal nodes are summarization of their child nodes. Then, we applied a two-layer GRU-based module to obtain the representation of each leaf node. After that, we design a subtree encoder module to aggregate the representation of each leaf node, which is used to calculate an embedding for each layer in the EOT. Finally, we feed all the layer embedding into an output module, the process is over and we get the examination paper representation that can be used for downstream tasks. Extensive experiments on real-world data demonstrate the effectiveness and interpretability of our method.
Learning to Rank for Mathematical Formula Retrieval
Authors: Behrooz Mansouri, Richard Zanibbi, Douglas W. Oard
Venue: SIGIR
Year: 2021

In Mathematical Information Retrieval (MIR), formulae can be used in a query to match other similar formulae in documents. However, due to the structural complexity of formulae, specialized processing is needed for formula matching. Formulae may be represented by their appearance in Symbol Layout Trees (SLTs) or by their syntax in Operator Trees (OPTs). Previous approaches for formula retrieval used one or both of these representations and used unification to improve search results for inexact matches (e.g., allowing different variable names to match). On these representations, models for matching full expressions (trees), subexpressions, and paths have been used. Recently embedding models were used to represent formulae as vectors. In this paper, the effectiveness of retrieval models and formula representations are studied to identify their relative strengths and weaknesses. Then, a learning to rank model is proposed, using SVM-rank over similarity scores from different formula retrieval models as features. Experiments on the ARQMath formula retrieval task results show that the proposed learning to rank model is effective, producing new state-of-the-art results.
Do Affective Cues Validate Behavioural Metrics for Search?
Authors: Daniel McDuff, Paul Thomas 0001, Nick Craswell, Kael Rowan, Mary Czerwinski
Venue: SIGIR
Year: 2021

Traces of searcher behaviour, such as query reformulation or clicks, are commonly used to evaluate a running search engine. The underlying expectation is that these behaviours are proxies for something more important, such as relevance, utility, or satisfaction. Affective computing technology gives us the tools to help confirm some of these expectations, by examining visceral expressive responses during search sessions. However, work to date has only studied small populations in laboratory settings and with a limited number of contrived search tasks. In this study, we analysed longitudinal, in-situ, search behaviours of 152 information workers, over the course of several weeks while simultaneously tracking their facial expressions. Results from over 20,000 search sessions and 45,000 queries allow us to observe that indeed affective expressions are consistent with, and complementary to, existing "click-based'' metrics. On a query-level, searches that result in a short dwell time are associated with a decrease in smiles (expressions of "happiness'') and that if a query is reformulated the results of the reformulation are associated with an increase in smiling---suggesting a positive outcome as people converge on the information they need. On a session-level, sessions that feature reformulations are more commonly associated with fewer smiles and more furrowed brows (expressions of "anger/frustration''). Similarly, sessions with short-dwell clicks are also associated with fewer smiles. These data provide an insight into visceral aspects of search experience and present a new dimension for evaluating engine performance.
DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization
Authors: Safa Messaoud, Ismini Lourentzou, Assma Boughoula, Mona Zehni, Zhizhen Zhao 0001, Chengxiang Zhai, Alexander G. Schwing
Venue: SIGIR
Year: 2021

The recent growth of web video sharing platforms has increased the demand for systems that can efficiently browse, retrieve and summarize video content. Query-aware multi-video summarization is a promising technique that caters to this demand. In this work, we introduce a novel Query-Aware Hierarchical Pointer Network for Multi-Video Summarization, termed DeepQAMVS, that jointly optimizes multiple criteria: (1) conciseness, (2) representativeness of important query-relevant events and (3) chronological soundness. We design a hierarchical attention model that factorizes over three distributions, each collecting evidence from a different modality, followed by a pointer network that selects frames to include in the summary. DeepQAMVS is trained with reinforcement learning, incorporating rewards that capture representativeness, diversity, query-adaptability and temporal coherence. We achieve state-of-the-art results on the MVS1K dataset, with inference time scaling linearly with the number of input video frames.
Ranking User-Generated Content via Multi-Relational Graph Convolution
Authors: Kanika Narang, Adit Krishnan, Junting Wang 0001, Chaoqi Yang, Hari Sundaram, Carolyn Sutter
Venue: SIGIR
Year: 2021

The quality variance in user-generated content is a major bottleneck to serving communities on online platforms. Current content ranking methods primarily evaluate text and non-textual content features of each user post in isolation. In this paper, we demonstrate the utility of considering the implicit and explicit relational aspects across user content to assess their quality. First, we develop a modular platform-agnostic framework to represent the contrastive (or competing) and similarity-based relational aspects of user-generated content via independently induced content graphs. Second, we develop two complementary graph convolutional operators that enable feature contrast for competing content and feature smoothing/sharing for similar content. Depending on the edge semantics of each content graph, we embed its nodes via one of the above two mechanisms. We also show that our contrastive operator creates discriminative magnification across the embeddings of competing posts. Third, we show a surprising result-applying classical boosting techniques to combine final-layer embeddings across the content graphs significantly outperforms the typical stacking, fusion, or neighborhood embedding aggregation methods in graph convolutional architectures. We exhaustively validate our method via accepted answer prediction over fifty diverse Stack-Exchange (https://stackexchange.com/) websites with consistent relative gains of over 5% accuracy over state-of-the-art neural, multi-relational and textual baselines.
On the Privacy of Federated Pipelines
Authors: Reza Nasirigerdeh, Reihaneh Torkzadehmahani, Jan Baumbach, David B. Blumenthal
Venue: SIGIR
Year: 2021

Federated learning (FL) is becoming an increasingly popular machine learning paradigm in application scenarios where sensitive data available at various local sites cannot be shared due to privacy protection regulations. In FL, the sensitive data never leaves the local sites and only model parameters are shared with a global aggregator. Nonetheless, it has recently been shown that, under some circumstances, the private data can be reconstructed from the model parameters, which implies that data leakage can occur in FL. In this paper, we draw attention to another risk associated with FL: Even if federated algorithms are individually privacy-preserving, combining them into pipelines is not necessarily privacy-preserving. We provide a concrete example from genome-wide association studies, where the combination of federated principal component analysis and federated linear regression allows the aggregator to retrieve sensitive patient data by solving an instance of the multidimensional subset sum problem. This supports the increasing awareness in the field that, for FL to be truly privacy-preserving, measures have to be undertaken to protect against data leakage at the aggregator.
Multimodal Activation: Awakening Dialog Robots without Wake Words
Authors: Liqiang Nie, Mengzhao Jia, Xuemeng Song, Ganglu Wu, Harry Cheng 0002, Jian Gu
Venue: SIGIR
Year: 2021

When talking to the dialog robots, users have to activate the robot first from the standby mode with special wake words, such as "Hey Siri", which is apparently not user-friendly. The latest generation of dialog robots have been equipped with advanced sensors, like the camera, enabling multimodal activation. In this work, we work towards awaking the robot without wake words. To accomplish this task, we present a Multimodal Activation Scheme (MAS), consisting of two key components: audio-visual consistency detection and semantic talking intention inference. The first one is devised to measure the consistency between the audio and visual modalities in order to figure out weather the heard speech comes from the detected user in front of the camera. Towards this end, two heterogeneous CNN-based networks are introduced to convolutionalize the fine-grained facial landmark features and the MFCC audio features, respectively. The second one is to infer the semantic talking intention of the recorded speech, where the transcript of the speech is recognized and matrix factorization is utilized to uncover the latent human-robot talking topics. We ultimately devise different fusion strategies to unify these two components. To evaluate MAS, we construct a dataset containing 12,741 short videos recorded by 194 invited volunteers. Extensive experiments demonstrate the effectiveness of our scheme.
Interpretable Document Representations for Fast and Accurate Retrieval of Mathematical Information
Authors: Vít Novotný
Venue: SIGIR
Year: 2021

A study conducted by the International Data Corporation predicted that by the year 2021, the total amount of digital information resources would have reached the 40 zettabyte mark [2]. According to a rule formulated by Merrill Lynch, 80 to 90% of these resources are unstructured [7]. Despite this, users expect digital libraries to provide them with fast and interpretable access to digital information resources that will satisfy their information need. Math information retrieval emerged as a subfield of information retrieval in 2008 [8], when it became clear that standard information retrieval techniques used for text documents are inadequate to accurately retrieve documents in digital mathematical libraries.
REGIS: A Test Collection for Geoscientific Documents in Portuguese
Authors: Lucas Lima de Oliveira, Regis Kruel Romeu, Viviane Pereira Moreira
Venue: SIGIR
Year: 2021

Experimental validation is key to the development of Information Retrieval (IR) systems. The standard evaluation paradigm requires a test collection with documents, queries, and relevance judgments. Creating test collections requires significant human effort, mainly for providing relevance judgments. As a result, there are still many domains and languages that, to this day, lack a proper evaluation testbed. Portuguese is an example of a major world language that has been overlooked in terms of IR research -- the only test collection available is composed of news articles from 1994 and a hundred queries. With the aim of bridging this gap, in this paper, we developed REGIS (Retrieval Evaluation for Geoscientific Information Systems), a test collection for the geoscientific domain in Portuguese. REGIS contains 20K documents and 34 query topics along with relevance assessments. We describe the procedures for document collection, topic creation, and relevance assessment. In addition, we report on results of standard IR techniques on REGIS so that they can serve as a baseline for future research.
Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness
Authors: Harrie Oosterhuis
Venue: SIGIR
Year: 2021

Recent work has proposed stochastic Plackett-Luce (PL) ranking models as a robust choice for optimizing relevance and fairness metrics. Unlike their deterministic counterparts that require heuristic optimization algorithms, PL models are fully differentiable. Theoretically, they can be used to optimize ranking metrics via stochastic gradient descent. However, in practice, the computation of the gradient is infeasible because it requires one to iterate over all possible permutations of items. Consequently, actual applications rely on approximating the gradient via sampling techniques. In this paper, we introduce a novel algorithm: PL-Rank, that estimates the gradient of a PL ranking model w.r.t. both relevance and fairness metrics. Unlike existing approaches that are based on policy gradients, PL-Rank makes use of the specific structure of PL models and ranking metrics. Our experimental analysis shows that PL-Rank has a greater sample-efficiency and is computationally less costly than existing policy gradients, resulting in faster convergence at higher performance. PL-Rank further enables the industry to apply PL models for more relevant and fairer real-world ranking systems.
Propensity-Independent Bias Recovery in Offline Learning-to-Rank Systems
Authors: Zohreh Ovaisi, Kathryn Vasilaky, Elena Zheleva
Venue: SIGIR
Year: 2021

Learning-to-rank systems often utilize user-item interaction data (e.g., clicks) to provide users with high-quality rankings. However, this data suffers from several biases, and if naively used as training data, it can lead to suboptimal ranking algorithms. Most existing bias-correcting methods focus on position bias, the fact that higher-ranked results are more likely to receive interaction, and address this bias by leveraging inverse propensity weighting. However, it is not always possible to accurately estimate propensity scores, and in addition to position bias, selection bias is often encountered in real-world recommender systems. Selection bias occurs because users are exposed to a truncated list of results, which gives a zero chance for some items to be observed and, therefore, interacted with, even if they are relevant. Here, we propose a new counterfactual method that uses a two-stage correction approach and jointly addresses selection and position bias in learning-to-rank systems without relying on propensity scores. Our experimental results show that our method is better than state-of-the-art propensity-independent methods and either better than or comparable to methods that make the strong assumption for which the propensity model is known.
Beyond Probability Ranking Principle: Modeling the Dependencies among Documents
Authors: Liang Pang 0001, Qingyao Ai, Jun Xu 0001
Venue: SIGIR
Year: 2021

Probability Ranking Principle (PRP)[31], which assumes that each document has a unique and independent probability to satisfy a particular information need, is one of the fundamental principles for ranking. Traditionally, heuristic ranking features and well-known learning-to-rank approaches have been designed by following the PRP principle. Recently, neural IR models, which adopt deep learning to enhance the ranking performances, also obey the PRP principle. Though it has been widely used for nearly five decades, in-depth analysis shows that PRP is not an optimal principle for ranking, due to its independent assumption that each document should be independent of the rest candidates. Counter examples include pseudo relevance feedback[24], interactive information retrieval[46], search result diversification[10] etc. To solve the problem, researchers recently proposed to model the dependencies among the documents during the designing of ranking models. A number of ranking models have been proposed and state-of-the-art ranking performances have been achieved. This tutorial aims to give a comprehensive survey on these recently developed ranking models that go beyond the PRP principle. The tutorial tries to categorize these models based on their intrinsic assumptions: assuming that the documents are independent, sequentially dependent, or globally dependent. In this way, we expect the researchers focusing on ranking in search and recommendation can have a novel angle of view on the designing of ranking models, and therefore can stimulate new ideas on developing novel ranking models.