OpenRaiser/Intern-Atlas huggingface dataset:
On the Orthogonality of Bias and Utility in Ad hoc Retrieval
Authors: Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Morteza Zihayat, Ebrahim Bagheri
Venue: SIGIR
Year: 2021

Various researchers have recently explored the impact of different types of biases on information retrieval tasks such as ad hoc retrieval and question answering. While the impact of bias needs to be controlled in order to avoid increased prejudices, the literature has often viewed the relationship between increased retrieval utility (effectiveness) and reduced bias as a tradeoff where one can suffer from the other. In this paper, we empirically study this tradeoff and explore whether it would be possible to reduce bias while maintaining similar retrieval utility. We show this would be possible by revising the input query through a bias-aware pseudo-relevance feedback framework. We report our findings based on four widely used TREC corpora namely Robust04, Gov2, ClueWeb09 and ClueWeb12 and using two classes of bias metrics. The findings of this paper are significant as they are among the first to show that decrease in bias does not necessarily need to come at the cost of reduced utility.
Files of a Feather Flock Together? Measuring and Modeling How Users Perceive File Similarity in Cloud Storage
Authors: Will Brackenbury, Galen Harrison, Kyle Chard, Aaron J. Elmore, Blase Ur
Venue: SIGIR
Year: 2021

Prior work suggests that users conceptualize the organization of personal collections of digital files through the lens of similarity. However, it is unclear to what degree similar files are actually located near one another (e.g., in the same directory) in actual file collections, or whether leveraging file similarity can improve information retrieval and organization for disorganized collections of files. To this end, we conducted an online study combining automated analysis of 50 Google Drive and Dropbox users' cloud accounts with a survey asking about pairs of files from those accounts. We found that many files located in different parts of file hierarchies were similar in how they were perceived by participants, as well as in their algorithmically extractable features. Participants often wished to co-manage similar files (e.g., deleting one file implied deleting the other file) even if they were far apart in the file hierarchy. To further understand this relationship, we built regression models, finding several algorithmically extractable file features to be predictive of human perceptions of file similarity and desired file co-management. Our findings pave the way for leveraging file similarity to automatically recommend access, move, or delete operations based on users' prior interactions with similar files.
When Choice Happens: A Systematic Examination of Mouse Movement Length for Decision Making in Web Search
Authors: Lukas Brückner, Ioannis Arapakis, Luis A. Leiva
Venue: SIGIR
Year: 2021

Searchers often make a choice in a matter of seconds on SERPs. As a result of a dynamic cognitive process, choice is ultimately reflected in motor movement and thus can be modeled by tracking the computer mouse. However, because not all movements have equal value, it is important to understand how do they and, critically, their sequence length impact model performance. We study three different SERP scenarios where searchers (1)~noticed an advertisement, (2)~abandoned the page, and (3)~became frustrated. We model these scenarios with recurrent neural nets and study the effect of mouse sequence padding and truncating to different lengths. We find that it is possible to predict the aforementioned tasks sometimes using just 2 seconds of movement. Ultimately, by efficiently recording the right amount of data, we can save valuable bandwidth and storage, respect the users' privacy, and increase the speed at which machine learning models can be trained and deployed. Considering the web scale, doing so will have a net benefit on our environment.
Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations
Authors: Nghi D. Q. Bui, Yijun Yu 0001, Lingxiao Jiang
Venue: SIGIR
Year: 2021

We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through acontrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.
Entity Retrieval Using Fine-Grained Entity Aspects
Authors: Shubham Chatterjee, Laura Dietz
Venue: SIGIR
Year: 2021

Using entity aspect links, we improve upon the current state-of-the-art in entity retrieval. Entity retrieval is the task of retrieving relevant entities for search queries, such as "Antibiotic Use In Livestock". Entity aspect linking is a new technique to refine the semantic information of entity links. For example, while passages relevant to the query above may mention the entity "USA", there are many aspects of the USA of which only few, such as "USA/Agriculture", are relevant for this query. By using entity aspect links that indicate which aspect of an entity is being referred to in the context of the query, we obtain more specific relevance indicators for entities. We show that our approach improves upon all baseline methods, including the current state-of-the-art using a standard entity retrieval test collection. With this work, we release a large collection of entity-aspect-links for a large TREC corpus.
Adapted Graph Reasoning and Filtration for Description-Image Retrieval
Authors: Shiqian Chen, Zhiling Luo, Yingqi Gao, Wei Zhou, Chenliang Li, Haiqing Chen
Venue: SIGIR
Year: 2021

Due to the significant cognition reduction, multi-media content has become an increasingly important information type nowadays. More and more descriptions are coupled with images to make them more attractive and persuasive. Currently, several text-image retrieval methods have been developed to improve the efficiency of the time-consuming and professional process. However, in practical retrieval applications, it is the vivid and terse descriptions that are widely used, instead of the shallow captions that describe what is contained. Therefore, the most existing methods designed for the caption-style text can not achieve this purpose. To eliminate the mismatch, we introduce a novel problem about description-image retrieval and propose the specially designed method, named Adapted Graph Reasoning and Filtration (AGRF). In AGRF, we firstly leverage an adapted graph reasoning network to discover the combination of visual objects in the image. Then, a cross-modal gate mechanism is proposed to cast aside those description-independent combinations. Experiment results on the real-world dataset demonstrate the advantages of the AGRF over the state-of-the-art methods.
Adapting Interactional Observation Embedding for Counterfactual Learning to Rank
Authors: Mouxiang Chen, Chenghao Liu, Jianling Sun, Steven C. H. Hoi
Venue: SIGIR
Year: 2021

Counterfactual Learning to Rank (CLTR) becomes an attractive research topic due to its capability of training ranker with click logs. However, CLTR inherently suffers from a large amount of bias caused by confounders, variables that affect both the observation (examination) behavior and click behavior. Recent efforts to correct bias mostly focus on position bias, which assumes that each observation in a ranking list is isolated and only depends on the position. Though effective, users often engage with documents in an interactive manner. Ignoring the interactions between observations/clicks would incur a large interactional observation bias no matter how much data is collected. In this work, we leverage the embedding method to develop an Interactional Observation-Based Model (IOBM) to estimate the observation probability. We argue that while there exist complex observed and unobserved confounders for observation/click interactions, it is sufficient to use the embedding as a proxy confounder to uncover the relevant information for the prediction of the observation propensity. Moreover, the embedding could offer an alternative to the fully specified generative model for observation and decouples the complex interaction structure of observations/clicks. In our IOBM, we first learn the individual observation embedding to capture position and click information. Then, we learn the interactional observation embedding to uncover their local interaction structure. To filter out irrelevant information and reduce contextual bias, we utilize query context information and propose the intra-observation attention and the inter-observation attention, respectively. We conduct extensive experiments on two LTR benchmark datasets, demonstrating that the proposed IOBM consistently achieves better performance over the baseline models in various click situations and verifying its effectiveness of eliminating interactional observation bias.
DAIR: A Query-Efficient Decision-based Attack on Image Retrieval Systems
Authors: Mingyang Chen, Junda Lu 0001, Yi Wang 0017, Jianbin Qin, Wei Wang 0011
Venue: SIGIR
Year: 2021

There is an increasing interest in studying adversarial attacks on image retrieval systems. However, most of the existing attack methods are based on the white-box setting, where the attackers have access to all the model and database details, which is a strong assumption for practical attacks. The generic transfer-based attack also requires substantial resources yet the effect was shown to be unreliable. In this paper, we make the first attempt in proposing a query-efficient decision-based attack framework for the image retrieval (DAIR) to completely subvert the top-K retrieval results with human imperceptible perturbations. We propose an optimization-based method with a smoothed utility function to overcome the challenging discrete nature of the problem. To further improve the query efficiency, we propose a novel sampling method that can achieve the transferability between the surrogate and the target model efficiently. Our comprehensive experimental evaluation on the benchmark datasets shows that our DAIR method outperforms significantly the state-of-the-art decision-based methods. We also demonstrate that real image retrieval engines (Bing Visual Search and Face++ engines) can be attacked successfully with only several hundreds of queries.
Adversarial-Enhanced Hybrid Graph Network for User Identity Linkage
Authors: Xiaolin Chen 0001, Xuemeng Song, Guozhen Peng, Shanshan Feng 0001, Liqiang Nie
Venue: SIGIR
Year: 2021

In this work, we investigate the user identity linkage task across different social media platforms based on heterogeneous multi-modal posts and social connections. This task is non-trivial due to the following two challenges. 1) As each user involves both intra multi-modal posts and inter social connections, how to accurately fulfil the user representation learning from both intra and inter perspectives constitutes the main challenge. And 2) even representations distributed on different platforms of the same identity tend to be distinct (i.e., the semantic gap problem) owing to discrepant data distribution of different platforms. Hence, how to alleviate the semantic gap problem poses another tough challenge. To this end, we propose a novel adversarial-enhanced hybrid graph network (AHG-Net), consisting of three key components: user representation extraction, hybrid user representation learning, and adversarial learning. Specifically, AHG-Net first employs advanced deep learning techniques to extract the user's intermediate representations from his/her heterogeneous multi-modal posts and social connections. Then AHG-Net unifies the intra-user representation learning and inter-user representation learning with a hybrid graph network. Finally, AHG-Net adopts adversarial learning to encourage the learned user presentations of the same identity to be similar using a semantic discriminator. Towards evaluation, we create a multi-modal user identity linkage dataset by augmenting an existing dataset with 62,021 images collected from Twitter and Foursquare. Extensive experiments validate the superiority of the proposed network. Meanwhile, we release the dataset, codes, and parameters to facilitate the research community.
Structured Graph Convolutional Networks with Stochastic Masks for Recommender Systems
Authors: Huiyuan Chen, Lan Wang, Yusan Lin, Chin-Chia Michael Yeh, Fei Wang 0062, Hao Yang 0007
Venue: SIGIR
Year: 2021

Graph Convolutional Networks (GCNs) are powerful for collaborative filtering. The key component of GCNs is to explore neighborhood aggregation mechanisms to extract high-level representations of users and items. However, real-world user-item graphs are often incomplete and noisy. Aggregating misleading neighborhood information may lead to sub-optimal performance if GCNs are not regularized properly. Also, the real-world user-item graphs are often sparse and low rank. These two intrinsic graph properties are widely used in shallow matrix completion models, but far less studied in graph neural models. Here we propose Structured Graph Convolutional Networks (SGCNs) to enhance the performance of GCNs by exploiting graph structural properties of sparsity and low rank. To achieve sparsity, we attach each layer of a GCN with a trainable stochastic binary mask to prune noisy and insignificant edges, resulting in a clean and sparsified graph. To preserve its low-rank property, the nuclear norm regularization is applied. We jointly learn the parameters of stochastic binary masks and original GCNs by solving a stochastic binary optimization problem. An unbiased gradient estimator is further proposed to better backpropagate the gradients of binary variables. Experimental results demonstrate that SGCNs achieve better performance compared with the state-of-the-art GCNs.
Knowledge Graph Embedding via Metagraph Learning
Authors: Chanyoung Chung 0001, Joyce Jiyoung Whang
Venue: SIGIR
Year: 2021

Knowledge graph embedding aims to represent entities and relations in a continuous feature space while preserving the structure of a knowledge graph. Most existing knowledge graph embedding methods either focus only on a flat structure of the given knowledge graph or exploit the predefined types of entities to explore an enriched structure. In this paper, we define the metagraph of a knowledge graph by proposing a new affinity metric that measures the structural similarity between entities, and then grouping close entities by hypergraph clustering. Without any prior information about entity types, a set of semantically close entities is successfully merged into one super-entity in our metagraph representation. We propose the metagraph-based pre-training model of knowledge graph embedding where we first learn representations in the metagraph and initialize the entities and relations in the original knowledge graph with the learned representations. Experimental results show that our method is effective in improving the accuracy of state-of-the-art knowledge graph embedding methods.
Evaluation Measures Based on Preference Graphs
Authors: Charles L. A. Clarke, Chengxi Luo, Mark D. Smucker
Venue: SIGIR
Year: 2021

The offline evaluation of search requires us to define a standard against which we measure the quality of results returned by a ranker. Frequently this standard is defined in absolute terms through relevance grades, but it can also be defined in relative terms through preferences. These preferences might be created through explicit preference judgments, derived from relevance grades, or inferred from clicks and other signals. Preferences from multiple sources might even be combined. In contrast to absolute grades, preferences avoid complex definitions of relevance, indicating only that a ranker should favor one result over another. Despite the simplicity and flexibility of preferences, widespread adoption has been limited by the lack of established evaluation measures. Recent work in this direction has taken two approaches: 1) measures based on weighted counts of agreements and disagreements between a set of preferences and an actual ranking generated by a ranker; and 2) measures that translate preferences into gain values for use with traditional measures, such as nDCG. Both approaches require methods for specifying weights or gains that have little or no theoretical foundation, and the values of these measures have no clear and meaningful interpretation. To address these problems, we propose an evaluation measure that computes the similarity between a directed multigraph of preferences and an actual ranking generated by a ranker. The measure computes an ordering for the vertices of the preference graph that maximizes its similarity to the actual ranking under a rank similarity measure. This maximum similarity becomes the value of the measure. Preference graphs are often acyclic, or nearly so, and to compute the measure we extend an approximate greedy algorithm that is known to produce good results for nearly acyclic graphs. For the rank similarity measure we employ Rank Biased Overlap (RBO) which was explicitly created to match the requirements of search and related applications. We validate the new measure over several collections of preferences explored in recent work.
Allowing for The Grounded Use of Temporal Difference Learning in Large Ranking Models via Substate Updates
Authors: Daniel Cohen
Venue: SIGIR
Year: 2021

We introduce a modification of an established reinforcement learning method to facilitate the widespread use of temporal difference learning for IR: interpolated substate temporal difference (ISSTD) learning. While reinforcement learning methods have shown success in document ranking, these contributions have relied on relatively antiquated policy gradient methods like REINFORCE. These methods bring associated issues like high variance gradient estimates and sample inefficiency, which presents significant obstacles when training deep neural retrieval models. Within the reinforcement learning community, there exists a substantial body of work on alternative methods of training which revolve around temporal difference updates, such as Q-learning, Actor-Critic, or SARSA, that resolve some of the issues seen in REINFORCE. However, temporal difference methods require the full size of the state to be modeled internally within the ranking model, which is unrealistic for deep full text retrieval or first stage retrieval. We therefore propose ISSTD, operating on the substate, or individual documents in the case of matching models, and interpolating the temporal difference updates to the rest of the state. We provide theoretical guarantees on convergence, enabling the drop in use of ISSTD for any algorithm that relies on temporal difference updates. Furthermore, empirical results demonstrate the robustness of this approach for deep neural models, outperforming the current policy gradient approach for training deep neural retrieval models.
Supporting Metacognition during Exploratory Search with the OrgBox
Authors: Anita Crescenzi, Austin R. Ward, Yuan Li 0035, Rob Capra 0001
Venue: SIGIR
Year: 2021

Current search systems provide effective support to users engaged in fact-finding and look-up oriented tasks. However, they provide relatively little support for users engaged in exploratory search tasks that involve cognitive and metacognitive activities such as learning, synthesis, planning, and reflection. We conducted a within-subject user study (N=24) that investigated the effects of a novel knowledge organization tool called the OrgBox, designed to assist users with organizing and synthesizing information, and metacognitive activities. The OrgBox included features to allow users to drag-drop information they found through search into "boxes" that could be created, labelled, and re-arranged. Study participants completed two exploratory search tasks, one with the OrgBox, and one with the OrgDoc, a baseline tool that included features of a rich-text editor (e.g., formatting, bullets) for taking notes.
Motif-aware Sequential Recommendation
Authors: Zeyu Cui, Yinjiang Cai, Shu Wu, Xibo Ma, Liang Wang 0001
Venue: SIGIR
Year: 2021

Sequential recommendation is intended to model the dynamic behavior regularity through users' behavior sequences. Recently, various deep learning techniques are applied to model the relation of items in the sequences. Despite their effectiveness, we argue that the aforementioned methods only consider the macro-structure of the behavior sequence, but neglect the micro-structure in the sequence which is important to sequential recommendation. To address the above limitation, we propose a novel model called Motif-aware Sequential Recommendation (MoSeR), which captures the motifs hidden in behavior sequences to model the micro-structure features. MoSeR extracts the motifs that contain both the last behavior and the target item. These motifs reflect the topological relations among local items in the form of directed graphs. Thus our method can make a more accurate prediction with the awareness of the inherent patterns between local items. Extensive experiments on three benchmark datasets demonstrate that our model outperforms the state-of-the-art sequential recommendation models.
An Image is Worth a Thousand Terms? Analysis of Visual E-Commerce Search
Authors: Arnon Dagan, Ido Guy, Slava Novgorodov
Venue: SIGIR
Year: 2021

Visual search has become popular in recent years, allowing users to search by an image they are taking using their mobile device or uploading from their photo library. One domain in which visual search is especially valuable is electronic commerce, where users seek for items to purchase. In this work, we present an in-depth comprehensive study of visual e-commerce search. We perform query log analysis of one of the largest e-commerce platforms' mobile search application. We compare visual and textual search by a variety of characteristics, with special focus on the retrieved results and user interaction with them. We also examine image query characteristics, refinement by attributes, and performance prediction for visual search queries. Our analysis points out a variety of differences between visual and textual e-commerce search. We discuss the implications of these differences for the design of future e-commerce search systems.