OpenRaiser/Intern-Atlas huggingface dataset:
Session Search with Pre-trained Graph Classification Model
Authors: Shengjie Ma, Chong Chen, Jiaxin Mao, Qi Tian, Xuhui Jiang
Venue: SIGIR
Year: 2023

Session search is a widely adopted technique in search engines that seeks to leverage the complete interaction history of a search session to better understand the information needs of users and provide more relevant ranking results. The vast majority of existing methods model a search session as a sequence of queries and previously clicked documents. However, if we simply represent a search session as a sequence we will lose the topological information in the original search session. It is non-trivial to model the intra-session interactions and complicated structural patterns among the previously issued queries, clicked documents, as well as the terms or entities that appeared in them. To solve this problem, in this paper, we propose a novel Session Search with Graph Classification Model (SSGC), which regards session search as a graph classification task on a heterogeneous graph that represents the search history in each session. To improve the performance of the graph classification, we design a specific pre-training strategy for our proposed GNN-based classification model. Extensive experiments on two public session search datasets demonstrate the effectiveness of our model in the session search task.
Defining and Measuring Cost, Effort, and Load in Information Retrieval
Authors: Molly McGregor
Venue: SIGIR
Year: 2023

The demands imposed on the user during the Information Retrieval (IR) process are well-recognised, with numerous studies highlighting the influence and importance of cost, effort, and load in explaining user search behaviour and experience. Despite this understanding, there exists no universal definitions of these constructs or standardised methods of measuring them within the field of IR. As a result, an open research problem has emerged in relation to how these constructs have been defined, interpreted, and subsequently measured. To address this research problem, the aim of this research is two-fold. Firstly, to establish working definitions and a conceptual framework for defining cost, effort, and load; and secondly, to examine and evaluate existing measures of effort and load within an IR context.
Denoise to Protect: A Method to Robustify Visual Recommenders from Adversaries
Authors: Felice Antonio Merra, Vito Walter Anelli, Tommaso Di Noia, Daniele Malitesta, Alberto Carlo Maria Mancino
Venue: SIGIR
Year: 2023

While the integration of product images enhances the recommendation performance of visual-based recommender systems (VRSs), this can make the model vulnerable to adversaries that can produce noised images capable to alter the recommendation behavior. Recently, stronger and stronger adversarial attacks have emerged to raise awareness of these risks; however, effective defense methods are still an urgent open challenge. In this work, we propose "Adversarial Image Denoiser" (AiD), a novel defense method that cleans up the item images by malicious perturbations. In particular, we design a training strategy whose denoising objective is to minimize both the visual differences between clean and adversarial images and preserve the ranking performance in authentic settings. We perform experiments to evaluate the efficacy of AiD using three state-of-the-art adversarial attacks mounted against standard VRSs. Code and datasets at https://github.com/sisinflab/Denoise-to-protect-VRS.
Measuring Service-Level Learning Effects in Search Via Query-Randomized Experiments
Authors: Paul Musgrave, Cuize Han, Parth Gupta
Venue: SIGIR
Year: 2023

In order to determine the relevance of a given item to a query, most modern search ranking systems make use of features which aggregate prior user behavior for that item and query (e.g. click rate). For practical reasons, when running A/B tests on ranking systems, these features are generally shared between all treatments. For the most common experiment designs, which randomize traffic by user or by session, this creates a pathway by which the behavior of units in one treatment can effect the outcomes for units in other treatments, violating the Stable Unit Treatment Value Assumption (SUTVA) and biasing measured outcomes. Moreover, for experiments targeting improvements to the behavior data available to such features (e.g. online exploration), this pathway is precisely the one we are trying to affect; if such changes occur identically in treatment and control, then they cannot be measured. To address this, we propose the use of experiments which instead randomize traffic based on the search query. To validate our approach, we perform a pair of A/B tests on an explore-exploit framework in the Amazon search page: one under query randomization, and one under user randomization. In line with the theoretical predictions, we find that the query-randomized iteration is able to measure a statistically significant effect (+0.66% Purchases, p=0.001) where the user-randomized iteration does not (-0.02% Purchases, p=0.851).
BLADE: Combining Vocabulary Pruning and Intermediate Pretraining for Scaleable Neural CLIR
Authors: Suraj Nair, Eugene Yang, Dawn J. Lawrie, James Mayfield, Douglas W. Oard
Venue: SIGIR
Year: 2023

Learning sparse representations using pretrained language models enhances the monolingual ranking effectiveness. Such representations are sparse vectors in the vocabulary of a language model projected from document terms. Extending such approaches to Cross-Language Information Retrieval (CLIR) using multilingual pretrained language models poses two challenges. First, the larger vocabularies of multilingual models affect both training and inference efficiency. Second, the representations of terms from different languages with similar meanings might not be sufficiently similar. To address these issues, we propose a learned sparse representation model, BLADE, combining vocabulary pruning with intermediate pre-training based on cross-language supervision. Our experiments reveal BLADE significantly reduces indexing time compared to its monolingual counterpart, SPLADE, on machine-translated documents, and it generates rankings with strengths complementary to those of other efficient CLIR methods.
Generative Information Retrieval
Authors: Marc Najork
Venue: SIGIR
Year: 2023

Historically, information retrieval systems have all followed the same paradigm: information seekers frame their needs in the form of a short query, the system selects a small set of relevant results from a corpus of available documents, rank-orders the results by decreasing relevance, possibly excerpts a responsive passage for each result, and returns a list of references and excerpts to the user. Retrieval systems typically did not attempt fusing information from multiple documents into an answer and displaying that answer directly. This was largely due to available technology: at the core of each retrieval system is an index that maps lexical tokens or semantic embeddings to document identifiers. Indices are designed for retrieving responsive documents; they do not support integrating these documents into a holistic answer. More recently, the coming-of-age of deep neural networks has dramatically improved the capabilities of large language models (LLMs). Trained on a large corpus of documents, these models not only memorize the vocabulary, morphology and syntax of human languages, but have shown to be able to memorize facts and relations. Generative language models, when provided with a prompt, will extend the prompt with likely completions -- an ability that can be used to extract answers to questions from the model. Two years ago, Metzler et al. argued that this ability of LLMs will allow us to rethink the search paradigm: to answer information needs directly rather that directing users to responsive primary sources. Their vision was not without controversy; the following year Shaw and Bender argued that such a system is neither feasible nor desirable. Nonetheless, the past year has seen the emergence of such systems, with offerings from established search engines and multiple new entrants to the industry. The keynote will summarize the short history of these generative information retrieval systems, and focus on the many open challenges in this emerging field: ensuring that answers are grounded, attributing answer passages to a primary source, providing nuanced answers to non-factoid-seeking questions, avoiding bias, and going beyond simple regurgitation of memorized facts. It will also touch on the changing nature of the content ecosystem. LLMs are starting to be used to generate web content. Should search engines treat such derived content equal to human-authored content? Is it possible to distinguish generated from original content? How should we view hybrid authorship where humans contribute ideas and LLMs shape these ideas into prose? And how will this parallel technical evolution of search engines and content ecosystems affect their respective business models?
News Popularity Beyond the Click-Through-Rate for Personalized Recommendations
Authors: Ashutosh Nayak, Mayur Garg, Rajasekhara Reddy Duvvuru Muni
Venue: SIGIR
Year: 2023

Popularity detection of news articles is critical for making relevant recommendations for users and drive user engagement for maximum business value. Among several well-known metrics such as likes, shares, comments, Click-Through-Rate (CTR) has evolved as a default metric of popularity. However, CTR is highly influenced by the probability of news articles getting an impression, which in turn depends on the recommendation algorithm. Furthermore, it does not consider the age of the news articles, which are highly perishable and also misses out on human contextual behavioral preferences towards news. Here, we use the MIND dataset, open sourced by Microsoft to investigate the existing metrics of popularity and propose six new metrics. Our aim is to create awareness about the different perspectives of measuring popularity while discussing the advantages and disadvantages of the proposed metrics with respect to the human click behavior. We evaluated the predictability of the proposed metrics in comparison to CTR prediction. We further evaluated the utility of the proposed metrics through different test cases. Our results indicate that by using appropriate popularity metrics, we can reduce the initial news corpus (item set) by 50% and still could achieve 99% of the total clicks as compared to unfiltered news corpus based recommender systems. Similarly, our results show that we can reduce the effective number of articles recommended per impression that could improve user experience with the news platforms. The metrics proposed in this paper can be useful in other contexts, especially in recommenders with perishable items e.g. video reels or blogs.
A Retrieval System for Images and Videos based on Aesthetic Assessment of Visuals
Authors: Daniel Vera Nieto, Saikishore Kalloori, Fabio Zund, Clara Fernandez-Labrador, Marc Willhaus, Severin Klingler, Markus Gross
Venue: SIGIR
Year: 2023

Attractive images or videos are the visual backbones of journalism and social media to gain the user's attention. From trailers to teaser images to image galleries, appealing visuals have only grown in importance over the years. However, selecting eye-catching shots from a video or the perfect image from large image collections is a challenging and time-consuming task. We present our tool that can assess image and video content from an aesthetic standpoint. We discovered that it is possible to perform such an assessment by combining expert knowledge with data-driven information. We combine the relevant aesthetic features and machine learning algorithms into an aesthetics retrieval system, which enables users to sort uploaded visuals based on an aesthetic score and interact with additional photographic, cinematic, and person-specific features.
Facebook Content Search: Efficient and Effective Adapting Search on A Large Scale
Authors: Xiangyu Niu, Yu-Wei Wu, Xiao Lu, Gautam Nagpal, Philip Pronin, Kecheng Hao, Zhen Liao, Guangdeng Liao
Venue: SIGIR
Year: 2023

Facebook content search is a critical channel that enables people to discover the best content to deepen their engagement with friends and family, creators, and communities. Building a highly personalized search engine to serve billions of daily active users to find the best results from a large scale of candidates is a challenging task. The search engine must take multiple dimensions into consideration, including different content types, different query intents, and user social graph, etc. In this paper, we discuss the challenges of Facebook content search in depth, and then describe our novel approach to efficiently handling a massive number of documents with advanced query understanding, retrieval, and machine learning techniques. The proposed system has been fully verified and applied to the production system of Facebook Search, which serves billions of users.
Improved Vector Quantization For Dense Retrieval with Contrastive Distillation
Authors: James O&apos, Neill, Sourav Dutta
Venue: SIGIR
Year: 2023

Recent work has identified that distillation can be used to create vector quantization based ANN indexes by learning the inverted file index and product quantization. The argued advantage of using a fixed teacher encoder for queries and documents is that the scores produced by the teacher can be used instead of the label judgements that are required when using traditional supervised learning, such as contrastive learning. However, current work only distills the teacher encoder outputs of dot products between quantized query embedddings and product quantized document embeddings. Our work combines the benefits of contrastive learning and distillation by using contrastive distillation whereby the teacher outputs contrastive scores that the student learns from. Our experimental results on MSMARCO passage retrieval and NQ open question answering datasets show that contrastive distillation improves over current state of the art for vector quantized dense retrieval.
Curse of "Low" Dimensionality in Recommender Systems
Authors: Naoto Ohsaka, Riku Togashi
Venue: SIGIR
Year: 2023

Beyond accuracy, there are a variety of aspects to the quality of recommender systems, such as diversity, fairness, and robustness. We argue that many of the prevalent problems in recommender systems are partly due to low-dimensionality of user and item embeddings, particularly when dot-product models, such as matrix factorization, are used. In this study, we showcase empirical evidence suggesting the necessity of sufficient dimensionality for user/item embeddings to achieve diverse, fair, and robust recommendation. We then present theoretical analyses of the expressive power of dot-product models. Our theoretical results demonstrate that the number of possible rankings expressible under dot-product models is exponentially bounded by the dimension of item factors. We empirically found that the low-dimensionality contributes to a popularity bias, widening the gap between the rank positions of popular and long-tail items; we also give a theoretical justification for this phenomenon.
A Critical Reexamination of Intra-List Distance and Dispersion
Authors: Naoto Ohsaka, Riku Togashi
Venue: SIGIR
Year: 2023

Diversification of recommendation results is a promising approach for coping with the uncertainty associated with users' information needs. Of particular importance in diversified recommendation is to define and optimize an appropriate diversity objective. In this study, we revisit the most popular diversity objective called intra-list distance (ILD), defined as the average pairwise distance between selected items, and a similar but lesser known objective called dispersion, which is the minimum pairwise distance. Owing to their simplicity and flexibility, ILD and dispersion have been used in a plethora of diversified recommendation research. Nevertheless, we do not actually know what kind of items are preferred by them. We present a critical reexamination of ILD and dispersion from theoretical and experimental perspectives. Our theoretical results reveal that these objectives have potential drawbacks: ILD may select duplicate items that are very close to each other, whereas dispersion may overlook distant item pairs. As a competitor to ILD and dispersion, we design a diversity objective called Gaussian ILD, which can interpolate between ILD and dispersion by tuning the bandwidth parameter. We verify our theoretical results by experimental results using real-world data and confirm the extreme behavior of ILD and dispersion in practice.
A Unified Formulation for the Frequency Distribution of Word Frequencies using the Inverse Zipf's Law
Authors: Can Özbey, Talha Çolakoglu, M. Safak Bilici, Ekin Can Erkus
Venue: SIGIR
Year: 2023

The power-law approximation for the frequency distribution of words postulated by Zipf has been extensively studied for decades, which led to many variations on the theme. However, comparatively less attention has been paid to the investigation of the case of word frequencies. In this paper, we derive its analytical expression from the inverse of the underlying rank-size distribution as a function of total word count, vocabulary size and the shape parameter, thereby providing a unified framework to explain the nonlinear behavior of low frequencies on the log-log scale. We also present an efficient method based on relative entropy minimization for a robust estimation of the shape parameter using a small number of empirical low-frequency probabilities. Experiments were carried out for a selected set of languages with varying degrees of inflection in order to demonstrate the effectiveness of the proposed approach.
Personalized Federated Relation Classification over Heterogeneous Texts
Authors: Ning Pang, Xiang Zhao, Weixin Zeng, Ji Wang, Weidong Xiao
Venue: SIGIR
Year: 2023

Relation classification detects the semantic relation between two annotated entities from a piece of text, which is a useful tool for structurization of knowledge. Recently, federated learning has been introduced to train relation classification models in decentralized settings. Current methods strive for a strong server model by decoupling the model training at server from direct access to texts at clients while taking advantage of them. Nevertheless, they overlook the fact that clients have heterogeneous texts (i.e., texts with diversely skewed distribution of relations), which renders existing methods less practical. In this paper, we propose to investigate personalized federated relation classification, in which strong client models adapted to their own data are desired. To further meet the challenges brought by heterogeneous texts, we present a novel framework, namely pf-RC, with several optimized designs. It features a knowledge aggregation method that exploits a relation-wise weighting mechanism, and a feature augmentation method that leverages prototypes to adaptively enhance the representations of instances of long-tail relations. We experimentally validate the superiority of pf-RC against competing baselines in various settings, and the results suggest that the tailored techniques mitigate the challenges.
Leader-Generator Net: Dividing Skill and Implicitness for Conquering FairytaleQA
Authors: Wei Peng, Wanshui Li, Yue Hu
Venue: SIGIR
Year: 2023

Machine reading comprehension requires systems to understand the given passage and answer questions. Previous methods mainly focus on the interaction between the question and passage. However, they ignore the deep exploration of cognitive elements behind questions, such as fine-grained reading skills (this paper focuses on narrative comprehension skills) and implicitness or explicitness of the question (whether the answer can be found in the passage). Grounded in prior literature on reading comprehension, the understanding of a question is a complex process where human beings need to understand the semantics of the question, use different reading skills for different questions, and then judge the implicitness of the question. To this end, a simple but effective Leader-Generator Network is proposed to explicitly separate and extract fine-grained reading skills and the implicitness or explicitness of the question. Specifically, the proposed skill leader accurately captures the semantic representation of fine-grained reading skills with contrastive learning. And the implicitness-aware pointer-generator adaptively extracts or generates the answer based on the implicitness or explicitness of the question. Furthermore, to validate the generalizability of the methodology, we annotate a new dataset named NarrativeQA 1.1. Experiments on the FairytaleQA and NarrativeQA 1.1 show that the proposed model achieves the state-of-the-art performance (about 5% gain on Rouge-L) on the question answering task. Our annotated data and code are available at https://github.com/pengwei-iie/Leader-Generator-Net.
What If: Generating Code to Answer Simulation Questions in Chemistry Texts
Authors: Gal Peretz, Mousa Arraf, Kira Radinsky
Venue: SIGIR
Year: 2023

Many texts, especially in chemistry and biology, describe complex processes. We focus on texts that describe a chemical reaction process and questions that ask about the process's outcome under different environmental conditions. To answer questions about such processes, one needs to understand the interactions between the different entities involved in the process and simulate their state transitions during the process execution under other conditions. We hypothesize that generating code and executing it to simulate the process will allow answering such questions. We, therefore, define a domain-specific language (DSL) to represent processes. We contribute to the community a unique dataset curated by chemists and annotated by computer scientists. The dataset is composed of process texts, simulation questions, and their corresponding computer codes represented by the DSL. We propose a neural program synthesis approach based on reinforcement learning with a novel state-transition semantic reward. The novel reward is based on the run-time semantic similarity between the predicted code and the reference code. This allows simulating complex process transitions and thus answering simulation questions. Our approach yields a significant boost in accuracy for simulation questions: we achieved 88% accuracy as opposed to 83% accuracy of the state-of-the-art neural program synthesis approaches and 54% accuracy of state-of-the-art end-to-end text-based approaches.
KATIE: A System for Key Attributes Identification in Product Knowledge Graph Construction
Authors: Btissam Er-Rahmadi, Arturo Oncevay, Yuanyi Ji, Jeff Z. Pan
Venue: SIGIR
Year: 2023

We present part of Huawei's efforts in building a Product Knowledge Graph (PKG). We want to identify which product attributes (i.e. properties) are relevant and important in terms of shopping decisions to product categories (i.e. classes). This is particularly challenging when the attributes and their values are mined from online product catalogues, i.e. HTML pages. These web pages contain semi-structured data, which do not follow a concerted format and use diverse vocabulary to designate the same features. We propose a system for key attribute identification (KATIE) based on fine-tuning pre-trained models (e.g., DistilBERT) to predict the applicability and importance of an attribute to a category. We also propose an attribute synonyms identification module that allows us to discover synonymous attributes by considering not only their labels' similarities but also the similarity of their values sets. We have evaluated our approach to Huawei categories taxonomy and a set of internally mined attributes from web pages. KATIE guarantees promising performance results compared to the most recent baselines.