OpenRaiser/Intern-Atlas huggingface dataset:
An Empirical Analysis on Multi-turn Conversational Recommender Systems
Authors: Lu Zhang, Chen Li, Yu Lei, Zhu Sun, Guanfeng Liu
Venue: SIGIR
Year: 2024

The rise of conversational recommender systems (CRSs) brings the evolution of the recommendation paradigm, which enables users to interact with the system and achieve dynamic recommendations. As one essential branch, multi-turn CRSs, built on the user simulator paradigm, have attracted great attention due to their powerful ability to accomplish recommendations without real dialogue resources. Recent multi-turn CRS models, equipped with various delicately designed components (e.g., conversation module), achieve state-of-the-art (SOTA) performance. We, for the first time, propose a comprehensive experimental evaluation for existing SOTA multi-turn CRSs to investigate three research questions: (1) reproducibility - are the designed components beneficial to target multi-turn CRSs? (2) scenario-specific adaptability - how do these components perform in various scenarios? and (3) generality - can the effective components from the target CRS be effectively transferred to other multi-turn CRSs? To answer these questions, we design and conduct experiments under different settings, including carefully selected SOTA baselines, components of CRSs, datasets, and evaluation metrics, thus providing an experimental aspect overview of multi-turn CRSs. As a result, we derive several significant insights whereby effective guidelines are provided for future multi-turn CRS model designs across diverse scenarios.
CIQA: A Coding Inspired Question Answering Model
Authors: Mousa Arraf, Kira Radinsky
Venue: SIGIR
Year: 2024

Methods in question-answering (QA) that transform texts detailing processes into an intermediate code representation, subsequently executed to generate a response to the presented question, have demonstrated promising results in analyzing scientific texts that describe intricate processes. The limitations of these existing text-to-code models are evident when attempting to solve QA problems that require knowledge beyond what is presented in the input text. We propose a novel domain-agnostic model to address the problem by leveraging domain-specific and open-source code libraries. We introduce an innovative QA text-to-code algorithm that learns to represent and utilize external APIs from code repositories, such as GitHub, within the intermediate code representation. The generated code is then executed to answer a question about a text. We present three QA datasets, focusing on scientific problems in the domains of chemistry, astronomy, and biology, for the benefit of the community. Our study demonstrates that our proposed method is a competitive alternative to current state-of-the-art (SOTA) QA text-to-code models and generic SOTA QA models.
Inferring Climate Change Stances from Multimodal Tweets
Authors: Nan Bai, Ricardo da Silva Torres, Anna Fensel, Tamara Metze, Art Dewulf
Venue: SIGIR
Year: 2024

Climate change is a heated discussion topic in public arenas such as social media. Both texts and visuals play key roles in the debate, as they can complement, contradict, or reinforce each other in nuanced ways. It is therefore urgently needed to study the messages as multimodal objects to better understand the polarized debate about climate change impacts and policies. Multimodal representation models such as CLIP are known to be able to transfer knowledge across domains and modalities, enabling the investigation of textual and visual semantics together. Yet they are not directly able to distinguish the nuances between supporting and sceptic climate change stances. This paper explores a simple but effective strategy combining modality fusion and domain-knowledge enhancing to prepare CLIP-based models with knowledge of climate change stances. A multimodal Dutch Twitter dataset is collected and experimented with the proposed strategy, which increased the macro-average F1 score across stances from 51% to 86%. The outcomes can be applied in both data science and public policy studies, to better analyse how the combined use of texts and visuals generates meanings during debates, in the context of climate change and beyond.
Large Language Models for Recommendation: Past, Present, and Future
Authors: Keqin Bao, Jizhi Zhang, Xinyu Lin, Yang Zhang, Wenjie Wang, Fuli Feng
Venue: SIGIR
Year: 2024

Large language models (LLMs) have significantly influenced recommender systems, spurring interest across academia and industry in leveraging LLMs for recommendation tasks. This includes using LLMs for generative item retrieval and ranking, and developing versatile LLMs for various recommendation tasks, potentially leading to a paradigm shift in the field of recommender systems. This tutorial aims to demystify the Large Language Model for Recommendation (LLM4Rec) by reviewing its evolution and delving into cutting-edge research. We will explore how LLMs enhance recommender systems in terms of architecture, learning paradigms, and functionalities such as conversational abilities, generalization, planning, and content generation. The tutorial will shed light on the challenges and open problems in this burgeoning field, including trustworthiness, efficiency, online training, and evaluation of LLM4Rec. We will conclude by summarizing key learnings from existing studies and outlining potential avenues for future research, with the goal of equipping the audience with a comprehensive understanding of LLM4Rec and inspiring further exploration in this transformative domain.
International Workshop on Algorithmic Bias in Search and Recommendation (BIAS)
Authors: Alejandro Bellogín, Ludovico Boratto, Styliani Kleanthous, Elisabeth Lex, Francesca Maridina Malloci, Mirko Marras
Venue: SIGIR
Year: 2024

Creating efficient and effective search and recommendation algorithms has been the main objective of industry practitioners and academic researchers over the years. However, recent research has shown how these algorithms trained on historical data lead to models that might exacerbate existing biases and generate potentially negative outcomes. Defining, assessing, and mitigating these biases throughout experimental pipelines is a primary step for devising search and recommendation algorithms that can be responsibly deployed in real-world applications. This workshop aims to collect novel contributions in this field and offer a common ground for interested researchers and practitioners. More information about the workshop is available at https://biasinrecsys.github.io/sigir2024/
The Second Workshop on Large Language Models for Individuals, Groups, and Society
Authors: Michael Bendersky, Cheng Li, Qiaozhu Mei, Vanessa Murdock, Jie Tang, Hongning Wang, Hamed Zamani, Mingyang Zhang, Xingjian Zhang
Venue: SIGIR
Year: 2024

This is the second workshop in the series which discusses the cutting-edge developments in research and applications of personalizing large language models (LLMs) and adapting them to the demands of diverse user populations and societal needs. The full-day workshop plan includes several keynotes and invited talks, a poster session and a panel discussion.
Gen-IR @ SIGIR 2024: The Second Workshop on Generative Information Retrieval
Authors: Gabriel Bénédict, Ruqing Zhang, Donald Metzler, Andrew Yates, Ziyan Jiang
Venue: SIGIR
Year: 2024

Generative information retrieval (Gen-IR) is a fast-growing interdisciplinary research area that investigates how to leverage advances in generative Artificial Intelligence (AI) to improve information retrieval systems. Gen-IR has attracted interest from the information retrieval, natural language processing, and machine learning communities, among others. Since the dawn of Gen-IR last year, there has been an explosion of Gen-IR systems that have launched and are now widely used. Interest in this area across academia and industry is only expected to continue to grow as new research challenges and application opportunities arise. The goal of this proposed workshop, The Second Workshop on Generative Information Retrieval (Gen-IR @ SIGIR 2024) is to provide an interactive venue for exploring a broad range of foundational and applied Gen-IR research. The workshop will focus on tasks such as generative document retrieval, grounded answer generation, generative recommendation, and generative knowledge graphs, all through the lens of model training, model behavior, and broader issues. The workshop will be highly interactive, favoring panel discussions, poster sessions, and roundtable discussions over one-sided keynotes and paper talks.
Relevance Feedback Method For Patent Searching Using Vector Subspaces
Authors: Sebastian Björkqvist
Venue: SIGIR
Year: 2024

Searching for novelty-destroying prior art is an important part of patent application drafting and invalidation. The task is challenging due to the detailed information needed to determine whether a document is novelty-destroying or simply closely related, resulting in the original search results not always being fully on target. Allowing the user to provide feedback on the relevance of the initial search results and iterating on the search may thus improve the results significantly. We present a relevance feedback method based on computing the affine vector subspace spanned by the relevant document vectors. The method can be used with any dense retrieval system, and we demonstrate its effectiveness in improving recall in prior art searches. We compare the subspace-based method to the Rocchio algorithm and show that the method is less sensitive to changes in hyperparameters when the number of relevant documents increases.
Personalized Large Language Models through Parameter Efficient Fine-Tuning Techniques
Authors: Marco Braga
Venue: SIGIR
Year: 2024

Personalization of the search experience according to the users and their context is an important topic in Information Retrieval (IR), studied by the research community for a long time. The IR field has witnessed a transformation with the recent availability of pre-trained Large Language Models. Typically, personalization requires the model to incorporate user-specific information, through the definition of an appropriate prompting or injecting user knowledge into the model and then fine-tuning it. However, using prompting, we do not know where and how much the model is personalizing the output. Furthermore, fine-tuning such systems is computationally expensive: since they are characterized by billions of parameters, the fine-tuning process has introduced profound computational challenges. For these reasons, we propose a novel approach that combines personalization and Parameter Efficient Fine-Tuning methods.
Dataset and Models for Item Recommendation Using Multi-Modal User Interactions
Authors: Simone Borg Bruun, Krisztian Balog, Maria Maistro
Venue: SIGIR
Year: 2024

While recommender systems with multi-modal item representations (image, audio, and text), have been widely explored, learning recommendations from multi-modal user interactions (e.g., clicks and speech) remains an open problem. We study the case of multi-modal user interactions in a setting where users engage with a service provider through multiple channels (website and call center). In such cases, incomplete modalities naturally occur, since not all users interact through all the available channels. To address these challenges, we publish a real-world dataset that allows progress in this under-researched area. We further present and benchmark various methods for leveraging multi-modal user interactions for item recommendations, and propose a novel approach that specifically deals with missing modalities by mapping user interactions to a common feature space. Our analysis reveals important interactions between the different modalities and that a frequently occurring modality can enhance learning from a less frequent one.
Counterfactual Ranking Evaluation with Flexible Click Models
Authors: Alexander Buchholz, Ben London, Giuseppe Di Benedetto, Jan Malte Lichtenberg, Yannik Stein, Thorsten Joachims
Venue: SIGIR
Year: 2024

Evaluating a new ranking policy using data logged by a previously deployed policy requires a counterfactual (off-policy) estimator that corrects for presentation and selection biases. Some estimators (e.g., the position-based model) perform this correction by making strong assumptions about user behavior, which can lead to high bias if the assumptions are not met. Other estimators (e.g., the item-position model) rely on randomization to avoid these assumptions, but they often suffer from high variance. In this paper, we develop a new counterfactual estimator, called Interpol, that provides a tunable trade-off in the assumptions it makes, thus providing a novel ability to optimize the bias-variance trade-off. We analyze the bias of our estimator, both theoretically and empirically, and show that it achieves lower error than both the position-based model and the item-position model, on both synthetic and real datasets. This improvement in accuracy not only benefits offline evaluation of ranking policies, we also find that Interpol improves learning of new ranking policies when used as the training objective for learning-to-rank.
ScienceDirect Topic Pages: A Knowledge Base of Scientific Concepts Across Various Science Domains
Authors: Artemis Çapari, Hosein Azarbonyad, Georgios Tsatsaronis, Zubair Afzal, Judson Dunham
Venue: SIGIR
Year: 2024

From undergraduate students to renowned scholars, everyone occasionally encounters unknown concepts within their field of interest, especially when reading scientific articles. ScienceDirectTopic Pages (TP) are intended to facilitate learning and to provide users with a structured overview of sources to deepen their knowledge about such unfamiliar topics. Our free service provides insight into a vast set of technical topics across 20 different scientific domains. Designed to emulate the natural flow of learning, TPs are embedded within millions of articles so that users can click on unfamiliar concepts they come across whilst reading an article. This redirects the user to a TP, consisting of a definition of the concept, which provides the user with a basic understanding of the concept. The TP further presents a collection of relevant snippets extracted from books and review articles published by ScienceDirect for users interested in references and more detailed explanations and applications of the concept. Finally, a set of related topics is provided to extend the user's knowledge even further. To build TPs, we utilize various information retrieval methods across our product. We retrieve the most relevant snippets for each topic/concept using a semantic search model fine-tuned on our scientific database. We further leverage the power of Retrieval Augmented Generation to generate reliable definitions on the topics sourced from ScienceDirect's content. To retrieve a list of relevant concepts for each topic, we use the co-occurrence statistics of concepts within books and articles.
Neural Passage Quality Estimation for Static Pruning
Authors: Xuejun Chang, Debabrata Mishra, Craig Macdonald, Sean MacAvaney
Venue: SIGIR
Year: 2024

Neural networks-especially those that use large, pre-trained language models-have improved search engines in various ways. Most prominently, they can estimate the relevance of a passage or document to a user's query. In this work, we depart from this direction by exploring whether neural networks can effectively predict which of a document's passages are unlikely to be relevant to any query submitted to the search engine.We refer to this query-agnostic estimation of passage relevance as a passage's quality.We find that our novel methods for estimating passage quality allow passage corpora to be pruned considerably while maintaining statistically equivalent effectiveness; our best methods can consistently prune >25% of passages in a corpora, across various retrieval pipelines. Such substantial pruning reduces the operating costs of neural search engines in terms of computing resources, power usage, and carbon footprint-both when processing queries (thanks to a smaller index size) and when indexing (lightweight models can prune low-quality passages prior to the costly dense or learned sparse encoding step). This work sets the stage for developing more advanced neural "learning-what-to-index" methods.
Enhancing Dataset Search with Compact Data Snippets
Authors: Qiaosheng Chen, Jiageng Chen, Xiao Zhou, Gong Cheng
Venue: SIGIR
Year: 2024

In light of the growing availability and significance of open data, the problem of dataset search has attracted great attention in the field of information retrieval. Nevertheless, current metadata-based approaches have revealed shortcomings due to the low quality and availability of dataset metadata, while the magnitude and heterogeneity of actual data hindered the development of content-based solutions. To address these challenges, we propose to convert different formats of structured data into a unified form, from which we extract a compact data snippet that indicates the relevance of the whole data. Thanks to its compactness, we feed it into a dense reranker to improve search accuracy. We also convert it back to the original format to be presented for assisting users in relevance judgment. The effectiveness of our approach has been demonstrated by extensive experiments on two test collections for dataset search.
Evaluating Search System Explainability with Psychometrics and Crowdsourcing
Authors: Catherine Chen, Carsten Eickhoff
Venue: SIGIR
Year: 2024

As information retrieval (IR) systems, such as search engines and conversational agents, become ubiquitous in various domains, the need for transparent and explainable systems grows to ensure accountability, fairness, and unbiased results. Despite recent advances in explainable AI and IR techniques, there is no consensus on the definition of explainability. Existing approaches often treat it as a singular notion, disregarding the multidimensional definition postulated in the literature. In this paper, we use psychometrics and crowdsourcing to identify human-centered factors of explainability in Web search systems and introduce SSE (Search System Explainability), an evaluation metric for explainable IR (XIR) search systems. In a crowdsourced user study, we demonstrate SSE's ability to distinguish between explainable and non-explainable systems, showing that systems with higher scores indeed indicate greater interpretability. We hope that aside from these concrete contributions to XIR, this line of work will serve as a blueprint for similar explainability evaluation efforts in other domains of machine learning and natural language processing.
IdmGAE: Importance-Inspired Dynamic Masking for Graph Autoencoders
Authors: Ge Chen, Yulan Hu, Sheng Ouyang, Zhirui Yang, Yong Liu, Cuicui Luo
Venue: SIGIR
Year: 2024

Generative self-supervised learning, exemplified by masked graph autoencoders (GAEs), aims to reconstruct the masked graph characteristics, garnering increasing research interest and achieving promising results. Despite the progress made, existing research on masked GAEs predominantly relies on a random strategy to obscure input graph characteristics, disregarding the potential variability in node importance during the reconstruction process. In this study, we propose an efficient masking strategy termed importance-inspired dynamic masking to explore diverse node sampling. Our approach employs an auxiliary network to assess the importance of nodes within the graph. Subsequently, we derive sampling probabilities guided by the learned node importance. To enhance model training, we introduce a dynamic sampling strategy that adapts to nodes of varying importance across different training stages. This importance-inspired dynamic masking strategy empowers masked GAEs to acquire refined representations for graph-related tasks. We have conducted extensive experiments on node classification, which validate the efficacy of our proposed strategy.
ACORDAR 2.0: A Test Collection for Ad Hoc Dataset Retrieval with Densely Pooled Datasets and Question-Style Queries
Authors: Qiaosheng Chen, Weiqing Luo, Zixian Huang, Tengteng Lin, Xiaxia Wang, Ahmet Soylu, Basil Ell, Baifan Zhou, Evgeny Kharlamov, Gong Cheng
Venue: SIGIR
Year: 2024

Dataset search, or more specifically, ad hoc dataset retrieval which is a trending specialized IR task, has received increasing attention in both academia and industry. While methods and systems continue evolving, existing test collections for this task exhibit shortcomings, particularly suffering from lexical bias in pooling and limited to keyword-style queries for evaluation. To address these limitations, in this paper, we construct ACORDAR 2.0, a new test collection for this task which is also the largest to date. To reduce lexical bias in pooling, we adapt dense retrieval models to large structured data, using them to find an extended set of semantically relevant datasets to be annotated. To diversify query forms, we employ a large language model to rewrite keyword queries into high-quality question-style queries. We use the test collection to evaluate popular sparse and dense retrieval models to establish a baseline for future studies. The test collection and source code are publicly available.
Unbiased Validation of Technology-Assisted Review for eDiscovery
Authors: Gordon V. Cormack, Maura R. Grossman, Andrew Harbison, Tom O&apos, Halloran, Bronagh McManus
Venue: SIGIR
Year: 2024

Although it is well established that recall estimates are valid only when based on independent relevance assessments, and useful only to compare the relative effectiveness of competing methods, these conditions are seldom met when validating eDiscovery efforts in litigation. We present two unbiased validation strategies that embed blind relevance assessments into a technology-assisted review (TAR) process, so as to compare its recall to that which would have been achieved by exhaustive manual review. We illustrate the use of these strategies within the context of TAR occasioned by litigation over accounting practices preceding the collapse of a major insurance company.
The Treatment of Ties in Rank-Biased Overlap
Authors: Matteo Corsi, Julián Urbano
Venue: SIGIR
Year: 2024

Rank-Biased Overlap (RBO) is a similarity measure for indefinite rankings: it is top-weighted, and can be computed when only a prefix of the rankings is known or when they have only some items in common. It is widely used for instance to analyze differences between search engines by comparing the rankings of documents they retrieve for the same queries. In these situations, though, it is very frequent to find tied documents that have the same score. Unfortunately, the treatment of ties in RBO remains superficial and incomplete, in the sense that it is not clear how to calculate it from the ranking prefixes only. In addition, the existing way of dealing with ties is very different from the one traditionally followed in the field of Statistics, most notably found in rank correlation coefficients such as Kendall's and Spearman's. In this paper we propose a generalized formulation for RBO to handle ties, thanks to which we complete the original definitions by showing how to perform prefix evaluation. We also use it to fully develop two variants that align with the ones found in the Statistics literature: one when there is a reference ranking to compare to, and one when there is not. Overall, these three variants provide researchers with flexibility when comparing rankings with RBO, by clearly determining what ties mean, and how they should be treated. Finally, using both synthetic and TREC data, we demonstrate the use of these new tie-aware RBO measures. We show that the scores may differ substantially from the original tie-unaware RBO measure, where ties had to be broken at random or by arbitrary criteria such as by document ID. Overall, these results evidence the need for a proper account of ties in rank similarity measures such as RBO.