OpenRaiser/Intern-Atlas huggingface dataset:
Cross-reconstructed Augmentation for Dual-target Cross-domain Recommendation
Authors: Qingyang Mao, Qi Liu, Zhi Li, Likang Wu, Bing Lv, Zheng Zhang
Venue: SIGIR
Year: 2024

To alleviate the long-standing data sparsity issue in recommender systems, numerous studies in cross-domain recommendation (CDR) have been conducted to facilitate information transfer processes across domains. In recent years, dual-target CDR has been introduced to gain mutual improvements between two domains through more general bidirectional transfer rather than traditional one-way transit. Existing methods in dual-target CDR focus primarily on designing powerful encoders to learn representative cross-domain information, without tackling the fundamental issue of interaction data shortage. In this paper, we present CrossAug, a novel data augmentation approach to leverage interactions more efficiently in two domains. Specifically, we propose intra-domain and inter-domain augmentations based on cross-reconstructed representations in terms of sampled records. To reduce the harm of domain shift, we project domain-shared representations in two domains into a joint space with Householder transformations and apply center alignments. All these modules boost the utilization of interactions with little influence from negative transfer. Extensive experimental results over public datasets demonstrate the effectiveness of CrossAug and its components in dual-target CDR.
Dense Retrieval with Continuous Explicit Feedback for Systematic Review Screening Prioritisation
Authors: Xinyu Mao, Shengyao Zhuang, Bevan Koopman, Guido Zuccon
Venue: SIGIR
Year: 2024

The goal of screening prioritisation in systematic reviews is to identify relevant documents with high recall and rank them in early positions for review. This saves reviewing effort if paired with a stopping criterion, and speeds up review completion if performed alongside downstream tasks. Recent studies have shown that neural models have good potential on this task, but their time-consuming fine-tuning and inference discourage their widespread use for screening prioritisation. In this paper, we propose an alternative approach that still relies on neural models, but leverages dense representations and relevance feedback to enhance screening prioritisation, without the need for costly model fine-tuning and inference. This method exploits continuous relevance feedback from reviewers during document screening to efficiently update the dense query representation, which is then applied to rank the remaining documents to be screened. We evaluate this approach across the CLEF TAR datasets for this task. Results suggest that the investigated dense query-driven approach is more efficient than directly using neural models and shows promising effectiveness compared to previous methods developed on the considered datasets. Our code is available at https://github.com/ielab/dense-screening-feedback.
Unmasking Privacy: A Reproduction and Evaluation Study of Obfuscation-based Perturbation Techniques for Collaborative Filtering
Authors: Alex Martinez, Mihnea Tufis, Ludovico Boratto
Venue: SIGIR
Year: 2024

Recommender systems (RecSys) solve personalisation problems and therefore heavily rely on personal data - demographics, user preferences, user interactions - each baring important privacy risks. It is also widely accepted that in RecSys performance and privacy are at odds, with the increase of one resulting in the decrease of the other. Among the diverse approaches in privacy enhancing technologies (PET) for RecSys, perturbation stands out for its simplicity and computational efficiency. It involves adding noise to sensitive data, thus hiding its real value from an untrusted actor. We reproduce and test a set of four randomization-based perturbation techniques developed by Batmaz and Polat \citebatmaz2016randomization for privacy preserving collaborative filtering. While the framework presents great advantages - low computational requirements, several useful privacy-enhancing parameters - the supporting paper lacks conclusions drawn from empirical evaluation. We address this shortcoming by proposing - in absence of an implementation by the authors - our own implementation of the obfuscation framework. We then develop an evaluation framework to test the main assumption of the reference paper - that RecSys privacy and performance are competing goals. We extend this study to understand how much we can enhance privacy, within reasonable losses of the RecSys performance. We reproduce and test the framework for the more realistic scenario where only implicit feedback is available, using two well-known datasets (MovieLens-1M and Last.fm-1K), and several state-of-the-art recommendation algorithms (NCF and LightGCN from the Microsoft Recommenders public repository).
Bi-Objective Negative Sampling for Sensitivity-Aware Search
Authors: Jack McKechnie, Graham McDonald, Craig Macdonald
Venue: SIGIR
Year: 2024

Cross-encoders leverage fine-grained interactions between documents and queries for effective relevance ranking. Such ranking models are typically trained to satisfy the single objective of providing relevant information to the users. However, not all information should be made available. For example, documents containing sensitive information, such as personal or confidential information, should not be returned in the search results. Sensitivity-aware search (SAS) aims to develop retrieval models that can satisfy two objectives, namely: (1) providing the user with relevant search results, while (2) ensuring that no documents that contain sensitive information are included in the ranking. In this work, we propose three novel negative sampling strategies that enable cross-encoders to be trained to satisfy the bi-objective task of SAS. Additionally, we investigate and compare with filtering sensitive documents in ranking pipelines. Our experiments on a collection labelled for sensitivity show that our proposed negative sampling strategies lead to a ~37% increase in terms of cost-sensitive nDCG (nCSDCG) for SAS.
Multilingual Meta-Distillation Alignment for Semantic Retrieval
Authors: Meryem M&apos, hamdi, Jonathan May, Franck Dernoncourt, Trung Bui, Seunghyun Yoon
Venue: SIGIR
Year: 2024

Multilingual semantic retrieval involves retrieving semantically relevant content to a query irrespective of the language. Compared to monolingual and bilingual semantic retrieval, multilingual semantic retrieval requires a stronger alignment approach to pull the contents to be retrieved close to the representation of their corresponding queries, no matter their language combinations. Traditionally, this is achieved through more supervision in the form of multilingual parallel resources, which are expensive to obtain, especially for low-resource languages. In this work, on top of an optimization-based Model-Agnostic Meta-Learner (MAML), we propose a data-efficient meta-distillation approach: MAML-Align,1 specifically for low-resource multilingual semantic retrieval. Our approach simulates a gradual feedback loop from monolingual to bilingual and from bilingual to multilingual semantic retrieval. We systematically compare multilingual meta-distillation learning to different baselines and conduct ablation studies on the role of different sampling approaches in the meta-task construction. We show that MAML-Align's gradual feedback loop boosts the generalization to different languages, including zero-shot ones, better than naive fine-tuning and vanilla MAML.
A Study on Unsupervised Question and Answer Generation for Legal Information Retrieval and Precedents Understanding
Authors: Johny Moreira, Altigran S. da Silva, Edleno Silva de Moura, Leandro Bezerra Marinho
Venue: SIGIR
Year: 2024

Traditional retrieval systems are hardly adequate for Legal Research, mainly because only returning the documents related to a given query is usually insufficient. Legal documents are extensive, and we posit that generating questions about them and detecting the answers provided by these documents help the Legal Research journey. This paper presents a pipeline that relates Legal Questions with documents answering them. We align features generated by Large Language Models with traditional clustering methods to find convergent and divergent answers to the same legal matter. We performed a case study with 50 legal documents on the Brazilian judiciary system. Our pipeline found convergent and divergent answers to 23 major legal questions regarding the case law for daily fines in Civil Procedural Law. The pipeline manual evaluation shows it managed to group diverse similar answers to the same question with an average precision of 0.85. It also managed to detect two divergent legal matters with an average F1 Score of 0.94.
ScholarNodes: Applying Content-based Filtering to Recommend Interdisciplinary Communities within Scholarly Social Networks
Authors: Md Asaduzzaman Noor, Jason A. Clark, John W. Sheppard
Venue: SIGIR
Year: 2024

Detecting communities within dynamic academic social networks and connecting these community detection findings to search and retrieval interfaces presents a multifaceted challenge. We explore an information retrieval method that integrates both partition-based and similarity-based network analysis to identify and recommend communities within content-based datasets. Our prototype "ScholarNodes" web interface bridges the gap between community detection algorithms (Louvain, K-means, Spectral clustering) and the BM25 (Best Matching 25) ranking algorithm within a cohesive user interface. From free-text keyword queries, ScholarNodes recommends collaborations, identifies local and external researcher networks, and visualizes an interdisciplinarity graph for individual researchers using the OpenAlex dataset, a global collection of academic papers and authors. Beyond the specific information retrieval use case, we discuss the broader applicability of the methods to generic social network analysis, community detection, and recommender systems. Additionally, we delve into the technical aspects of generating topical terms, community alignment techniques, and interface design considerations for integrating community detection algorithms into a search experience.
On Backbones and Training Regimes for Dense Retrieval in African Languages
Authors: Akintunde Oladipo, Mofetoluwa Adeyemi, Jimmy Lin
Venue: SIGIR
Year: 2024

The effectiveness of dense retrieval models trained with multilingual language models as backbones has been demonstrated in multilingual and cross-lingual information retrieval contexts. The optimal choice of a backbone model for a given retrieval task is dependent on the target retrieval domain as well as the pre-training domain of available language models and their generalization capabilities, the availability of relevance judgements, etc. In this work, we study the impact of these factors on retrieval effectiveness for African languages using three multilingual benchmark datasets: Mr. TyDi, MIRACL, and the newly released CIRAL dataset. We compare the effectiveness of mBERT as a backbone for dense retrieval models against multilingual language models such as AfriBERTa and AfroXLMR, which are specialized for African languages. Furthermore, we examine the impact of different training regimes on the effectiveness of dense retrieval in different domains for African languages. Our findings show that the pre-training domain of the backbone LM plays a huge role in retrieval effectiveness, especially in the absence of retrieval training data. Code artifacts are available at https://github.com/castorini/afridpr_backbones.
FedUD: Exploiting Unaligned Data for Cross-Platform Federated Click-Through Rate Prediction
Authors: Wentao Ouyang, Rui Dong, Ri Tao, Xiangzheng Liu
Venue: SIGIR
Year: 2024

Click-through rate (CTR) prediction plays an important role in online advertising platforms. Most existing methods use data from the advertising platform itself for CTR prediction. As user behaviors also exist on many other platforms, e.g., media platforms, it is beneficial to further exploit such complementary information for better modeling user interest and for improving CTR prediction performance. However, due to privacy concerns, data from different platforms cannot be uploaded to a server for centralized model training. Vertical federated learning (VFL) provides a possible solution which is able to keep the raw data on respective participating parties and learn a collaborative model in a privacy-preserving way. However, traditional VFL methods only utilize aligned data with common keys across parties, which strongly restricts their application scope. In this paper, we propose FedUD, which is able to exploit unaligned data, in addition to aligned data, for more accurate federated CTR prediction. FedUD contains two steps. In the first step, FedUD utilizes aligned data across parties like traditional VFL, but it additionally includes a knowledge distillation module. This module distills useful knowledge from the guest party's high-level representations and guides the learning of a representation transfer network. In the second step, FedUD applies the learned knowledge to enrich the representations of the host party's unaligned data such that both aligned and unaligned data can contribute to federated model training. Experiments on two real-world datasets demonstrate the superior performance of FedUD for federated CTR prediction.
Pacer and Runner: Cooperative Learning Framework between Single- and Cross-Domain Sequential Recommendation
Authors: Chung Park, Taesan Kim, Hyungjun Yoon, Junui Hong, Yelim Yu, Mincheol Cho, Minsung Choi, Jaegul Choo
Venue: SIGIR
Year: 2024

Cross-Domain Sequential Recommendation (CDSR) improves recommendation performance by utilizing information from multiple domains, which contrasts with Single-Domain Sequential Recommendation (SDSR) that relies on a historical interaction within a specific domain. However, CDSR may underperform compared to the SDSR approach in certain domains due to negative transfer, which occurs when there is a lack of relation between domains or different levels of data sparsity. To address the issue of negative transfer, our proposed CDSR model estimates the degree of negative transfer of each domain and adaptively assigns it as a weight factor to the prediction loss, to control gradient flows through domains with significant negative transfer. To this end, our model compares the performance of a model trained on multiple domains (CDSR) with a model trained solely on the specific domain (SDSR) to evaluate the negative transfer of each domain using our asymmetric cooperative network. In addition, to facilitate the transfer of valuable cues between the SDSR and CDSR tasks, we developed an auxiliary loss that maximizes the mutual information between the representation pairs from both tasks on a per-domain basis. This cooperative learning between SDSR and CDSR tasks is similar to the collaborative dynamics between pacers and runners in a marathon. Our model outperformed numerous previous works in extensive experiments on two real-world industrial datasets across ten service domains. We also have deployed our model in the recommendation system of our personal assistant app service, resulting in 21.4% increase in click-through rate compared to existing models, which is valuable to real-world business1.
SLH-BIA: Short-Long Hawkes Process for Buy It Again Recommendations at Scale
Authors: Rankyung Park, Amit Pande, David Relyea, Pushkar Chennu, Prathyusha Kanmanth Reddy
Venue: SIGIR
Year: 2024

Buy It Again (BIA) recommendations are a crucial component in enhancing the customer experience and site engagement for retailers. In this paper, we build a short (S) and long (L) term Hawkes (H) process for each item and use it to obtain BIA recommendations for each customer. The challenges of deploying into a production environment including model scalability, an evolving item catalog, and real-time inference are discussed along with solutions such as model compression, frequency-based item filtering, training data sampling, data parallelization, parallel execution and microservice-based real-time recommendations. We significantly reduced model training time from roughly 250 hours to about 3 hours by applying the solutions, while serving real-time inference with less than 70ms latency. We compare our BIA model against state-of-the-art baselines using three publicly available datasets and provide results from A/B tests with millions of live customers. On 3 public datasets, our model outperforms SOTA baseline models in recall and NDCG metrics by around 85% and 10%, respectively, and in live A/B testing it exhibited more than 30% increase in click-through rate and roughly 30% revenue increase compared to other state of the art models.
Axiomatic Guidance for Efficient and Controlled Neural Search
Authors: Andrew Parry
Venue: SIGIR
Year: 2024

Pre-trained language models based on the transformer architecture provide solutions to general ad-hoc search tasks--ranging from news search to question-answering--vastly outperforming statistical approaches in terms of both precision and recall. These models operate over "semantics'', removing the need for bespoke features based on proprietary data (e.g., interaction logs). In doing so, this paradigm may lead to further adoption of the idealised "end-to-end'' retrieval system as an elegant and powerful search solution. However, outside of sanitised benchmarks, these models present exploitable and untrustworthy biases relinquishing any control over inference due to their black-box nature. Such biases threaten the viability of neural models in production. Without greater control over model output, stakeholders could raise concerns hindering the adoption of effective and efficient search. Today, feature-based search systems are still performant relative to state-of-the-art neural search and can adapt to a changing corpus and the needs of system stakeholders. As agency over information access is further reduced via emerging paradigms such as Retrieval-Augmented-Generation, we must retain control over the output of a search system. We consider that bias in neural search systems is an artefact of the training and underlying mechanisms of current pre-trained models but is not present in statistical models. Features such as statistical models are principled and arbitrarily controllable; these features can adapt to a corpus and meet the demands of a given search task. Conversely, the output of a current neural system can only be changed by post hoc constraints or by re-training the underlying model. We posit that by allowing external features to influence the semantic interactions within neural search at inference time, we can not only allow control over system output but reduce the need to model corpus-specific priors, which can instead be modelled by external features, allowing for greater generalisation and training efficiency gains. We aim to reduce the complexity of neural ranker training and inference, applying classical IR principles and systems that align with such principles as a generalisable process as opposed to the ad-hoc constraint of prior work. Such an approach can reduce the need for larger models whilst improving generalisation. Axiomatic signals can guide and control neural ranking models to reduce spurious factors in semantic relevance estimation by compensating for the frozen priors of neural systems whilst still operating over flexible latent space. Given the biases observed in current systems, this may satiate the concerns of multiple stakeholders, leading to broader adoption of the paradigm.
IR-RAG @ SIGIR24: Information Retrieval's Role in RAG Systems
Authors: Fabio Petroni, Federico Siciliano, Fabrizio Silvestri, Giovanni Trappolini
Venue: SIGIR
Year: 2024

In recent years, Retrieval Augmented Generation (RAG) systems have emerged as a pivotal component in the field of artificial intelligence, gaining significant attention and importance across various domains. These systems, which combine the strengths of information retrieval and generative models, have shown promise in enhancing the capabilities and performance of machine learning applications. However, despite their growing prominence, RAG systems are not without their limitations and continue to be in need of exploration and improvement. This workshop seeks to focus on the critical aspect of information retrieval and its integral role within RAG frameworks. We argue that current efforts have undervalued the role of Information Retrieval (IR) in the RAG and have concentrated their attention on the generative part. As the cornerstone of these systems, IR's effectiveness dramatically influences the overall performance and outcomes of RAG models. We call for papers that will seek to revisit and emphasize the fundamental principles underpinning RAG systems. At the end of the workshop, we aim to have a clearer understanding of how robust information retrieval mechanisms can significantly enhance the capabilities of RAG systems. The workshop will serve as a platform for experts, researchers, and practitioners. We intend to foster discussions, share insights, and encourage research that underscores the vital role of Information Retrieval in the future of generative systems.
Shadowfax: Harnessing Textual Knowledge Base Population
Authors: Maxime Prieur, Cédric du Mouza, Guillaume Gadek, Bruno Grilhères
Venue: SIGIR
Year: 2024

Knowledge base population (KBP) from texts involves the extraction and organization of information from unstructured textual data to enhance or create a structured knowledge base. This process is crucial for various applications, such as natural language understanding, question-answering systems, and knowledge-driven decision-making. However the difficulty lies in the complexity of natural language, which is nuanced, ambiguous, and context-dependent. Extracting accurate and reliable information requires overcoming challenges such as entity disambiguation and relation extraction which are time-consuming tasks for users.Shadowfax is an interactive platform designed to support users by streamlining the process of knowledge base population (KPB) from text documents. Unlike other existing tools, it relies on a unified machine learning model to extract relevant information from unstructured text, enabling operational agents to gain a quick overview. The proposed system supports a variety of natural language processing (NLP) tasks using a single architecture, while presenting information in the most comprehensive way possible to the end user.
LLM4Eval: Large Language Model for Evaluation in IR
Authors: Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L. A. Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, Emine Yilmaz
Venue: SIGIR
Year: 2024

Large language models (LLMs) have demonstrated increasing task-solving abilities not present in smaller models. Utilizing the capabilities and responsibilities of LLMs for automated evaluation (LLM4Eval) has recently attracted considerable attention in multiple research communities. For instance, LLM4Eval models have been studied in the context of automated judgments, natural language generation, and retrieval augmented generation systems. We believe that the information retrieval community can significantly contribute to this growing research area by designing, implementing, analyzing, and evaluating various aspects of LLMs with applications to LLM4Eval tasks. The main goal of LLM4Eval workshop is to bring together researchers from industry and academia to discuss various aspects of LLMs for evaluation in information retrieval, including automated judgments, retrieval-augmented generation pipeline evaluation, altering human evaluation, robustness, and trustworthiness of LLMs for evaluation in addition to their impact on real-world applications. We also plan to run an automated judgment challenge prior to the workshop, where participants will be asked to generate labels for a given dataset while maximising correlation with human judgments. The format of the workshop is interactive, including roundtable and keynote sessions and tends to avoid the one-sided dialogue of a mini-conference.