OpenRaiser/Intern-Atlas huggingface dataset:
Old IR Methods Meet RAG
Authors: Oz Huly, Idan Pogrebinsky, David Carmel, Oren Kurland, Yoelle Maarek
Venue: SIGIR
Year: 2024

Retrieval augmented generation (RAG) is an important approach to provide large language models (LLMs) with context pertaining to the text generation task: given a prompt, passages are retrieved from external corpora to ground the generation with more relevant and/or fresher data. Most previous studies used dense retrieval methods for applying RAG in question answering scenarios. However, recent work showed that traditional information retrieval methods (a.k.a. sparse methods) can do as well as or even better than dense retrieval ones. In particular, it was shown that Okapi BM25 outperforms dense retrieval methods, in terms of perplexity, for the fundamental text completion task in LLMs. We extend this study and show, using two popular LLMs, that a broad set of sparse retrieval methods achieve better results than all the dense retrieval methods we experimented with, for varying lengths of queries induced from the prompt. Furthermore, we found that Okapi BM25 is substantially outperformed by a term-proximity retrieval method (MRF), which is in turn outperformed by a pseudo-feedback-based bag-of-terms approach (relevance model). Additional exploration sheds some light on the effectiveness of lexical retrieval methods for RAG. Our findings call for further study of classical retrieval methods for RAG.
MANILA24: SIGIR 2024 Workshop on Information Retrieval and Climate Impact
Authors: Bart van den Hurk, Maarten de Rijke, Flora D. Salim
Venue: SIGIR
Year: 2024

The MANILA24 workshop brings together researchers and practitioners from academia, industry, governments, and NGO's to identify and discuss core research problems in information retrieval for climate impact. The workshop aims to foster collaboration by bringing communities together that have so far not been very well connected -- IR, systematic reviews, and climate change. The purpose is to help accelerate the development of IR technology that supports our understanding of climate impact publications and the articulation of recommended actions. Importantly, this includes introducing IR researchers climate impact, introducing researchers in climate to state-of-the-art IR technology, and developing a shared research agenda.
Multi-Domain Sequential Recommendation via Domain Space Learning
Authors: Junyoung Hwang, Hyunjun Ju, SeongKu Kang, Sanghwan Jang, Hwanjo Yu
Venue: SIGIR
Year: 2024

This paper explores Multi-Domain Sequential Recommendation (MDSR), an advancement of Multi-Domain Recommendation that incorporates sequential context. Recent MDSR approach exploits domain-specific sequences, decoupled from mixed-domain histories, to model domain-specific sequential preference, and use mixeddomain histories to model domain-shared sequential preference. However, the approach faces challenges in accurately obtaining domain-specific sequential preferences in the target domain, especially when users only occasionally engage with it. In such cases, the history of users in the target domain is limited or not recent, leading the sequential recommender system to capture inaccurate domain-specific sequential preferences. To address this limitation, this paper introduces Multi-Domain Sequential Recommendation via Domain Space Learning (MDSR-DSL). Our approach utilizes cross-domain items to supplement missing sequential context in domain-specific sequences. It involves creating a "domain space" to maintain and utilize the unique characteristics of each domain and a domain-to-domain adaptation mechanism to transform item representations across domain spaces. To validate the effectiveness of MDSR-DSL, this paper extensively compares it with state-of-the-art MD(S)R methods and provides detailed analyses.
Fairness-Aware Exposure Allocation via Adaptive Reranking
Authors: Thomas Jänich, Graham McDonald, Iadh Ounis
Venue: SIGIR
Year: 2024

In the first stage of a re-ranking pipeline, an inexpensive ranking model is typically deployed to retrieve a set of documents that are highly likely to be relevant to the user's query. The retrieved documents are then re-ranked by a more effective but expensive ranking model, e.g., a deep neural ranker such as BERT. However, in such a standard pipeline, no new documents are typically discovered after the first stage retrieval. Hence, the amount of exposure that a particular group of documents - e.g., documents from a particular demographic category - can receive is limited by the number of documents that are retrieved in the first stage retrieval. Indeed, if too few documents from a group are retrieved in the first stage retrieval, ensuring that the group receives a fair amount of exposure to the user may become infeasible. Therefore, it is useful to identify more documents from underrepresented groups that are potentially relevant to the query during the re-ranking stage. In this work, we investigate how deploying adaptive re-ranking, which enables the discovery of additional potentially relevant documents in the re-ranking stage, can improve the exposure that a given group of documents receives in the final ranking. We propose six adaptive re-ranking policies that can discover documents from underrepresented groups to increase the disadvantaged groups' exposure in the final ranking. Our experiments on the TREC 2021 and 2022 Fair Ranking Track test collections show that our policies consistently improve the fairness of the exposure distribution in the final ranking, compared to standard adaptive re-ranking approaches, resulting in increases of up to ~13% in Attention Weighted Ranked Fairness (AWRF). Moreover, our best performing policy, Policy 6, consistently maintains and frequently increases the utility of the search results in terms of nDCG.
GATS: Generative Audience Targeting System for Online Advertising
Authors: Cong Jiang, Zhongde Chen, Bo Zhang, Yankun Ren, Xin Dong, Lei Cheng, Xinxing Yang, Longfei Li, Jun Zhou, Linjian Mo
Venue: SIGIR
Year: 2024

This paper presents GATS (Generative Audience Targeting S ystem for Online Advertising), a new framework using large language models (LLMs) to improve audience targeting in online advertising. GATS overcomes the shortcomings of rule-based, look-alike, and graph-based methods by facilitating flexible and interpretable audience criteria expression. The framework integrates intent recognition, knowledge mining, and Data Management Platform (DMP) mapping to translate advertiser demands into actionable user tags and correlate them within a DMP. A small, white-box model called LightGATS (base on QWen-14B), fine-tuned with a high-quality LLM corpus, ensures the framework's safety and efficiency, operating within a scalable hybrid online-offline architecture. GATS's effectiveness is validated through extensive experiments, marking a significant advancement in audience targeting technology.
A Semantic Search Engine for Helping Patients Find Doctors and Locations in a Large Healthcare Organization
Authors: Mayank Kejriwal, Hamid Haidarian, Min-Hsueh Chiu, Andy Xiang, Deep Shrestha, Faizan Javed
Venue: SIGIR
Year: 2024

Efficiently finding doctors and locations (FDL) is an important search problem for patients in the healthcare domain, for which traditional information retrieval (IR) methods tend to be sub-optimal. This paper introduces and defines FDL as an important healthcare industry-specific problem in IR. We then propose a semantic search engine as a robust solution to FDL in Kaiser Permanente (KP), a large healthcare organization with 12 million members. Our solution meets practical needs of data security and privacy, scalability, cost-effectiveness, backward compatibility with existing indexes and search infrastructure, and interpretability of outputs for patients. It uses a concept-rich ontology to model raw data from multiple sources as entities, relations, and attributes in a knowledge graph that is stored and indexed in an industry-scale graph database. We evaluate the solution on a real patient-query log and demonstrate its practical utility. The system has been implemented and deployed live to KP customers.
Retrieval-Augmented Conversational Recommendation with Prompt-based Semi-Structured Natural Language State Tracking
Authors: Sara Kemper, Justin Cui, Kai Dicarlantonio, Kathy Lin, Danjie Tang, Anton Korikov, Scott Sanner
Venue: SIGIR
Year: 2024

Conversational recommendation (ConvRec) systems must understand rich and diverse natural language (NL) expressions of user preferences and intents, often communicated in an indirect manner (e.g., "I'm watching my weight''). Such complex utterances make retrieving relevant items challenging, especially if only using often incomplete or out-of-date metadata. Fortunately, many domains feature rich item reviews that cover standard metadata categories and offer complex opinions that might match a user's interests (e.g., "classy joint for a date''). However, only recently have large language models (LLMs) let us unlock the commonsense connections between user preference utterances and complex language in user-generated reviews. Further, LLMs enable novel paradigms for semi-structured dialogue state tracking, complex intent and preference understanding, and generating recommendations, explanations, and question answers. We thus introduce a novel technology RA-Rec, a Retrieval-Augmented, LLM-driven dialogue state tracking system for ConvRec, showcased with a video, open source GitHub repository, and interactive Google Colab notebook.
Content-based Graph Reconstruction for Cold-start Item Recommendation
Authors: Jinri Kim, Eungi Kim, Kwangeun Yeo, Yujin Jeon, Chanwoo Kim, Sewon Lee, Joonseok Lee
Venue: SIGIR
Year: 2024

Graph convolutions have been successfully applied to recommendation systems, utilizing high-order collaborative signals present in the user-item interaction graph. This idea, however, has not been applicable to the cold-start items, since cold nodes are isolated in the graph and thus do not take advantage of information exchange from neighboring nodes. Recently, there have been a few attempts to utilize graph convolutions on item-item or user-user attribute graphs to capture high-order collaborative signals for cold-start cases, but these approaches are still limited in that the item-item or user-user graph falls short in capturing the dynamics of user-item interactions, as their edges are constructed based on arbitrary and heuristic attribute similarity. In this paper, we introduce Content-based Graph Reconstruction for Cold-start item recommendation (CGRC), employing a masked graph autoencoder structure and multimodal contents to directly incorporate interaction-based high-order connectivity, applicable even in cold-start scenarios. To address the cold-start items directly on the interaction graph, our approach trains the model to reconstruct plausible user-item interactions from masked edges of randomly chosen cold items, simulating fresh items without connection to users. This strategy enables the model to infer potential edges for unseen cold-start nodes. Extensive experiments on real-world datasets demonstrate the superiority of our model.
Self-Referential Review: Exploring the Impact of Self-Reference Effect in Review
Authors: Kyusik Kim, Hyungwoo Song, Bongwon Suh
Venue: SIGIR
Year: 2024

The self-reference effect is a psychological phenomenon where information relating to oneself is processed more deeply and remembered more effectively than other information. We propose "self-referential reviews," crafted by merging personal information with existing reviews using the novel "Self-Referential ReviewMaker" prototype, which leverages Large Language Models (LLMs). The essence of the "self-referential review" lies in harnessing the self-reference effect, making the readers feel as if they are the protagonist of the review. To validate the efficacy of self-referential reviews, we conducted a user study focusing on online reviews with thirty-four participants. The contributions of our paper are centered around self-referential reviews, highlighting (1) the creation of these reviews using our new prototype, Self-Referential ReviewMaker, (2) their effectiveness in enhancing review helpfulness through the self-reference effect, and (3) the identification of additional factors influencing the self-reference effect with further discussion on enhancing user-focused review systems.
On the Negative Perception of Cross-domain Recommendations and Explanations
Authors: Denis Kotkov, Alan Medlar, Yang Liu, Dorota Glowacka
Venue: SIGIR
Year: 2024

Recommender systems typically operate within a single domain, for example, recommending books based on users' reading habits. If such data is unavailable, it may be possible to make cross-domain recommendations and recommend books based on user preferences from another domain, such as movies. However, despite considerable research on cross-domain recommendations, no studies have investigated their impact on users' behavioural intentions or system perceptions compared to single-domain recommendations. Similarly, while single-domain explanations have been shown to improve users' perceptions of recommendations, there are no comparable studies for the cross-domain case. In this article, we present a between-subject study (N=237) of users' behavioural intentions and perceptions of book recommendations. The study was designed to disentangle the effects of whether recommendations were single- or cross-domain from whether explanations were present or not. Our results show that cross-domain recommendations have lower trust and interest than single-domain recommendations, regardless of their quality. While these negative effects can be ameliorated by cross-domain explanations, they are still perceived as inferior to single-domain recommendations without explanations. Last, we show that explanations decrease interest in the single-domain case, but increase perceived transparency and scrutability in both single- and cross-domain recommendations. Our findings offer valuable insights into the impact of recommendation provenance on user experience and could inform the future development of cross-domain recommender systems.
5th Workshop on Patent Text Mining and Semantic Technologies (PatentSemTech2024)
Authors: Ralf Krestel, Hidir Aras, Linda Andersson, Florina Piroi, Allan Hanbury, Dean Alderucci
Venue: SIGIR
Year: 2024

Information retrieval systems for the patent domain have a long history. They can support patent experts in a variety of daily tasks: from analyzing the patent landscape to support experts in the patenting process and large-scale information extraction. Advances in machine learning and natural language processing allow to further automate tasks, such as paragraph retrieval, question answering (QA) or even patent text generation. Uncovering the potential of semantic technologies for the intellectual property (IP) industry is just getting started. Investigating the use of artificial intelligence methods for the patent domain is therefore not only of academic interest, but also highly relevant for practitioners. Compared to other domains, high quality, semi-structured, annotated data is available in large volumes (a requirement for supervised machine learning models), making training large models easier. On the other hand, domain-specific challenges arise, such as very technical language or legal requirements for patent documents. With the 5th edition of this workshop we will provide a platform for researchers and industry to learn about novel and emerging technologies for semantic patent retrieval and big analytics employing sophisticated methods ranging from patent text mining, domain-specific information retrieval to large language models targeting next generation applications and use cases for the IP and related domains.
Improving Embedding-Based Retrieval in Friend Recommendation with ANN Query Expansion
Authors: Pau Perng-Hwa Kung, Zihao Fan, Tong Zhao, Yozen Liu, Zhixin Lai, Jiahui Shi, Yan Wu, Jun Yu, Neil Shah, Ganesh Venkataraman
Venue: SIGIR
Year: 2024

Embedding-based retrieval in graph-based recommendation has shown great improvements over traditional graph walk retrieval methods, and has been adopted in large-scale industry applications such as friend recommendations [16]. However, it is not without its challenges: retraining graph embeddings frequently due to changing data is slow and costly, and producing high recall of approximate nearest neighbor search (ANN) on such embeddings is challenging due to the power law distribution of the indexed users. In this work, we address theses issues by introducing a simple query expansion method in ANN, called FriendSeedSelection, where for each node query, we construct a set of 1-hop embeddings and run ANN search. We highlight our approach does not require any model-level tuning, and is inferred from the data at test-time. This design choice effectively enables our recommendation system to adapt to the changing graph distribution without frequent heavy model retraining. We also discuss how we design our system to efficiently construct such queries online to support 10k+ QPS. For friend recommendation, our method shows improvements of recall, and 11% relative friend reciprocated communication metric gains, now serving over 800 million monthly active users at Snapchat.
Disentangled Contrastive Hypergraph Learning for Next POI Recommendation
Authors: Yantong Lai, Yijun Su, Lingwei Wei, Tianqi He, Haitao Wang, Gaode Chen, Daren Zha, Qiang Liu, Xingxing Wang
Venue: SIGIR
Year: 2024

Next point-of-interest (POI) recommendation has been a prominent and trending task to provide next suitable POI suggestions for users. Most existing sequential-based and graph neural network-based methods have explored various approaches to modeling user visiting behaviors and have achieved considerable performances. However, two key issues have received less attention: i) Most previous studies have ignored the fact that user preferences are diverse and constantly changing in terms of various aspects, leading to entangled and suboptimal user representations. ii) Many existing methods have inadequately modeled the crucial cooperative associations between different aspects, hindering the ability to capture complementary recommendation effects during the learning process. To tackle these challenges, we propose a novel framework Disentangled Contrastive Hypergraph Learning (DCHL) for next POI recommendation. Specifically, we design a multi-view disentangled hypergraph learning component to disentangle intrinsic aspects among collaborative, transitional and geographical views with adjusted hypergraph convolutional networks. Additionally, we propose an adaptive fusion method to integrate multi-view information automatically. Finally, cross-view contrastive learning is employed to capture cooperative associations among views and reinforce the quality of user and POI representations based on self-discrimination. Extensive experiments on three real-world datasets validate the superiority of our proposal over various state-of-the-arts. To facilitate future research, our code is available at https://github.com/icmpnorequest/SIGIR2024_DCHL.
Third Workshop on Personalization and Recommendations in Search (PaRiS)
Authors: Sudarshan Lamkhede, Hamed Zamani, Moumita Bhattacharya, Hongning Wang
Venue: SIGIR
Year: 2024

With proliferation of personal computing devices and large number of logged-in experiences, search has evolved to a stage with many different product scenarios where personalization plays a crucial role for relevance quality and user satisfaction. The purpose of this workshop is have a forum where latest research and advancements specifically on Personalization and Recommendations in Search (PaRiS) can be discussed in conjunction with SIGIR 2024. This will be the third instance of this workshop. We held two very successful instances of this workshop at the WebConf 2023 [5] and WSDM 2022. This year we will especially focus on applications of LLMs and Generative AI to enable personalization and recommendations [1] in the context of search, for example, conversational assistants, while continuing to use this workshop for discussing other advances and applications in the context of personalized search and recommendations in the context of search.
Grand: A Fast and Accurate Graph Retrieval Framework via Knowledge Distillation
Authors: Lin Lan, Pinghui Wang, Rui Shi, Tingqing Liu, Juxiang Zeng, Feiyang Sun, Yang Ren, Jing Tao, Xiaohong Guan
Venue: SIGIR
Year: 2024

Graph retrieval aims to find the most similar graphs in a graph database given a query graph, which is a fundamental problem with many real-world applications in chemical engineering, code analysis, etc. To date, existing neural graph retrieval methods generally fall into two categories: Embedding Based Paradigm (Ebp) and Matching Based Paradigm (Mbp). The Ebp models learn an individual vectorial representation for each graph and the retrieval process can be accelerated by pre-computing these representations. The Mbp models learn a neural matching function to compare graphs on a pair-by-pair basis, in which the fine-grained pairwise comparison leads to higher retrieval accuracy but severely degrades retrieval efficiency. In this paper, to combine the advantage of Ebp in retrieval efficiency with that of Mbp in retrieval accuracy, we propose a novel Graph RetrievAl framework via KNowledge Distillation, namely GRAND. The key point is to leverage the idea of knowledge distillation to transfer the fine-grained graph comparison knowledge from an Mbp model to an Ebp model, such that the Ebp model can generate better graph representations and thus yield higher retrieval accuracy. At the same time, we can still pre-compute and index the improved graph representations to retain the retrieval speed of Ebp. Towards this end, we propose to perform knowledge distillation from three perspectives: score, node, and subgraph levels. In addition, we propose to perform mutual two-way knowledge transfer between Mbp and Ebp, such that Mbp and Ebp complement and benefit each other. Extensive experiments on three real-world datasets show that GRAND improves the performance of Ebp by a large margin and the improvement is consistent for different combinations of Ebp and Mbp models. For example, GRAND achieves performance gains of mostly more than 10% and up to 16.88% in terms of Recall@K on different datasets.
PLAID SHIRTTT for Large-Scale Streaming Dense Retrieval
Authors: Dawn J. Lawrie, Efsun Selin Kayi, Eugene Yang, James Mayfield, Douglas W. Oard
Venue: SIGIR
Year: 2024

PLAID, an efficient implementation of the ColBERT late interaction bi-encoder using pretrained language models for ranking, consistently achieves state-of-the-art performance in monolingual, cross-language, and multilingual retrieval. PLAID differs from ColBERT by assigning terms to clusters and representing those terms as cluster centroids plus compressed residual vectors. While PLAID is effective in batch experiments, its performance degrades in streaming settings where documents arrive over time because representations of new tokens may be poorly modeled by the earlier tokens used to select cluster centroids. PLAID Streaming Hierarchical Indexing that Runs on Terabytes of Temporal Text (PLAID HIRTTT) addresses this concern using multi-phase incremental indexing based on hierarchical sharding. Experiments on ClueWeb09 and the multilingual NeuCLIR collection demonstrate the effectiveness of this approach both for the largest collection indexed to date by the ColBERT architecture and in the multilingual setting, respectively.