OpenRaiser/Intern-Atlas huggingface dataset:
Patterns of Gender-Specializing Query Reformulation
Authors: Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
Venue: SIGIR
Year: 2023

Users of search systems often reformulate their queries by adding query terms to reflect their evolving information need or to more precisely express their information need when the system fails to surface relevant content. Analyzing these query reformulations can inform us about both system and user behavior. In this work, we study a special category of query reformulations that involve specifying demographic group attributes, such as gender, as part of the reformulated query (e.g., ''olympic 2021 soccer results'' -> ''olympic 2021 women's soccer results"). There are many ways a query, the search results, and a demographic attribute such as gender may relate, leading us to hypothesize different causes for these reformulation patterns, such as under-representation on the original result page or based on the linguistic theory of markedness. This paper reports on an observational study of gender-specializing query reformulations---their contexts and effects---as a lens on the relationship between system results and gender, based on large-scale search log data from Bing. We find that these reformulations sometimes correct for and other times reinforce gender representation on the original result page, but typically yield better access to the ultimately-selected results. The prevalence of these reformulations---and which gender they skew towards---differ by topical context. However, we do not find evidence that either group under-representation or markedness alone adequately explains these reformulations. We hope that future research will use such reformulations as a probe for deeper investigation into gender (and other demographic) representation on the search result page.
DeviceGPT: A Generative Pre-Training Transformer on the Heterogenous Graph for Internet of Things
Authors: Yimo Ren, Jinfa Wang, Hong Li, Hongsong Zhu, Limin Sun
Venue: SIGIR
Year: 2023

Recently, Graph neural networks (GNNs) have been adopted to model a wide range of structured data from academic and industry fields. With the rapid development of Internet technology, there are more and more meaningful applications for Internet devices, including device identification, geolocation and others, whose performance needs improvement. To replicate the several claimed successes of GNNs, this paper proposes DeviceGPT based on a generative pre-training transformer on a heterogeneous graph via self-supervised learning to learn interactions-rich information of devices from its large-scale databases well. The experiments on the dataset constructed from the real world show DeviceGPT could achieve competitive results in multiple Internet applications.
GreenSeq: Automatic Design of Green Networks for Sequential Recommendation Systems
Authors: Yankun Ren, Xinxing Yang, Xingyu Lu, Longfei Li, Jun Zhou, Jinjie Gu, Guannan Zhang
Venue: SIGIR
Year: 2023

Transformer-based models have achieved tremendous success in sequential recommendation (SR), but they suffer from consuming excessive computational resources, particularly in the inference stage. Thus, developing lightweight yet effective SR models has become a frequent demand in industrial applications, which is also in line with the ideals of Green AI and Green IR. In this applied paper, we introduce GreenSeq deployed in Alipay to automatically design Green networks that can provide appropriate recommendations with lower computational consumption in SR. Specifically, GreenSeq uses a novel multi-layer search space that allows for flexible network design and a Greenness-aware loss term for balancing efficiency and effectiveness. Experiments on benchmark datasets and A/B testing show that GreenSeq performs well while using fewer resources. GreenSeq also reduces electricity and carbon emissions in Alipay.
Hear Me Out: A Study on the Use of the Voice Modality for Crowdsourced Relevance Assessments
Authors: Nirmal Roy, Agathe Balayn, David Maxwell, Claudia Hauff
Venue: SIGIR
Year: 2023

The creation of relevance assessments by human assessors (often nowadays crowdworkers) is a vital step when building IR test collections. Prior works have investigated assessor quality & behaviour, and tooling to support assessors in their task. We have few insights though into the impact of a document's presentation modality on assessor efficiency and effectiveness. Given the rise of voice-based interfaces, we investigate whether it is feasible for assessors to judge the relevance of text documents via a voice-based interface. We ran a user study (n = 49) on a crowdsourcing platform where participants judged the relevance of short and long documents- sampled from the TREC Deep Learning corpus-presented to them either in the text or voice modality. We found that: (i) participants are equally accurate in their judgements across both the text and voice modality; (ii) with increased document length it takes partic- ipants significantly longer (for documents of length > 120 words it takes almost twice as much time) to make relevance judgements in the voice condition; and (iii) the ability of assessors to ignore stimuli that are not relevant (i.e., inhibition) impacts the assessment quality in the voice modality-assessors with higher inhibition are significantly more accurate than those with lower inhibition. Our results indicate that we can reliably leverage the voice modality as a means to effectively collect relevance labels from crowdworkers.
Affective Relevance: Inferring Emotional Responses via fNIRS Neuroimaging
Authors: Tuukka Ruotsalo, Kalle Mäkelä, Michiel M. A. Spapé, Luis A. Leiva
Venue: SIGIR
Year: 2023

Information retrieval (IR) relies on a general notion of relevance, which is used as the principal foundation for ranking and evaluation methods. However, IR does not account for more a nuanced affective experience. Here, we consider the emotional response decoded directly from the human brain as an alternative dimension of relevance. We report an experiment covering seven different scenarios in which we measure and predict how users emotionally respond to visual image contents by using functional near-infrared spectroscopy (fNIRS) neuroimaging on two commonly used affective dimensions: valence (negativity and positivity) and arousal (boredness and excitedness). Our results show that affective states can be successfully decoded using fNIRS, and utilized to complement the present notion of relevance in IR studies. For example, we achieved 0.39 Balanced accuracy and 0.61 AUC in 4-class classification of affective states (vs. 0.25 Balanced accuracy and 0.5 AUC of a random classifier). Likewise, we achieved 0.684 Precision@20 when retrieving high-arousal images. Our work opens new avenues for incorporating emotional states in IR evaluation, affective feedback, and information filtering.
Synerise Monad: A Foundation Model for Behavioral Event Data
Authors: Barbara Rychalska, Szymon Lukasik, Jacek Dabrowski
Venue: SIGIR
Year: 2023

The complexity of industry-grade event-based datalakes grows dynamically each passing hour. Companies actively gather behavioral information on their customers, recording multiple types of events, such as clicks, likes, page views, card transactions, add-to-basket, or purchase events. In response to this, the Synerise Monad platform has been proposed. The primary focus of Monad is to produce Universal Behavioral Representations (UBRs) - large vectors encapsulating the behavioral patterns of each user. UBRs do not lose knowledge about individual events, in contrast to aggregated features or averaged embeddings. They are based on award-winning algorithms developed at Synerise - Cleora and EMDE - and allow to process real-life datasets composed of billions of events in record time. In this paper, we introduce a new aspect of Monad: private foundation models for behavioral data, trained on top of UBRs. The foundation models are trained in purely self-supervised manner and allow to exploit general knowledge about human behavior, which proves especially useful when multiple downstream models must be trained and time constraints are tight, or when labeled data is scarce. Experimental results show that the Monad foundation models can cut training time in half and require 3x less data to reach optimal results, often achieving state-of-the-art results.
Exploratory Visualization Tool for the Continuous Evaluation of Information Retrieval Systems
Authors: Gabriela González Sáez, Petra Galuscáková, Romain Deveaud, Lorraine Goeuriot, Philippe Mulhem
Venue: SIGIR
Year: 2023

This paper introduces a novel visualization tool that facilitates the exploratory analysis of continuous evaluation for information retrieval systems. We base our analysis on score standardization and meta-analysis techniques applied to Information Retrieval evaluation. We present three functionalities: evaluation overview, delta evaluation, and meta-analysis applied to three perspectives: evaluation rounds, queries, and systems. To illustrate the use of the tool, we provide an example using the TREC-COVID test collection.
Quantifying and Leveraging User Fatigue for Interventions in Recommender Systems
Authors: Hitesh Sagtani, Madan Gopal Jhawar, Akshat Gupta, Rishabh Mehrotra
Venue: SIGIR
Year: 2023

Predicting churn and designing intervention strategies are crucial for online platforms to maintain user engagement. We hypothesize that predicting churn, i.e. users leaving from the system without further return, is often a delayed act, and it might get too late for the system to intervene. We propose detecting early signs of users losing interest, allowing time for intervention, and introduce a new formulation ofuser fatigue as short-term dissatisfaction, providing early signals to predict long-term churn. We identify behavioral signals predicting fatigue and develop models for fatigue prediction. Furthermore, we leverage the predicted fatigue estimates to develop fatigue-aware ad-load balancing intervention strategy that reduces churn, improving short- and long-term user retention. Results from deployed recommendation system and multiple live A/B tests across over 80 million users generating over 200 million sessions highlight gains for user engagement and platform strategic metrics.
pybool_ir: A Toolkit for Domain-Specific Search Experiments
Authors: Harrisen Scells, Martin Potthast
Venue: SIGIR
Year: 2023

Undertaking research in domain-specific scenarios such as systematic review literature search, legal search, and patent search can often have a high barrier of entry due to complicated indexing procedures and complex Boolean query syntax. Indexing and searching document collections like PubMed in off-the-shelf tools such as Elasticsearch and Lucene often yields less accurate (and less effective) results than the PubMed search engine, i.e., retrieval results do not match what would be retrieved if one issued the same query to PubMed. Furthermore, off-the-shelf tools have their own nuanced query languages and do not allow directly using the often large and complicated Boolean queries seen in domain-specific search scenarios. The pybool_ir toolkit aims to address these problems and to lower the barrier to entry for developing new methods for domain-specific search. The toolkit is an open source package available at https://github.com/hscells/pybool_ir.
Smooth Operators for Effective Systematic Review Queries
Authors: Harrisen Scells, Ferdinand Schlatt, Martin Potthast
Venue: SIGIR
Year: 2023

Effective queries are crucial to minimising the time and cost of medical systematic reviews, as all retrieved documents must be judged for relevance. Boolean queries, developed by expert librarians, are the standard for systematic reviews. They guarantee reproducible and verifiable retrieval and more control than free-text queries. However, the result sets of Boolean queries are unranked and difficult to control due to the strict Boolean operators. We address these problems in a single unified retrieval model by formulating a class of smooth operators that are compatible with and extend existing Boolean operators. Our smooth operators overcome several shortcomings of previous extensions of the Boolean retrieval model. In particular, our operators are independent of the underlying ranking function, so that exact-match and large language model rankers can be combined in the same query. We found that replacing Boolean operators with equivalent or similar smooth operators often improves the effectiveness of queries. Their properties make tuning a query to precision or recall intuitive and allow greater control over how documents are retrieved. This additional control leads to more effective queries and reduces the cost of systematic reviews.
Knowledge Discovery from Unstructured Data in Financial Services (KDF) Workshop
Authors: Sameena Shah, Xiaodan Zhu, Wenhu Chen, Manling Li, Armineh Nourbakhsh, Xiaomo Liu, Zhiqiang Ma, Charese Smiley, Yulong Pei, Akshat Gupta
Venue: SIGIR
Year: 2023

Knowledge discovery from unstructured data, including business documents, web content, and news articles, has been a key AI challenge for the financial services industry. Comprehending these corpora and discovering knowledge from them, which could be textual, tabular, or graphic, are the cornerstone of supporting business decisions in the financial services domain, where information retrieval and content analysis techniques are of fundamental importance. We propose a workshop on knowledge discovery from unstructured data in financial services at SIGIR 2023 to highlight the current and emerging opportunities, invite original research, and prompt success sharing between researchers.
AMICA: Alleviating Misinformation for Chinese Americans
Authors: Xiaoxiao Shang, Ye Chen, Yi Fang, Yuhong Liu, Subramaniam Vincent
Venue: SIGIR
Year: 2023

The increasing popularity of social media promotes the proliferation of misinformation, especially in the communities of Chinese-speaking diasporas, which has caused significant negative societal impacts. In addition, most of the existing efforts on misinformation mitigation have focused on English and other western languages, which makes numerous overseas Chinese a very vulnerable population to online disinformation campaigns. In this paper, we present AMICA, an information retrieval system for alleviating misinformation for Chinese Americans. AMICA dynamically collects data from popular social media platforms for Chinese Americans, including WeChat, Twitter, YouTube, and Chinese forums. The data are stored and indexed in Elasticsearch to provide advanced search functionalities. Given a user query, the ranking of social media posts considers both topical relevance and the likelihood of being misinformation.
Relation-Aware Multi-Positive Contrastive Knowledge Graph Completion with Embedding Dimension Scaling
Authors: Bin Shang, Yinliang Zhao, Di Wang, Jun Liu
Venue: SIGIR
Year: 2023

Recently, a large amount of work has emerged for knowledge graph completion (KGC), which aims to reason over known facts and to infer the missing links. Meanwhile, contrastive learning has been applied to the KGC tasks, which can improve the representation quality of entities and relations. However, existing KGC approaches tend to improve their performance with high-dimensional embeddings and complex models, which make them suffer from large storage space and high training costs. Furthermore, contrastive loss with single positive sample learns little structural and semantic information in knowledge graphs due to the complex relation types. To address these challenges, we propose a novel knowledge graph completion model named ConKGC with the embedding dimension scaling and a relation-aware multi-positive contrastive loss. In order to achieve both space consumption reduction and model performance improvement, a new scoring function is proposed to map the raw low-dimensional embeddings of entities and relations to high-dimensional embedding space, and predict low-dimensional tail entities with latent semantic information of high-dimensional embeddings. In addition, ConKGC designs a multiple weak positive samples based contrastive loss under different relation types to maintain two important training targets, Alignment and Uniformity. This loss function and few parameters of the model ensure that ConKGC performs best and has fast convergence speed. Extensive experiments on three standard datasets confirm the effectiveness of our innovations, and the performance of ConKGC is significantly improved compared to the state-of-the-art methods.
Embedding Based Retrieval in Friend Recommendation
Authors: Jiahui Shi, Vivek Chaurasiya, Yozen Liu, Shubham Vij, Yan Wu, Satya Kanduri, Neil Shah, Peicheng Yu, Nik Srivastava, Lei Shi, Ganesh Venkataraman, Jun Yu
Venue: SIGIR
Year: 2023

Friend recommendation systems in online social and professional networks such as Snapchat helps users find friends and build connections, leading to better user engagement and retention. Traditional friend recommendation systems take advantage of the principle of locality and use graph traversal to retrieve friend candidates, e.g. Friends-of-Friends (FoF). While this approach has been adopted and shown efficacy in companies with large online networks such as Linkedin and Facebook, it suffers several challenges: (i) discrete graph traversal offers limited reach in cold-start settings, (ii) it is expensive and infeasible in realtime settings beyond 1 or 2 hop requests owing to latency constraints, and (iii) it cannot well-capture the complexity of graph topology or connection strengths, forcing one to resort to other mechanisms to rank and find top-K candidates. In this paper, we proposed a new Embedding Based Retrieval (EBR) system for retrieving friend candidates, which complements the traditional FoF retrieval by retrieving candidates beyond 2-hop, and providing a natural way to rank FoF candidates. Through online A/B test, we observe statistically significant improvements in the number of friendships made with EBR as an additional retrieval source in both low- and high-density network markets. Our contributions in this work include deploying a novel retrieval system to a large-scale friend recommendation system at Snapchat, generating embeddings for billions of users using Graph Neural Networks, and building EBR infrastructure in production to support Snapchat scale.
Exploration of Unranked Items in Safe Online Learning to Re-Rank
Authors: Hiroaki Shiino, Kaito Ariu, Kenshi Abe, Riku Togashi
Venue: SIGIR
Year: 2023

Bandit algorithms for online learning to rank (OLTR) problems often aim to maximize long-term revenue by utilizing user feedback. From a practical point of view, however, such algorithms have a high risk of hurting user experience due to their aggressive exploration. Thus, there has been a rising demand for safe exploration in recent years. One approach to safe exploration is to gradually enhance the quality of an original ranking that is already guaranteed acceptable quality. In this paper, we propose a safe OLTR algorithm that efficiently exchanges one of the items in the current ranking with an item outside the ranking (i.e., an unranked item) to perform exploration. We select an unranked item optimistically to explore based on Kullback-Leibler upper confidence bounds (KL-UCB) and safely re-rank the items including the selected one. Through experiments, we demonstrate that the proposed algorithm improves long-term regret from baselines without any safety violation.
Topic-enhanced Graph Neural Networks for Extraction-based Explainable Recommendation
Authors: Jie Shuai, Le Wu, Kun Zhang, Peijie Sun, Richang Hong, Meng Wang
Venue: SIGIR
Year: 2023

Review information has been demonstrated beneficial for the explainable recommendation. It can be treated as training corpora for generation-based methods or knowledge bases for extraction-based models. However, for generation-based methods, the sparsity of user-generated reviews and the high complexity of generative language models lead to a lack of personalization and adaptability. For extraction-based methods, focusing only on relevant attributes makes them invalid in situations where explicit attribute words are absent, limiting the potential of extraction-based models. To this end, in this paper, we focus on the explicit and implicit analysis of review information simultaneously and propose novel a Topic-enhanced Graph Neural Networks (TGNN) to fully explore review information for better explainable recommendations. To be specific, we first use a pre-trained topic model to analyze reviews at the topic level, and design a sentence-enhanced topic graph to model user preference explicitly, where topics are intermediate nodes between users and items. Corresponding sentences serve as edge features. Thus, the requirement of explicit attribute words can be mitigated. Meanwhile, we leverage a review-enhanced rating graph to model user preference implicitly, where reviews are also considered as edge features for fine-grained user-item interaction modeling. Next, user and item representations from two graphs are used for final rating prediction and explanation extraction. Extensive experiments on three real-world datasets demonstrate the superiority of our proposed TGNN with both recommendation accuracy and explanation quality.
Evaluating Task-oriented Dialogue Systems with Users
Authors: Clemencia Siro
Venue: SIGIR
Year: 2023

Evaluation is one of the major concerns when developing information retrieval systems. Especially in the field of conversational AI, this topic has been heavily studied in the setting of both non-task and task-oriented conversational agents (dialogue systems).[1] Recently, several automatic metrics e.g., BLEU and ROUGE, proposed for the evaluation of dialogue systems, have shown poor correlation with human judgment and are thus ineffective for the evaluation of dialogue systems. As a consequence, a significant amount of research relies on human evaluation to estimate the effectiveness of dialogue systems[1, 4}. An emerging approach for evaluating task-oriented dialogue systems (TDS) is to estimate a user's overall satisfaction with the system from explicit and implicit user interaction signals [2, 3]. Though useful and effective, overall user satisfaction does not necessarily give insights into what aspects or dimensions a TDS is performing well on. Understanding why a user is satisfied or dissatisfied helps the TDS recover from an error and optimize towards an individual aspect to avoid total dissatisfaction during an interaction session. Understanding a user's satisfaction with TDS is crucial, mainly for two reasons. First, it allows system designers to understand different user perceptions regarding satisfaction, which in turn leads to better user personalization. Secondly, it can be used to avoid total dialogue failure by the system by deploying adaptive conversational approaches, such as failure recovery or switching topics. And, thus, fine-grained evaluation of TDS gives the system an opportunity to learn an individual user's interaction preferences leading to a fulfilled user goal. Therefore in this research, we take the first initiative toward understanding user satisfaction with TDS. We mainly focus on the fine-grained evaluation of conversational systems in a task-oriented setting.