OpenRaiser/Intern-Atlas huggingface dataset:
Short Video Ordering via Position Decoding and Successor Prediction
Authors: Shiping Ge, Qiang Chen, Zhiwei Jiang, Yafeng Yin, Ziyao Chen, Qing Gu
Venue: SIGIR
Year: 2024

Short video collection is an easy way for users to consume coherent content on various online short video platforms, such as TikTok, YouTube, Douyin, and WeChat Channel. These collections cover a wide range of content, including online courses, TV series, movies, and cartoons. However, short video creators occasionally publish videos in a disorganized manner due to various reasons, such as revisions, secondary creations, deletions, and reissues, which often result in a poor browsing experience for users. Therefore, accurately reordering videos within a collection based on their content coherence is a vital task that can enhance user experience and presents an intriguing research problem in the field of video narrative reasoning. In this work, we curate a dedicated multimodal dataset for this Short Video Ordering (SVO) task and present the performance of some benchmark methods on the dataset. In addition, we further propose an advanced SVO framework with the aid of position decoding and successor prediction. The proposed framework combines both pairwise and listwise ordering paradigms, which can get rid of the issues from both quadratic growth and cascading conflict in the pairwise paradigm, and improve the performance of existing listwise methods. Extensive experiments demonstrate that our method achieves the best performance on our open SVO dataset, and each component of the framework contributes to the final performance. Both the SVO dataset and code will be released at https://github.com/ShipingGe/SVO.
A Predictive Framework for Query Reformulation
Authors: Reyhaneh Goli
Venue: SIGIR
Year: 2024

Web search services are widely employed for various purposes. After identifying information needs, users attempt to articulate them in web queries that express their intentions. Then, they submit these queries to the chosen search engine with the hope of obtaining relevant results to meet their needs. In some cases, users may not immediately find precisely what they are seeking, prompting them to rewrite the query to obtain a greater number of relevant results or results that are perhaps more related to their intent. While significant work has been done on developing features such as query auto-completion, query suggestion, and query recommendation, the majority of these efforts were based on query co-occurrence or query similarity by clustering them or constructing query flow graphs to capture query connections. These approaches operate under the assumption that frequently observed follow-up queries are more likely to be submitted by users [1, 2, 4]. In this research, we investigate user query reformulation behavior. To achieve this, we will utilize the Trip Click dataset, a large-scale collection of user click data within the context of a health web search engine [3]. The log data from 2018 to 2020 will be considered, comprising 1,803,493 records representing the clicks that occurred across 527,749 sessions. Specifically, the focus will be on the impact of user interactions with the search result page when forming subsequent queries.
Graph Reasoning Enhanced Language Models for Text-to-SQL
Authors: Zheng Gong, Ying Sun
Venue: SIGIR
Year: 2024

Text-to-SQL parsing has attracted substantial attention recently due to its potential to remove barriers for non-expert end users interacting with databases. A key challenge in Text-to-SQL parsing is developing effective encoding mechanisms to capture the complex relationships between question words, database schemas, and their associated connections within the heterogeneous graph structure. Existing approaches typically introduce some useful multi-hop structures manually and then incorporate them into graph neural networks (GNNs) by stacking multiple layers, which (1) ignore the difficult-to-identify but meaningful semantics embedded in the multi-hop reasoning path, and (2) are limited by the expressive capability of GNN to capture long-range dependencies among the heterogeneous graph. To address these shortcomings, we introduce GRL-SQL, a graph reasoning enhanced language model, which innovatively applies structure encoding to capture the dependencies between node pairs, encompassing one-hop, multi-hop and distance information, subsequently enriched through self-attention for enhanced representational power over GNNs. Furthermore, GRL-SQL incorporates an interaction module that enables joint reasoning and fusion over the question-schema representations for enhancing global context modeling. Comprehensive experiments demonstrate the effectiveness and robustness of our proposed GRL-SQL.
Optimizing Learning-to-Rank Models for Ex-Post Fair Relevance
Authors: Sruthi Gorantla, Eshaan Bhansali, Amit Deshpande, Anand Louis
Venue: SIGIR
Year: 2024

Learning-to-rank (LTR) models rank items based on specific features, aiming to maximize ranking utility by prioritizing highly relevant items. However, optimizing only for ranking utility can lead to representational harm and may fail to address implicit bias in relevance scores. Prior studies introduced algorithms to train stochastic ranking models, such as the Plackett-Luce ranking model, that maximize expected ranking utility while achieving fairness in expectation (ex-ante fairness). Still, every sampled ranking may not satisfy group fairness (ex-post fairness). Post-processing methods ensure ex-post fairness; however, the LTR model lacks awareness of this step, creating a mismatch between the objective function the LTR model optimizes and the one it is supposed to optimize. In this paper, we first propose a novel objective where the relevance (or the expected ranking utility) is computed over only those rankings that satisfy given representation constraints for groups of items. We call this the ex-post fair relevance. We then give a framework for training Group-Fair LTR models to maximize our proposed ranking objective. Leveraging an efficient sampler for ex-post group-fair rankings and efficient algorithms to train the Plackett-Luce LTR model, we demonstrate their use in training the Group-Fair Plackett-Luce model in our framework. Experiments on MovieLens and Kiva datasets reveal improved fairness and relevance with our group-fair Plackett-Luce model compared to post-processing. In scenarios with implicit bias, our algorithm generally outperforms existing LTR baselines in both fairness and relevance.
High Recall Retrieval Via Technology-Assisted Review
Authors: Lenora Gray, David D. Lewis, Jeremy Pickens, Eugene Yang
Venue: SIGIR
Year: 2024

High Recall Retrieval (HRR) tasks, including eDiscovery in the law, systematic literature reviews, and sunshine law requests focus on efficiently prioritizing relevant documents for human review.Technology-assisted review (TAR) refers to iterative human-in-the-loop workflows that combine human review with IR and AI techniques to minimize both time and manual effort while maximizing recall. This full-day tutorial provides a comprehensive introduction to TAR. The morning session presents an overview of the key technologies and workflow designs used, the basics of practical evaluation methods, and the social and ethical implications of TAR deployment. The afternoon session provides more technical depth on the implications of TAR workflows for supervised learning algorithm design, how generative AI is can be applied in TAR, more sophisticated statistical evaluation techniques, and a wide range of open research questions.
Configurable Fairness for New Item Recommendation Considering Entry Time of Items
Authors: Huizhong Guo, Dongxia Wang, Zhu Sun, Haonan Zhang, Jinfeng Li, Jie Zhang
Venue: SIGIR
Year: 2024

Recommender systems tend to excessively expose longer-standing items, resulting in significant unfairness to new items with little interaction records, despite they may possess potential to attract considerable amount of users. The existing fairness-based solutions do not specifically consider the exposure fairness of new items, for which a systematic definition also lacks, discouraging the promotion of new items or contents. In this work, we introduce a multi-degree new-item exposure fairness definition, which considers item entry-time, and also is configurable regarding different fairness requirements. We then propose a configurable new-item fairness-aware framework named CNIF, which employs two-stage training where fairness degrees are incorporated for guidance. Extensive experiments on multiple popular datasets and backbone models demonstrate that CNIF can effectively enhance fairness of the existing models regarding the exposure resources of new items (including the brand-new items with no interaction). Specifically, CNIF demonstrates a substantial advancement with a 65.59% improvement in fairness metric and a noteworthy 9.97% improvement in recommendation accuracy compared to backbone models on the KuaiRec dataset. In comparison to various fairness-based solutions, it stands out by achieving the best trade-off between fairness and recommendation accuracy, surpassing the best baseline by 14.20%.
Improving the Accuracy of Locally Differentially Private Community Detection by Order-consistent Data Perturbation
Authors: Taolin Guo, Shunshun Peng, Zhejian Zhang, Mengmeng Yang, Kwok-Yan Lam
Venue: SIGIR
Year: 2024

Community detection refers to mechanisms that aim to identify groups of interacting nodes in a network according to the structural properties of the network. It has been used to analyze various graphs. In the context of social networks, it requires the collection of each user's social relations, posing the risk of user privacy intrusion caused by untrusted servers. Local differential privacy is a widely adopted approach for providing privacy protection while allowing acceptable utility of the protected data for analytics. There has been growing research interest in applying local differential privacy protection to community detection. However, such protection approaches typically suffer from poor accuracy due to the excessive noise in the protected data. This paper proposes LDP-Cd, a two-phase community detection framework under local differential privacy. LDP-Cd initializes the community groups using the Louvain community detection algorithm and iteratively refines the community in the second phase. Besides, we propose an order-consistent data perturbation method over the degree vector, thus ensuring the ordering consistency of the fitness between the user and community groups, thereby improving the accuracy of community detection. Experimental results on real datasets show that LDP-Cd has significant advantages over existing methods regarding community detection accuracy and a trade-off between user privacy and community detection utility.
Steering Large Language Models for Cross-lingual Information Retrieval
Authors: Ping Guo, Yubing Ren, Yue Hu, Yanan Cao, Yunpeng Li, Heyan Huang
Venue: SIGIR
Year: 2024

In today's digital age, accessing information across language barriers poses a significant challenge, with conventional search systems often struggling to interpret and retrieve multilingual content accurately. Addressing this issue, our study introduces a novel integration of applying Large Language Models (LLMs) as Cross-lingual Readers in information retrieval systems, specifically targeting the complexities of cross-lingual information retrieval (CLIR). We present an innovative approach: Activation Steered Multilingual Retrieval (ASMR) that employs "steering activations''-a method to adjust and direct the LLM's focus-enhancing its ability to understand user queries and generate accurate, language-coherent responses. ASMR adeptly combines a Multilingual Dense Passage Retrieval (mDPR) system with an LLM, overcoming the limitations of traditional search engines in handling diverse linguistic inputs. This approach is particularly effective in managing the nuances and intricacies inherent in various languages. Rigorous testing on established benchmarks such as XOR-TyDi QA, and MKQA demonstrates that ASMR not only meets but surpasses existing standards in CLIR, achieving state-of-the-art performance. The results of our research hold significant implications for understanding the inherent features of how LLMs understand and generate natural languages, offering an attempt towards more inclusive, effective, and linguistically diverse information access on a global scale.
SCM4SR: Structural Causal Model-based Data Augmentation for Robust Session-based Recommendation
Authors: Muskan Gupta, Priyanka Gupta, Jyoti Narwariya, Lovekesh Vig, Gautam Shroff
Venue: SIGIR
Year: 2024

With mounting privacy concerns, and movement towards a cookie-less internet, session-based recommendation (SR) models are gaining increasing popularity. The goal of SR models is to recommend top-K items to a user by utilizing information from past actions within a session. Many deep neural networks (DNN) based SR have been proposed in the literature, however, they experience performance declines in practice due to inherent biases (e.g., popularity bias) present in training data. To alleviate this, we propose an underlying neural-network (NN) based Structural Causal Model (SCM) which comprises an evolving user behavior (simulator) and recommendation model. The causal relations between the two sub-models and variables at consecutive timesteps are defined by a sequence of structural equations, whose parameters are learned using logged data. The learned SCM enables the simulation of a user's response on a counterfactual list of recommended items (slate). For this, we intervene on recommendation slates with counterfactual slates and simulate the user's response through learned SCM thereby generating counterfactual sessions to augment the training data. Through extensive empirical evaluation on simulated and real-world datasets, we show that the augmented data mitigates the impact of sparse training data and improves the performance of the SR models.
Misinformation Mitigation Praxis: Lessons Learned and Future Directions from Co·Insights
Authors: Scott A. Hale, Kiran Garimella, Shiri Dori-Hacohen
Venue: SIGIR
Year: 2024

Misinformation is a global challenge, but successful mitigations must come from the communities affected and not be imposed by external entities. Co·Insights is a multi-year NSF-funded initiative building capacity to respond to misinformation and harmful narratives in Asian American and Pacific Islander (AAPI) communities, based on a deep involvement with grassroots organizations and a co-construction of tools grounded on community needs. Co·Insights' unique cross-sectoral, cross-disciplinary collaboration is a convergence of information retrieval, computational social science, and ethnographic inquiry with a unique platform that enables community organizations, fact-checkers, and academics to work together to respond effectively to harmful content targeting communities. In this SIRIP talk designed for a technical audience, we will share lessons learned from the first 2.5 years of Co·Insights, and how we are bridging the academic--praxis divide by integrating state-of-the-art research developments into large-scale deployed systems. Topics covered will include: the challenges and joys of collaborating across disciplinary and sectoral boundaries; community-driven approaches that utilize information retrieval techniques such as claim matching in concert with emerging best practices in misinformation mitigation; open problems we have encountered in the space; and future directions we find promising. Co·Insights is led by Meedan, a global non-profit providing award-winning software solutions to mitigate misinformation; and in partnership with AAPI community organizations and several academic institutions.
Hypergraph Convolutional Network for User-Oriented Fairness in Recommender Systems
Authors: Zhongxuan Han, Chaochao Chen, Xiaolin Zheng, Li Zhang, Yuyuan Li
Venue: SIGIR
Year: 2024

The service system involves multiple stakeholders, making it crucial to ensure fairness. In this paper, we take the example of a typical service system, the recommender system, to investigate how to identify and tackle fairness issues within the service system. Recommender systems often exhibit bias towards a small user group, resulting in pronounced unfairness in recommendation performance, specifically the User-Oriented Fairness (UOF) issue. Existing research on UOF faces limitations in addressing two pivotal challenges: CH1: Current methods fall short in addressing the root cause of the UOF issue, stemming from an unfair training process between advantaged and disadvantaged users. CH2: Current methods struggle to unveil compelling correlations among users in sparse datasets. In this paper, we propose a novel Hypergraph Convolutional Network for User-Oriented Fairness, namely HyperUOF, to address the aforementioned challenges. HyperUOF serves as a versatile framework applicable to various backbone recommendation models for achieving UOF. To address CH1, HyperUOF employs an in-processing method that enhances the training process of disadvantaged users during model training. To addressCH2, HyperUOF incorporates a hypergraph-based approach, proven effective in sparse datasets, to explore high-order correlations among users. We conduct extensive experiments on three real-world datasets based on four backbone recommendation models to prove the effectiveness of our proposed HyperUOF.
LADy 💃: A Benchmark Toolkit for Latent Aspect Detection Enriched with Backtranslation Augmentation
Authors: Farinam Hemmatizadeh, Christine Wong, Alice Yu, Hossein Fani
Venue: SIGIR
Year: 2024

We present LADy ᖡ, a Python-based benchmark toolkit to facilitate extracting aspects of products or services in reviews toward which customers target their opinions and sentiments. While there has been a significant increase in aspect-based sentiment analysis, yet the proposed methods' practical implications in real-world settings remain moot for their closed and irreproducible codebases, inability to accommodate datasets from various domains, and poor evaluation methodologies. LADy is an open-source benchmark toolkit with a standard pipeline and experimental details to fill the gaps. It incorporates a host of canonical models along with benchmark datasets from varying domains, including unsolicited online reviews. Leveraging an object-oriented design, LADy readily extends to new models and training datasets. The first of its kind, LADy also features review augmentation via natural language backtranslation that can be integrated into the training phase of the models to boost efficiency and improve efficacy during inference. LADy's codebase, along with the installation instructions and case studies on five datasets for seven methods with backtranslation augmentation over ten languages, can be obtained under cc-by-nc-sa-4.0 license at https://github.com/fani-lab/LADy.
Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies
Authors: Chih-Wei Hsu, Martin Mladenov, Ofer Meshi, James Pine, Hubert Pham, Shane Li, Xujian Liang, Anton Polishko, Li Yang, Ben Scheetz, Craig Boutilier
Venue: SIGIR
Year: 2024

Evaluation of policies in recommender systems typically involves A/B live experiments on real users to assess a new policy's impact on relevant metrics. This "gold standard'' comes at a high cost, however, in terms of cycle time, user cost, and potential user retention. In developing policies for onboarding users, these costs can be especially problematic, since on-boarding occurs only once. In this work, we describe a simulation methodology used to augment (and reduce) the use of live experiments. We illustrate its deployment for the evaluation of preference elicitation algorithms used to onboard new users of the YouTube Music platform. By developing counterfactually robust user behavior models, and a simulation service that couples such models with production infrastructure, we can test new algorithms in a way that reliably predicts their performance on key metrics when deployed live.
Exploring Multi-Scenario Multi-Modal CTR Prediction with a Large Scale Dataset
Authors: Zhaoxin Huan, Ke Ding, Ang Li, Xiaolu Zhang, Xu Min, Yong He, Liang Zhang, Jun Zhou, Linjian Mo, Jinjie Gu, Zhongyi Liu, Wenliang Zhong, Guannan Zhang, Chenliang Li, Fajie Yuan
Venue: SIGIR
Year: 2024

Click-through rate (CTR) prediction plays a crucial role in recommendation systems, with significant impact on user experience and platform revenue generation. Despite the various public CTR datasets available due to increasing interest from both academia and industry, these datasets have limitations. They cover a limited range of scenarios and predominantly focus on ID-based features, neglecting the vital role of multi-modal features for effective multi-scenario CTR prediction. Moreover, their scale is modest compared to real-world industrial datasets, hindering robust and comprehensive evaluation of complex models. To address these challenges, we introduce a large-scale Multi-Scenario Multi-Modal CTR dataset named AntM2 C, built from real industrial data from Alipay. This dataset offers an impressive breadth and depth of information, covering CTR data from four diverse business scenarios, including advertisements, consumer coupons, mini-programs, and videos. Unlike existing datasets, AntM2 C provides not only ID-based features but also five textual features and one image feature for both users and items, supporting more delicate multi-modal CTR prediction. AntM2 C is also substantially larger than existing datasets, comprising 100 million CTR data. This scale allows for robust and comprehensive evaluation and comparison of CTR prediction models. We employ AntM2 C to construct several typical CTR tasks, including multi-scenario modeling, item and user cold-start modeling, and multi-modal modeling. Initial experiments and comparisons with baseline methods have shown that AntM2 C presents both new challenges and opportunities for CTR models, with the potential to significantly advance CTR research. The AntM2 C dataset is available at https://www.atecup.cn/OfficalDataSet.
A Large-scale Offer Alignment Model for Partitioning Filtering and Matching Product Offers
Authors: Wenyu Huang, André Melo, Jeff Z. Pan
Venue: SIGIR
Year: 2024

Offer alignment is a key step in a product knowledge graph construction pipeline. It aims to align retailer offers of the same product for better coverage of product details. With the rapid development of online shopping services, the offer alignment task is applied in ever larger datasets. This work aims to build an offer alignment system that can efficiently be used in large-scale offer data. The key components of this system include: 1) common offer encoders for encoding text offer data into representations; 2) trainable LSH partitioning module to divide similar offers into small blocks; 3) lightweight sophisticated late-interactions for efficient filtering and scoring of offer alignment candidate pairs. We evaluate the system on public WDC offer alignment dataset, as well as DBLP-Scholar and DBLP-ACM.