OpenRaiser/Intern-Atlas huggingface dataset:
MEME: Multi-Encoder Multi-Expert Framework with Data Augmentation for Video Retrieval
Authors: Seong-Min Kang, Yoon-Sik Cho
Venue: SIGIR
Year: 2023

Text-to-video(T2V) retrieval aims to find relevant videos from text queries. The recently introduced Contrastive Language Image Pretraining (CLIP), a pretrained language-vision model trained on large-scale image and caption pairs, has been extensively studied in the literature for this task. Existing studies on T2V task have aimed to transfer the CLIP knowledge and focus on enhancing retrieval performance through fine-grained representation learning. While fine-grained contrast has achieved some remarkable results, less attention has been paid to coarse-grained contrasts. To this end, we propose a method called Graph Patch Spreading (GPS) to aggregate patches across frames at the coarse-grained level. We apply GPS to our proposed framework called Multi-Encoder Multi-Expert (MEME) framework. Our proposed scheme is general enough to be applied to any existing CLIP-based video-text retrieval models. We demonstrate the effectiveness of our method on existing models over the benchmark datasets MSR-VTT, MSVD, and LSMDC datasets. Our code can be found at https://github.com/kang7734/MEME__.
BioSift: A Dataset for Filtering Biomedical Abstracts for Drug Repurposing and Clinical Meta-Analysis
Authors: David Kartchner, Irfan Al-Hussaini, Haydn Turner, Jennifer Deng, Shubham Lohiya, Prasanth Bathala, Cassie S. Mitchell
Venue: SIGIR
Year: 2023

This work presents a new, original document classification dataset, BioSift, to expedite the initial selection and labeling of studies for drug repurposing. The dataset consists of 10,000 human-annotated abstracts from scientific articles in PubMed. Each abstract is labeled with up to eight attributes necessary to perform meta-analysis utilizing the popular patient-intervention-comparator-outcome (PICO) method: has human subjects, is clinical trial/cohort, has population size, has target disease, has study drug, has comparator group, has a quantitative outcome, and an "aggregate" label. Each abstract was annotated by 3 different annotators (i.e., biomedical students) and randomly sampled abstracts were reviewed by senior annotators to ensure quality. Data statistics such as reviewer agreement, label co-occurrence, and confidence are shown. Robust benchmark results illustrate neither PubMed advanced filters nor state-of-the-art document classification schemes (e.g., active learning, weak supervision, full supervision) can efficiently replace human annotation. In short, BioSift is a pivotal but challenging document classification task to expedite drug repurposing. The full annotated dataset is publicly available and enables research development of algorithms for document classification that enhance drug repurposing.
Large-Scale Data Processing for Information Retrieval Applications
Authors: Pooya Khandel
Venue: SIGIR
Year: 2023

Developing Information Retrieval (IR) applications such as search engines and recommendation systems require training of models that are growing in complexity and size with immense collections of data that contain multiple dimensions (documents/items text, user profiles, and interactions). Much of the research in IR concentrates on improving the performance of ranking models; however, given the high training time and high computational resources required to improve the performance by designing new models, it is crucial to address efficiency aspects of the design and deployment of IR applications at large-scale. In my thesis, I aim to improve the training efficiency of IR applications and speed up the development phase of new models, by applying dataset distillation approaches to reduce the dataset size while preserving the ranking quality and employing efficient High-Performance Computing (HPC) solutions to increase the processing speed.
LOVF: Layered Organic View Fusion for Click-through Rate Prediction in Online Advertising
Authors: Lingwei Kong, Lu Wang, Xiwei Zhao, Junsheng Jin, Zhangang Lin, Jinghe Hu, Jingping Shao
Venue: SIGIR
Year: 2023

Organic recommendation and advertising recommendation usually coexist on e-commerce platforms. In this paper, we study the problem of utilizing data from organic recommendation to reinforce click-through rate prediction in advertising scenarios from a multi-view learning perspective. We propose a novel method, termed LOVF (Layered Organic View Fusion). LOVF implements a multi-view fusion mechanism - for each advertising instance, LOVF derives deep representations layer-by-layer from the organic recommendation view and these deep representations are then fused into the corresponding vanilla representations of the advertising view. Extensive experiments across a variety of backbones demonstrate LOVF's generality, effectiveness and efficiency on a new real-world production dataset. The dataset encompasses data from both the organic recommendation and advertising scenarios. Notably, LOVF has been successfully deployed in the advertising recommender system of JD.com, which is one of the world's largest e-commerce platforms; online A/B testing shows that LOVF achieves impressive improvement on advertising clicks and revenue. Our code and dataset are available at https://github.com/adsturing/lovf for facilitating further research.
4th Workshop on Patent Text Mining and Semantic Technologies (PatentSemTech2023)
Authors: Ralf Krestel, Hidir Aras, Linda Andersson, Florina Piroi, Allan Hanbury, Dean Alderucci
Venue: SIGIR
Year: 2023

Information retrieval systems for the patent domain have a long history. They can support patent experts in a variety of daily tasks: from analyzing the patent landscape to support experts in the patenting process and large-scale information extraction. Advances in machine learning and natural language processing allow to further automate tasks, such as paragraph retrieval or even patent text generation. Uncovering the potential of semantic technologies for the intellectual property (IP) industry is just getting started. Investigating the use of artificial intelligence methods for the patent domain is therefore not only of academic interest, but also highly relevant for practitioners. Compared to other domains, high quality, semi-structured, annotated data is available in large volumes (a requirement for supervised machine learning models), making training large models easier. On the other hand, domain-specific challenges arise, such as very technical language or legal requirements for patent documents. The focus of the 4th edition of this workshop will be on two-way communication between industry and academia from all areas of information retrieval in particular with the Asian community. We want to bring together novel research results and the latest systems and methods employed by practitioners in the field.
Bridging Quantitative and Qualitative Digital Experience Testing
Authors: Ranjitha Kumar
Venue: SIGIR
Year: 2023

Digital user experiences are a mainstay of modern communication and commerce; multi-billion dollar industries have arisen around optimizing digital design. Usage analytics and A/B testing solutions allow growth hackers to quantitatively compute conversion over key user journeys, while user experience (UX) testing platforms enable UX researchers to qualitatively analyze usability and brand perception. Although these workflows are in pursuit of the same objective - producing better UX - the gulf between quantitative and qualitative testing is wide: they involve different stakeholders, and rely on disparate methodologies, budget, data streams, and software tools. This gap belies the opportunity to create a single platform that optimizes digital experiences holistically: using quantitative methods to uncover what and how much and qualitative analysis to understand why. Such a platform could monitor conversion funnels, identify anomalous behaviors, intercept live users exhibiting those behaviors, and solicit explicit feedback in situ. This feedback could take many forms: survey responses, screen recordings of participants performing tasks, think-aloud audio, and more. By combining data from multiple users and correlating across feedback types, the platform could surface not just insights that a particular conversion funnel had been affected, but hypotheses about what had caused the change in user behavior. The platform could then rank these insights by how often the observed behavior occurred in the wild, using large-scale analytics to contextualize the results from small-scale UX tests. To this end, a decade of research has focused on interaction mining: a set of techniques for capturing interaction and design data from digital artifacts, and aggregating these multimodal data streams into structured representations bridging quantitative and qualitative experience testing[1-4]. During user sessions, interaction mining systems record user interactions (e.g., clicks, scrolls, text input), screen captures, and render-time data structures (e.g., website DOMs, native app view hierarchies). Once captured, these data streams are aligned and combined into user traces, sequences of user interactions semanticized by the design data of their UI targets [5]. The structure of these traces affords new workflows for composing quantitative and qualitative methods, building toward a unified platform for optimizing digital experiences.
Multi-lingual Semantic Search for Domain-specific Applications: Adobe Photoshop and Illustrator Help Search
Authors: Jayant Kumar, Ashok Gupta, Zhaoyu Lu, Andrei Stefan, Tracy Holloway King
Venue: SIGIR
Year: 2023

Search has become an integral part of Adobe products and users rely on it to learn about tool usage, shortcuts, quick links, and ways to add creative effects and to find assets such as backgrounds, templates, and fonts. Within applications such as Photoshop and Illustrator, users express domain-specific search intents via short text queries. In this work, we leverage sentence-BERT models fine-tuned on Adobe's HelpX data to perform multi-lingual semantic search on help and tutorial documents. We used behavioral data (queries, clicks, and impressions) and additional annotated data to train several BERT-based models for scoring query-document pairs for semantic similarity. We benchmarked the keyword-based production system against semantic search. Subsequent AB tests demonstrate that this approach improves engagement for longer queries while reducing null results significantly.
CEC: Towards Learning Global Optimized Recommendation through Causality Enhanced Conversion Model
Authors: Ran Le, Guoqing Jiang, Xiufeng Shu, Ruidong Han, Qianzhong Li, Yacheng Li, Xiang Li, Wei Lin
Venue: SIGIR
Year: 2023

Most e-commerce platforms consist of multiple entries (e.g., recommendation, search, shopping cart and etc.) for users to purchase their liked items. Among the research on the recommendation entry, most of them focus on improving the conversion volumes merely in the recommendation entry. However, such way could not ensure an increase in the global conversion volumes of the e-commerce platform. To achieve this goal by optimizing the recommendation entry only, in this paper, we focus on modeling the causality between the recommendation-entry-impression and the conversion by proposing the two-stage Causality Enhanced Conversion (CEC) model. In the first stage, we define the recommendation-entry-impression as treatment, then we estimate the conversion rate conditioned on the inclusion or exclusion of treatment respectively and calculate the corresponding individual treatment effect (ITE). In the second stage, we propose a propensity-normalization (PN) based method to transform the learned ITE to a weight term for instance weighting in the conversion loss. Extensive offline and online experiments on a large-scale food e-commerce scenario demonstrate that the CEC model could focus more on those conversed instances that can improve the global conversion volumes of the platform.
Computational Versus Perceived Popularity Miscalibration in Recommender Systems
Authors: Oleg Lesota, Gustavo Escobedo, Yashar Deldjoo, Bruce Ferwerda, Simone Kopeinik, Elisabeth Lex, Navid Rekabsaz, Markus Schedl
Venue: SIGIR
Year: 2023

Popularity bias in recommendation lists refers to over-representation of popular content and is a challenge for many recommendation algorithms. Previous research has suggested several offline metrics to quantify popularity bias, which commonly relate the popularity of items in users' recommendation lists to the popularity of items in their interaction history. Discrepancies between these two factors are referred to as popularity miscalibration. While popularity metrics provide a straightforward and well-defined means to measure popularity bias, it is unknown whether they actually reflect users' perception of popularity bias. To address this research gap, we conduct a crowd-sourced user study on Prolific, involving 56 participants, to (1) investigate whether the level of perceived popularity miscalibration differs between common recommendation algorithms, (2) assess the correlation between perceived popularity miscalibration and its corresponding quantification according to a common offline metric. We conduct our study in a well-defined and important domain, namely music recommendation using the standardized LFM-2b dataset, and quantify popularity miscalibration of five recommendation algorithms by utilizing Jensen-Shannon distance (JSD). Challenging the findings of previous studies, we observe that users generally do perceive significant differences in terms of popularity bias between algorithms if this bias is framed as popularity miscalibration. In addition, JSD correlates moderately with users' perception of popularity, but not with their perception of unpopularity.
Context-Aware Modeling via Simulated Exposure Page for CTR Prediction
Authors: Xiang Li, Shuwei Chen, Jian Dong, Jin Zhang, Yongkang Wang, Xingxing Wang, Dong Wang
Venue: SIGIR
Year: 2023

Click-through rate (CTR) prediction plays a crucial role in industrial recommendation and advertising systems, which generate and expose multiple items for each user request. Although the user's click action on an item will be affected by the other exposed items (called contextual items), current CTR prediction methods do not exploit this context because CTR prediction is performed before the contextual items are generated. This paper introduces a solution Contextual Items Simulation and Modeling (CISM) to tackle this limitation. Specifically, we propose a near-line Context Simulation Center to simulate exposure page without affecting online service latency, and an online Context Modeling Transformer to learn user-wise context from the simulated results w.r.t. the candidate item. In addition, knowledge distillation is introduced to further improve CTR prediction. Extensive experiments on both public and industrial datasets demonstrate the effectiveness of CISM. Currently, CISM has been deployed in the online display advertising system of Meituan Waimai, serving the main traffic.
Dataset Preparation for Arbitrary Object Detection: An Automatic Approach based on Web Information in English
Authors: Shucheng Li, Boyu Chang, Bo Yang, Hao Wu, Sheng Zhong, Fengyuan Xu
Venue: SIGIR
Year: 2023

Automatic dataset preparation can help users avoid labor-intensive and costly manual data annotations. The difficulty in preparing a high-quality dataset for object detection involves three key aspects: relevance, naturality, and balance, which are not addressed by existing works. In this paper, we leverage information from the web, and propose a fully-automatic dataset preparation mechanism without any human annotation, which can automatically prepare a high-quality training dataset for the detection task with English text terms describing target objects. It contains three key designs, i.e., keyword expansion, data de-noising, and data balancing. Our experiments demonstrate that the object detectors trained with auto-prepared data are comparable to those trained with benchmark datasets and outperform other baselines. We also demonstrate the effectiveness of our approach in several more challenging real-world object categories that are not included in the benchmark datasets.
Multimodal Counterfactual Learning Network for Multimedia-based Recommendation
Authors: Shuaiyang Li, Dan Guo, Kang Liu, Richang Hong, Feng Xue
Venue: SIGIR
Year: 2023

Multimedia-based recommendation (MMRec) utilizes multimodal content (images, textual descriptions, etc.) as auxiliary information on historical interactions to determine user preferences. Most MMRec approaches predict user interests by exploiting a large amount of multimodal contents of user-interacted items, ignoring the potential effect of multimodal content of user-uninteracted items. As a matter of fact, there is a small portion of user preference-irrelevant features in the multimodal content of user-interacted items, which may be a kind of spurious correlation with user preferences, thereby degrading the recommendation performance. In this work, we argue that the multimodal content of user-uninteracted items can be further exploited to identify and eliminate the user preference-irrelevant portion inside user-interacted multimodal content, for example by counterfactual inference of causal theory. Going beyond multimodal user preference modeling only using interacted items, we propose a novel model called Multimodal Counterfactual Learning Network (MCLN), in which user-uninteracted items' multimodal content is additionally exploited to further purify the representation of user preference-relevant multimodal content that better matches the user's interests, yielding state-of-the-art performance. Extensive experiments are conducted to validate the effectiveness and rationality of MCLN. We release the complete codes of MCLN at https://github.com/hfutmars/MCLN.
LibVQ: A Toolkit for Optimizing Vector Quantization and Efficient Neural Retrieval
Authors: Chaofan Li, Zheng Liu, Shitao Xiao, Yingxia Shao, Defu Lian, Zhao Cao
Venue: SIGIR
Year: 2023

Vector quantization is one of the critical techniques which enables dense retrieval for realtime applications. The recent study shows that vanilla vector quantization methods, like those implemented by FAISS [8], are lossy and prone to limited retrieval performances when large acceleration ratios are needed [14, 16, 18]. Besides, there have also been multiple algorithms which make the retriever and VQ better collaborated to alleviate such a loss. On top of these progresses, we develop LibVQ, which optimizes vector quantization for efficient dense retrieval. Our toolkit is highlighted for three advantages. 1. Effectiveness. The retrieval quality can be substantially improved over the vanilla implementations of VQ. 2. Simplicity. The optimization can be conducted in a lowcode fashion, and the optimization results can be easily loaded to ANN indexes to support downstream applications. 3. Universality. The optimization is agnostic to the embedding's learning process, and may accommodate different input conditions and ANN back-ends with little modification of the workflow. LibVQ may also support rich applications beyond dense retrieval, e.g., embedding compression, topic modeling, and de-duplication. In this demo, we provide comprehensive hand-on examples and evaluations for LibVQ. The toolkit is publicly released at: https://github.com/staoxiao/LibVQ/tree/demo.
Profiling and Visualizing Dynamic Pruning Algorithms
Authors: Zhixuan Li, Joel Mackenzie
Venue: SIGIR
Year: 2023

Efficiently retrieving the top-k documents for a given query is a fundamental operation in many search applications. Dynamic pruning algorithms accelerate top-k retrieval over inverted indexes by skipping documents that are not able to enter the current set of results. However, the performance of these algorithms depends on a number of variables such as the ranking function, the order of documents within the index, and the number of documents to be retrieved. In this paper, we propose a diagnostic framework, Dyno, for profiling and visualizing the performance of dynamic pruning algorithms. Our framework captures processing traces during retrieval, allowing the operations of the index traversal algorithm to be visualized. These visualizations support both query-level and system-to-system comparisons, enabling performance characteristics to be readily understood for different systems. Dyno benefits both academics and practitioners by furthering our understanding of the behavior of dynamic pruning algorithms, allowing better design choices to be made during experimentation and deployment.
Intent-aware Ranking Ensemble for Personalized Recommendation
Authors: Jiayu Li, Peijie Sun, Zhefan Wang, Weizhi Ma, Yangkun Li, Min Zhang, Zhoutian Feng, Daiyue Xue
Venue: SIGIR
Year: 2023

Ranking ensemble is a critical component in real recommender systems. When a user visits a platform, the system will prepare several item lists, each of which is generally from a single behavior objective recommendation model. As multiple behavior intents, e.g., both clicking and buying some specific item category, are commonly concurrent in a user visit, it is necessary to integrate multiple single-objective ranking lists into one. However, previous work on rank aggregation mainly focused on fusing homogeneous item lists with the same objective while ignoring ensemble of heterogeneous lists ranked with different objectives with various user intents. In this paper, we treat a user's possible behaviors and the potential interacting item categories as the user's intent. And we aim to study how to fuse candidate item lists generated from different objectives aware of user intents. To address such a task, we propose an Intent-aware ranking Ensemble Learning (IntEL) model to fuse multiple single-objective item lists with various user intents, in which item-level personalized weights are learned. Furthermore, we theoretically prove the effectiveness of IntEL with point-wise, pair-wise, and list-wise loss functions via error-ambiguity decomposition. Experiments on two large-scale real-world datasets also show significant improvements of IntEL on multiple behavior objectives simultaneously compared to previous ranking ensemble models.
Simple Approach for Aspect Sentiment Triplet Extraction Using Span-Based Segment Tagging and Dual Extractors
Authors: Dongxu Li, Zhihao Yang, Yuquan Lan, Yunqi Zhang, Hui Zhao, Gang Zhao
Venue: SIGIR
Year: 2023

Aspect sentiment triplet extraction (ASTE) is a task which extracts aspect terms, opinion terms, and sentiment polarities as triplets from review sentences. Existing approaches have developed bidirectional structures for term interaction. Sentiment polarities are subsequently extracted from aspect-opinion pairs. These solutions suffer from: 1) high dependency on custom bidirectional structures, 2) inadequate representation of the information through existing tagging schemes, and 3) insufficient usage of all available sentiment data. To address the above issues, we propose a simple span-based solution named SimSTAR with Segment Tagging And dual extRactors. SimSTAR does not introduce any additional bidirectional mechanism. The segment tagging scheme is capable to indicate all possible cases of spans and reveals more information through negative labels. Dual extractors are employed to make the sentiment extraction independent of the term extraction. We evaluate our model on four ASTE datasets. The experimental results show that our simple method achieves state-of-the-art performance.