OpenRaiser/Intern-Atlas huggingface dataset:
ElasticDL: A Kubernetes-native Deep Learning Framework with Fault-tolerance and Elastic Scheduling
Authors: Jun Zhou, Ke Zhang, Feng Zhu, Qitao Shi, Wenjing Fang, Lin Wang, Yi Wang
Venue: WSDM
Year: 2023

The power of artificial intelligence (AI) models originates with sophisticated model architecture as well as the sheer size of the model. These large-scale AI models impose new and challenging system requirements regarding scalability, reliability, and flexibility. One of the most promising solutions in the industry is to train these large-scale models on distributed deep-learning frameworks. With the power of all distributed computations, it is desired to achieve a training process with excellent scalability, elastic scheduling (flexibility), and fault tolerance (reliability). In this paper, we demonstrate the scalability, flexibility, and reliability of our open-source Elastic Deep Learning (ElasticDL) framework. Our ElasticDL utilizes an open-source system, i.e., Kubernetes, for automating deployment, scaling, and management of containerized application features to provide fault tolerance and support elastic scheduling for DL tasks.
Unbiased Learning to Rank: On Recent Advances and Practical Applications
Authors: Shashank Gupta, Philipp Hager, Jin Huang, Ali Vardasbi, Harrie Oosterhuis
Venue: WSDM
Year: 2024

Since its inception, the field of unbiased learning to rank (ULTR) has remained very active and has seen several impactful advancements in recent years. This tutorial provides both an introduction to the core concepts of the field and an overview of recent advancements in its foundations, along with several applications of its methods. The tutorial is divided into four parts: Firstly, we give an overview of the different forms of bias that can be addressed with ULTR methods. Secondly, we present a comprehensive discussion of the latest estimation techniques in the ULTR field. Thirdly, we survey published results of ULTR in real-world applications. Fourthly, we discuss the connection between ULTR and fairness in ranking. We end by briefly reflecting on the future of ULTR research and its applications. This tutorial is intended to benefit both researchers and industry practitioners interested in developing new ULTR solutions or utilizing them in real-world applications.
Domain Level Interpretability: Interpreting Black-box Model with Domain-specific Embedding
Authors: Ya-Lin Zhang, Caizhi Tang, Lu Yu, Jun Zhou, Longfei Li, Qing Cui, Fangfang Fan, Linbo Jiang, Xiaosong Zhao
Venue: WSDM
Year: 2024

The importance of incorporating interpretability into machine learning models has been increasingly emphasized. While previous literature has typically focused on feature level interpretability, such as analyzing which features are important and how they influence the final decision, real-world applications often require domain level interpretability, which relates to a group of features. Domain-level interpretability holds the potential for enhanced informativeness and comprehensibility. Unfortunately, there has been limited research in this direction. In this paper, we address this issue and introduce our proposed method DIDE, which obtains domain-level interpretability from domain-specific latent embeddings. To enhance the effectiveness of the framework, we draw inspiration from the gradient smooth philosophy and propose noisy injection in the embedding space, resulting in smoothed interpretability. We conduct extensive experiments to validate the effectiveness of DIDE, and demonstrate its applications in assisting daily business tasks in Alipay.
TTC-QuAli: A Text-Table-Chart Dataset for Multimodal Quantity Alignment
Authors: Haoyu Dong, Haochen Wang, Anda Zhou, Yue Hu
Venue: WSDM
Year: 2024

In modern documents, numerical information is often presented using multimodal formats such as text, tables, and charts. However, the heterogeneity of these sources poses a challenge for machines attempting to jointly read and understand the numerical semantics conveyed through text, tables, and charts. In this paper, we introduce a multimodal dataset called Text-Table-Chart Quantity Alignment (TTC-QuAli). This dataset is designed to facilitate a new task that involves linking related quantities across text, tables, and charts. TTC-QuAli is a comprehensive dataset that contains 4,498 quantities in text, aligned with 1,086 chart images and 1,503 tables from real-world statistical reports. It is the first dataset to provide high-quality annotations for linking quantities across multiple modalities, and it includes challenging composite (aggregated/calculated) quantity linking. To address the challenge of bridging representation gaps between different modalities and capturing their shared contextual semantic meaning, we introduce ConTTC, a novel transformer-based cross-modal contrastive learning architecture. This is the first architecture to jointly model text, tables, and charts, and contrastive learning is employed for multimodal quantity linking towards unified representation learning. Our experiments demonstrate that TTC-QuAli presents a significant challenge for existing baselines and serves as a valuable benchmark for future research. Experiment results show that ConTTC significantly outperforms all baseline methods.
Unlocking Human Curiosity
Authors: Elizabeth Reid
Venue: WSDM
Year: 2024

Human Curiosity has always been boundless. Yet for millenia, access to information limited our ability to explore that curiosity. The advent of the web transformed the information landscape, but too often information remained out of reach or required too much effort. Limitations in query understanding, the corpus of content, and information fragmentation continued to create substantial hurdles.
Real-time E-bike Route Planning with Battery Range Prediction
Authors: Zhao Li, Guoqi Ren, Yongchun Gu, Siwei Zhou, Xuanwu Liu, Jiaming Huang, Ming Li
Venue: WSDM
Year: 2024

Electric bicycles (EBs) have gained immense popularity as an environmentally friendly and convenient transportation mode. However, range anxiety remains a major concern for EB users. This paper presents a real-time route planning model focused on predicting the remaining range of EBs. First, we represent the user's interaction data and the real-time battery state as a dynamic graph. Then we propose a novel approach called the Real-Time Electric Bicycle Remaining Range (RtRR) prediction model, which leverages the graph structure and jointly optimizes temporal edge convolution, LSTM, and Transformer models to estimate the remaining EB battery range. Based on the prediction, we can update the optimal cycling routes for users in real-time, considering charging station locations. Extensive evaluations demonstrate that our proposed RtRR model outperforms 9 baseline methods on real-world datasets. The route planning based on RtRR prediction effectively alleviates range anxiety and enhances the user experience. It can be accessed at https://github.com/gu-yongchun/Real-time-E-bike-Route-Planning-with-Battery-Range-Prediction.
Practical Bandits: An Industry Perspective
Authors: Bram van den Akker, Olivier Jeunen, Ying Li, Ben London, Zahra Nazari, Devesh Parekh
Venue: WSDM
Year: 2024

The bandit paradigm provides a unified modeling framework for problems that require decision-making under uncertainty. Because many business metrics can be viewed as rewards (a.k.a. utilities) that result from actions, bandit algorithms have seen a large and growing interest from industrial applications, such as search, recommendation and advertising. Indeed, with the bandit lens comes the promise of direct optimisation for the metrics we care about. Nevertheless, the road to successfully applying bandits in production is not an easy one. Even when the action space and rewards are well-defined, practitioners still need to make decisions regarding multi-arm or contextual approaches, on- or off-policy setups, delayed or immediate feedback, myopic or long-term optimisation, etc. To make matters worse, industrial platforms typically give rise to large action spaces in which existing approaches tend to break down. The research literature on these topics is broad and vast, but this can overwhelm practitioners, whose primary aim is to solve practical problems, and therefore need to decide on a specific instantiation or approach for each project. This tutorial will take a step towards filling that gap between the theory and practice of bandits. Our goal is to present a unified overview of the field and its existing terminology, concepts and algorithms---with a focus on problems relevant to industry. We hope our industrial perspective will help future practitioners who wish to leverage the bandit paradigm for their application.
Some Useful Things to Know When Combining IR and NLP: The Easy, the Hard and the Ugly
Authors: Omar Alonso, Kenneth Church
Venue: WSDM
Year: 2024

Deep nets such as GPT are at the core of the current advances in many systems and applications. Things are moving very fast, and it appears that techniques are out of date within weeks. How can we take advantage of new discoveries and incorporate them into our existing work? Are these radical new developments, repetitions of older concepts, or both? In this tutorial, we aim to bring interested researchers and practitioners up to speed on the recent and ongoing techniques around ML and Deep learning in the context of IR and NLP. Additionally, our goal is to clarify terminology, emphasize fundamentals, and outline new research opportunities.
IDoFew: Intermediate Training Using Dual-Clustering in Language Models for Few Labels Text Classification
Authors: Abdullah Alsuhaibani, Hamad Zogan, Imran Razzak, Shoaib Jameel, Guandong Xu
Venue: WSDM
Year: 2024

Language models such as Bidirectional Encoder Representations from Transformers (BERT) have been very effective in various Natural Language Processing (NLP) and text mining tasks including text classification. However, some tasks still pose challenges for these models, including text classification with limited labels. This can result in a cold-start problem. Although some approaches have attempted to address this problem through single-stage clustering as an intermediate training step coupled with a pre-trained language model, which generates pseudo-labels to improve classification, these methods are often error-prone due to the limitations of the clustering algorithms. To overcome this, we have developed a novel two-stage intermediate clustering with subsequent fine-tuning that models the pseudo-labels reliably, resulting in reduced prediction errors. The key novelty in our model, IDoFew, is that the two-stage clustering coupled with two different clustering algorithms helps exploit the advantages of the complementary algorithms that reduce the errors in generating reliable pseudo-labels for fine-tuning. Our approach has shown significant improvements compared to strong comparative models.
Introduction to Responsible AI
Authors: Ricardo Baeza-Yates
Venue: WSDM
Year: 2024

In the first part of this tutorial we define responsible AI and we discuss the problems embedded in terms like ethical or trustworthy AI. In the second part, to set the stage, we cover irresponsible AI: discrimination (e.g., the impact of human biases); pseudo-science (e.g., biometric based behavioral predictions); human limitations (e.g., human incompetence, cognitive biases); technical limitations (data as a proxy of reality, wrong evaluation); social impact (e.g., unfair digital markets or mental health and disinformation issues created by large language models); environmental impact (e.g., indiscriminate use of computing resources). These examples do have a personal bias but set the context for the third part where we cover the current challenges: ethical principles, governance and regulation. We finish by discussing our responsible AI initiatives, many recommendations, and some philosophical issues.
MAD: Multi-Scale Anomaly Detection in Link Streams
Authors: Esteban Bautista, Laurent Brisson, Cécile Bothorel, Grégory Smits
Venue: WSDM
Year: 2024

Given an arbitrary group of computers, how to identify abnormal changes in their communication pattern? How to assess if the absence of some communications is normal or due to a failure? How to distinguish local from global events when communication data are extremely sparse and volatile? Existing approaches for anomaly detection in interaction streams, focusing on edge, nodes or graphs, lack flexibility to monitor arbitrary communication topologies. Moreover, they rely on structural features that are not adapted to highly sparse settings. In this work, we introduce MAD, a novel Multi-scale Anomaly Detection algorithm that (i) allows to query for the normality/abnormality state of an arbitrary group of observed/non-observed communications at a given time; and (ii) handles the highly sparse and uncertain nature of interaction data through a scoring method that is based on a novel probabilistic and multi-scale analysis of sub-graphs. In particular, MAD is (a) flexible: it can assess if any time-stamped subgraph is anomalous, making edge, node and graph anomalies particular instances; (b) interpretable: its multi-scale analysis allows to characterize the scope and nature of the anomalies; (c) efficient: given historical data of length N and M observed/non-observed communications to analyze, MAD produces an anomaly score in O (NM); and (d) effective: it significantly outperforms state-of-the-art alternatives tailored for edge, node or graph anomalies.
WSDM 2024 Workshop on Large Language Models for Individuals, Groups, and Society
Authors: Michael Bendersky, Cheng Li, Qiaozhu Mei, Vanessa Murdock, Jie Tang, Hongning Wang, Hamed Zamani, Mingyang Zhang
Venue: WSDM
Year: 2024

This workshop discusses the cutting-edge developments in research and applications of personalizing large language models (LLMs) and adapting them to the demands of diverse user populations and societal needs. The full-day workshop includes several keynotes and invited talks, a poster session and a panel discussion.
Leveraging User Simulation to Develop and Evaluate Conversational Information Access Agents
Authors: Nolwenn Bernard
Venue: WSDM
Year: 2024

We observe a change in the way users access information, that is, the rise of conversational information access (CIA) agents. However, the automatic evaluation of these agents remains an open challenge. Moreover, the training of CIA agents is cumbersome as it mostly relies on conversational corpora, expert knowledge, and reinforcement learning. User simulation has been identified as a promising solution to tackle automatic evaluation and has been previously used in reinforcement learning. In this research, we investigate how user simulation can be leveraged in the context of CIA. We organize the work in three parts. We begin with the identification of requirements for user simulators for training and evaluating CIA agents and compare existing types of simulator regarding these. Then, we plan to combine these different types of simulators into a new hybrid simulator. Finally, we aim to extend simulators to handle more complex information seeking scenarios.
IAI MovieBot 2.0: An Enhanced Research Platform with Trainable Neural Components and Transparent User Modeling
Authors: Nolwenn Bernard, Ivica Kostric, Krisztian Balog
Venue: WSDM
Year: 2024

While interest in conversational recommender systems has been on the rise, operational systems suitable for serving as research platforms for comprehensive studies are currently lacking. This paper introduces an enhanced version of the IAI MovieBot conversational movie recommender system, aiming to evolve it into a robust and adaptable platform for conducting user-facing experiments. The key highlights of this enhancement include the addition of trainable neural components for natural language understanding and dialogue policy, transparent and explainable modeling of user preferences, along with improvements in the user interface and research infrastructure.
Ranking with Long-Term Constraints
Authors: Kianté Brantley, Zhichong Fang, Sarah Dean, Thorsten Joachims
Venue: WSDM
Year: 2024

The feedback that users provide through their choices (e.g., clicks, purchases) is one of the most common types of data readily available for training search and recommendation algorithms. However, myopically training systems based on choice data may only improve short-term engagement, but not the long-term sustainability of the platform and the long-term benefits to its users, content providers, and other stakeholders. In this paper, we thus develop a new framework in which decision makers (e.g., platform operators, regulators, users) can express long-term goals for the behavior of the platform (e.g., fairness, revenue distribution, legal requirements). These goals take the form of exposure or impact targets that go well beyond individual sessions, and we provide new control-based algorithms to achieve these goals. In particular, the controllers are designed to achieve the stated long-term goals with minimum impact on short-term engagement. Beyond the principled theoretical derivation of the controllers, we evaluate the algorithms on both synthetic and real-world data. While all controllers perform well, we find that they provide interesting trade-offs in efficiency, robustness, and the ability to plan ahead.
Delphic Costs and Benefits in Web Search: A Utilitarian and Historical Analysis
Authors: Andrei Z. Broder
Venue: WSDM
Year: 2024

We present a new framework to conceptualize and operationalize the total user experience of search, by studying the entirety of a search journey from an utilitarian point of view. Web search engines are widely perceived as "free''. But search requires time and effort: in reality there are many intermingled non-monetary costs (e.g. time costs, cognitive costs, interactivity costs) and the benefits may be marred by various impairments, such as misunderstanding and misinformation. This characterization of costs and benefits appears to be inherent to the human search for information within the pursuit of some larger task: most of the costs and impairments can be identified in interactions with any web search engine, interactions with public libraries, and even in interactions with ancient oracles. To emphasize this innate connection, we call these costs and benefits "Delphic'', in contrast to explicitly financial costs and benefits. Our main thesis is that users' satisfaction with a search engine mostly depends on their experience of Delphic cost and benefits, in other words on their utility. The consumer utility is correlated with classic measures of search engine quality, such as ranking, precision, recall, etc., but is not completely determined by them. To argue our thesis, we catalog the Delphic costs and benefits and show how the development of web search over the last quarter century, from its classic Information Retrieval roots to the integration of Large Language Models and Generative AI, was driven to a great extent by the quest to decrease Delphic costs and increase Delphic benefits. We hope that the Delphic costs framework will engender new ideas and new research for evaluating and improving the web experience for everyone. This talk reflects joint work with Preston McAfee and Marc Najork.
Augmenting Keyword-based Search in Mobile Applications Using LLMs
Authors: Harikrishnan C, Giridhar Sreenivasa Murthy, Kumar Rangarajan
Venue: WSDM
Year: 2024

Search in mobile applications has traditionally been keyword driven and limited to simple queries, such as searching for product names, even when the apps support much richer, transactional experiences. On the other hand, search on the web has evolved into queries that are complex, objective-based and most often in natural language. The recent advances in Generative AI make it possible to bring the power of web-like, conversational searches into mobile applications. In this talk, we present the various problems, opportunities and challenges in harnessing LLMs to augment the traditional search experience in mobile applications.
To Copy, or not to Copy; That is a Critical Issue of the Output Softmax Layer in Neural Sequential Recommenders
Authors: Haw-Shiuan Chang, Nikhil Agarwal, Andrew McCallum
Venue: WSDM
Year: 2024

Recent studies suggest that the existing neural models have difficulty handling repeated items in sequential recommendation tasks. However, our understanding of this difficulty is still limited. In this study, we substantially advance this field by identifying a major source of the problem: the single hidden state embedding and static item embeddings in the output softmax layer. Specifically, the similarity structure of the global item embedding in the softmax layer sometimes forces the single hidden state embedding to be close to new items when copying is a better choice, while sometimes forcing the hidden state to be close to the items from the input inappropriately. To alleviate the problem, we adapt the recently-proposed softmax alternatives such as softmax-CPR to sequential recommendation tasks and demonstrate that the new softmax architectures unleash the capability of the neural encoder on learning when to copy and when to exclude the items from the input sequence. By only making some simple modifications on the output softmax layer for SASRec and GRU4Rec, softmax-CPR achieves consistent improvement in 12 datasets. With almost the same model size, our best method not only improves the average NDCG@10 of GRU4Rec in 5 datasets with duplicated items by 10% (4%-17% individually) but also improves 7 datasets without duplicated items by 24% (8%-39%)!
PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
Authors: Wei-Cheng Chang, Jyun-Yu Jiang, Jiong Zhang, Mutasem Al-Darabsah, Choon Hui Teo, Cho-Jui Hsieh, Hsiang-Fu Yu, S. V. N. Vishwanathan
Venue: WSDM
Year: 2024

Embedding-based Retrieval Models (ERMs) have emerged as a promising framework for large-scale text retrieval problems due to powerful large language models. Nevertheless, fine-tuning ERMs to reach state-of-the-art results can be expensive due to the extreme scale of data as well as the complexity of multi-stages pipelines (e.g., pre-training, fine-tuning, distillation). In this work, we propose the PEFA framework, namely ParamEter-Free Adapters, for fast tuning of ERMs without any backward pass in the optimization. At index building stage, PEFA equips the ERM with a non-parametric k-nearest neighbor (kNN) component. At inference stage, PEFA performs a convex combination of two scoring functions, one from the ERM and the other from the kNN. Based on the neighborhood definition, PEFA framework induces two realizations, namely PEFA-XL (i.e., extra large) using double ANN indices and PEFA-XS (i.e., extra small) using a single ANN index. Empirically, PEFA achieves significant improvement on two retrieval applications. For document retrieval, regarding Recall@100 metric, PEFA improves not only pre-trained ERMs on Trivia-QA by an average of 13.2%, but also fine-tuned ERMs on NQ-320K by an average of 5.5%, respectively. For product search, PEFA improves the Recall@100 of the fine-tuned ERMs by an average of 5.3% and 14.5%, for PEFA-XS and PEFA-XL, respectively. Our code is available at https://github.com/amzn/pecos/tree/mainline/examples/pefa-wsdm24.