OpenRaiser/Intern-Atlas huggingface dataset:
Automatic Multi-Task Learning Framework with Neural Architecture Search in Recommendations
Authors: Shen Jiang, Guanghui Zhu, Yue Wang 0084, Chunfeng Yuan, Yihua Huang 0001
Venue: KDD
Year: 2024

Multi-task learning (MTL), which aims to make full use of knowledge contained in multiple tasks to enhance overall performance and efficiency, has been broadly applied in recommendations. The main challenge for MTL models is negative transfer. Existing MTL models, mainly built on the Mixture-of-Experts (MoE) structure, seek enhancements in performance through feature selection and specific expert sharing mode design. However, one expert sharing mode may not be universally applicable due to the complex correlations and diverse demands among various tasks. Additionally, homogeneous expert architectures in such models further limit their performance. To address these issues, in this paper, we propose an innovative automatic MTL framework, AutoMTL, leveraging neural architecture search (NAS) to design optimal expert architectures and sharing modes. The Dual-level Expert Sharing mode and Architecture Navigator (DESAN) search space of AutoMTL can not only efficiently explore expert sharing modes and feature selection schemes but also focus on the architectures of expert subnetworks. Along with this, we introduce an efficient Progressively Discretizing Differentiable Architecture Search (PD-DARTS) algorithm for search space exploration. Extensive experiments demonstrate that AutoMTL can consistently outperform state-of-the-art, human-crafted MTL models. Moreover, the insights obtained from the discovered architectures provide valuable guidance for building new multi-task recommendation models.
RCTD: Reputation-Constrained Truth Discovery in Sybil Attack Crowdsourcing Environment
Authors: Xing Jin 0002, Zhihai Gong, Jiuchuan Jiang, Chao Wang, Jian Zhang 0023, Zhen Wang 0013
Venue: KDD
Year: 2024

Sybil attacks are a prevalent concern within the realm of crowdsourcing, underscoring the significance of quality control in this domain. Truth discovery has been extensively studied to deduce the most trustworthy information from conflicting data based on the principle that reliable workers yield reliable answers. However, existing truth discovery approaches overlook the metric of workers' reputations, e.g., workers' historical approval rates on crowdsourcing platforms, despite being inflated and noisy, they offer a rough indication of workers' ability. In this paper, we first refine the approval rate using Wilson Lower Bound to enhance its confidence, and then mitigate its noise and inflation through a method based on ranking similarity. Specifically, we propose a method called RCTD (Reputation-Constrained Truth Discovery), which introduces a similarity metric between the rankings of workers' weights and the refined approval rates. This metric serves as a penalizing factor in the objective function of the truth discovery, restricting workers' weights to avoid excessively deviating from their historical reputation during the weight estimation process. We solve the objective function by introducing the block coordinate descent coupled with heuristics approach method. Experimental results on real-world datasets demonstrate that our approach achieves more accurate inference of true results in the Sybil attack environment compared to the state-of-the-art methods.
RJUA-MedDQA: A Multimodal Benchmark for Medical Document Question Answering and Clinical Reasoning
Authors: Congyun Jin, Ming Zhang, Weixiao Ma, Yujiao Li, Yingbo Wang, Yabo Jia, Yuliang Du, Tao Sun 0018, Haowen Wang, Cong Fan, Jinjie Gu, Chenfei Chi, Xiangguo Lv, Fangzhou Li, Wei Xue, Yiran Huang
Venue: KDD
Year: 2024

Recent advancements in Large Language Models (LLMs) and Large Multi-modal Models (LMMs) have shown potential in various medical applications, such as Intelligent Medical Diagnosis. Although impressive results have been achieved, we find that existing benchmarks do not reflect the complexity of real medical reports and specialized in-depth reasoning capabilities. In this work, we establish a comprehensive benchmark in the field of medical specialization and introduced RJUA-MedDQA, which contains 2000 real-world Chinese medical report images poses several challenges: comprehensively interpreting imgage content across a wide variety of challenging layouts, possessing the numerical reasoning ability to identify abnormal indicators and demonstrating robust clinical reasoning ability to provide the statement of disease diagnosis, status and advice based on a collection of medical contexts. We carefully design the data generation pipeline and proposed the Efficient Structural Restoration Annotation (ESRA) Method, aimed at restoring textual and tabular content in medical report images. This method substantially enhances annotation efficiency, doubling the productivity of each annotator, and yields a 26.8% improvement in accuracy. We conduct extensive evaluations, including few-shot assessments of 5 LMMs which are capable of solving Chinese medical QA tasks. To further investigate the limitations and potential of current LMMs, we conduct comparative experiments on a set of strong LLMs by using image-text generated by ESRA method. We report the performance of baselines and offer several observations: (1) The overall performance of existing LMMs is still limited; however LMMs more robust to low-quality and diverse-structured images compared to LLMs. (3) Reasoning across context and image content present significant challenges. We hope this benchmark helps the community make progress on these challenging tasks in multi-modal medical document understanding and facilitate its application in healthcare. Our dataset will be publicly available for noncommercial use at https://github.com/Alipay-Med/medDQA_benchmark.git
Sketch-Based Replay Projection for Continual Learning
Authors: Jack Julian, Yun Sing Koh, Albert Bifet
Venue: KDD
Year: 2024

Continual learning closely emulates human learning, which allows a model to learn from a stream of tasks sequentially without forgetting previously learned knowledge. Replay-based continual learning methods mitigate forgetting and improve performance by reintroducing data belonging to old tasks, however a replay method's performance may deteriorate when the reintroduced data does not effectively represent all experienced data. To address this concern, we propose the Sketch-based Replay Projection (SRP) method to capture and retain the original data stream's distribution within stored memory. SRP augments existing replay frameworks and introduces a two-fold approach. First, we develop a sketch-based sample selection technique to approximate feature distributions within distinct tasks, thereby capturing a wide distribution of examples for subsequent replay. Second, we propose a data compression method which projects examples into a reduced-dimensional space while preserving inter-example relationships and emphasizing inter-class disparities, encouraging diverse representations of each class while maintaining memory requirements similar to existing replay methodologies. Our experimental results demonstrate that SRP enhances replay diversity and improves the performance of existing replay models.
Domain-Driven LLM Development: Insights into RAG and Fine-Tuning Practices
Authors: José Cassio dos Santos Junior, Rachel Hu, Richard Song, Yun-Fei Bai
Venue: KDD
Year: 2024

To improve Large Language Model (LLM) performance on domain specific applications, ML developers often leverage Retrieval Augmented Generation (RAG) and LLM Fine-Tuning. RAG extends the capabilities of LLMs to specific domains or an organization's internal knowledge base, without the need to retrain the model. On the other hand, Fine-Tuning approach updates LLM weights with domain-specific data to improve performance on specific tasks. The fine-tuned model is particularly effective to systematically learn new comprehensive knowledge in a specific domain that is not covered by the LLM pre-training. This tutorial walks through the RAG and Fine-Tuning techniques, discusses the insights of their advantages and limitations, and provides best practices of adopting the methodologies for the LLM tasks and use cases. The hands-on labs demonstrate the advanced techniques to optimize the RAG and fine-tuned LLM architecture that handles domain specific LLM tasks. The labs in the tutorial are designed by using a set of open-source python libraries to implement the RAG and fine-tuned LLM architecture.
Large Scale Hierarchical Industrial Demand Time-Series Forecasting incorporating Sparsity
Authors: Harshavardhan Kamarthi, Aditya B. Sasanur, Xinjie Tong, Xingyu Zhou, James Peters, Joe Czyzyk, B. Aditya Prakash
Venue: KDD
Year: 2024

Hierarchical time-series forecasting (HTSF) is an important problem for many real-world business applications where the goal is to simultaneously forecast multiple time-series that are related to each other via a hierarchical relation. Recent works, however, do not address two important challenges that are typically observed in many demand forecasting applications at large companies. First, many time-series at lower levels of the hierarchy have high sparsity i.e., they have a significant number of zeros. Most HTSF methods do not address this varying sparsity across the hierarchy. Further, they do not scale well to the large size of the real-world hierarchy typically unseen in benchmarks used in literature. We resolve both these challenges by proposing HAILS, a novel probabilistic hierarchical model that enables accurate and calibrated probabilistic forecasts across the hierarchy by adaptively modeling sparse and dense time-series with different distributional assumptions and reconciling them to adhere to hierarchical constraints. We show the scalability and effectiveness of our methods by evaluating them against real-world demand forecasting datasets. We deploy HAILS at a large chemical manufacturing company for a product demand forecasting application with over ten thousand products and observe a significant 8.5% improvement in forecast accuracy and 23% better improvement for sparse time-series. The enhanced accuracy and scalability make HAILS a valuable tool for improved business planning and customer experience.
Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)
Authors: Krishnaram Kenthapadi, Mehrnoosh Sameki, Ankur Taly
Venue: KDD
Year: 2024

With the ongoing rapid adoption of Artificial Intelligence (AI)-based systems in high-stakes domains, ensuring the trustworthiness, safety, and observability of these systems has become crucial. It is essential to evaluate and monitor AI systems not only for accuracy and quality-related metrics but also for robustness, bias, security, interpretability, and other responsible AI dimensions. We focus on large language models (LLMs) and other generative AI models, which present additional challenges such as hallucinations, harmful and manipulative content, and copyright infringement. In this survey article accompanying our <u>tutorial</u>, we highlight a wide range of harms associated with generative AI systems, and survey state of the art approaches (along with open challenges) to address these harms.
Gandalf: Learning Label-label Correlations in Extreme Multi-label Classification via Label Features
Authors: Siddhant Kharbanda, Devaansh Gupta, Erik Schultheis, Atmadeep Banerjee, Cho-Jui Hsieh, Rohit Babbar
Venue: KDD
Year: 2024

Publisher Copyright: © 2024 Copyright held by the owner/author(s).
Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender System
Authors: Sein Kim, Hongseok Kang, Seungyoon Choi, Donghyun Kim 0007, Min-Chul Yang, Chanyoung Park 0001
Venue: KDD
Year: 2024

Collaborative filtering recommender systems (CF-RecSys) have shown successive results in enhancing the user experience on social media and e-commerce platforms. However, as CF-RecSys struggles under cold scenarios with sparse user-item interactions, recent strategies have focused on leveraging modality information of user/items (e.g., text or images) based on pre-trained modality encoders and Large Language Models (LLMs). Despite their effectiveness under cold scenarios, we observe that they underperform simple traditional collaborative filtering models under warm scenarios due to the lack of collaborative knowledge. In this work, we propose an efficient All-round LLM-based Recommender system, called A-LLMRec, that excels not only in the cold scenario but also in the warm scenario. Our main idea is to enable an LLM to directly leverage the collaborative knowledge contained in a pre-trained state-of-the-art CF-RecSys so that the emergent ability of the LLM as well as the high-quality user/item embeddings that are already trained by the state-of-the-art CF-RecSys can be jointly exploited. This approach yields two advantages: (1) model-agnostic, allowing for integration with various existing CF-RecSys, and (2) efficiency, eliminating the extensive fine-tuning typically required for LLM-based recommenders. Our extensive experiments on various real-world datasets demonstrate the superiority of A-LLMRec in various scenarios, including cold/warm, few-shot, cold user, and cross-domain scenarios. Beyond the recommendation task, we also show the potential of A-LLMRec in generating natural language outputs based on the understanding of the collaborative knowledge by performing a favorite genre prediction task. Our code is available at https://github.com/ghdtjr/A-LLMRec.
A Survey on Hypergraph Neural Networks: An In-Depth and Step-By-Step Guide
Authors: Sunwoo Kim 0006, Soo Yong Lee, Yue Gao 0002, Alessia Antelmi, Mirko Polato, Kijung Shin
Venue: KDD
Year: 2024

Higher-order interactions (HOIs) are ubiquitous in real-world complex systems and applications. Investigation of deep learning for HOIs, thus, has become a valuable agenda for the data mining and machine learning communities. As networks of HOIs are expressed mathematically as hypergraphs, hypergraph neural networks (HNNs) have emerged as a powerful tool for representation learning on hypergraphs. Given the emerging trend, we present the first survey dedicated to HNNs, with an in-depth and step-by-step guide. Broadly, the present survey overviews HNN architectures, training strategies, and applications. First, we break existing HNNs down into four design components: (i) input features, (ii) input structures, (iii) message-passing schemes, and (iv) training strategies. Second, we examine how HNNs address and learn HOIs with each of their components. Third, we overview the recent applications of HNNs in recommendation, bioinformatics and medical science, time series analysis, and computer vision. Lastly, we conclude with a discussion on limitations and future directions.
Fast and Accurate Domain Adaptation for Irregular Tensor Decomposition
Authors: Junghun Kim, Ka Hyun Park, Jun-Gi Jang, U Kang
Venue: KDD
Year: 2024

Given an irregular tensor from a newly emerging domain, how can we quickly and accurately capture its patterns utilizing existing irregular tensors in multiple domains? The problem is of great importance for various tasks such as finding patterns of a new disease using pre-existing diseases data. This is challenging as new target tensors have limited information due to their recent emergence. Thus, carefully utilizing the existing source tensors for analyzing the target tensor is helpful. PARAFAC2 decomposition is a strong tool for finding the patterns of irregular tensors, and the patterns are used in many applications such as missing value prediction and anomaly detection. However, previous PARAFAC2-based works cannot adaptably handle newly emerging target tensors utilizing the source tensors.
False Positives in A/B Tests
Authors: Ron Kohavi, Nanyu Chen
Venue: KDD
Year: 2024

A/B tests, or online controlled experiments, are used heavily in the software industry to evaluate implementations of ideas, as the paradigm is the gold standard in science for establishing causality: the changes introduced in the treatment caused the changes to the metrics of interest with high probability. What distinguishes software experiments, or A/B tests, from experiments in many other domains is the scale (e.g., over 100 experiment treatments may launch on a given workday in large companies) and the effect sizes that matter to the business are small (e.g., a 3% improvement to conversion rate from a single experiment is a cause for celebration). The humbling reality is that most experiments fail to improve key metrics, and success rates of only about 10-20% are most common. With low success rates, the industry standard alpha threshold of 0.05 implies a high probability of false positives. We begin with motivation about why false positives are expensive in many software domains. We then offer several approaches to estimate the true success rate of experiments, given the observed "win" rate (statistically significant positive improvements), and show examples from Expedia and Optimizely. We offer a modified procedure for experimentation, based in sequential group testing, that selectively extends experiments to reduce false positives, increase power, at a small increase to runtime. We conclude with a discussion of the difference between ideas and experiments in practice, terms that are often incorrectly used interchangeably.
OntoType: Ontology-Guided and Pre-Trained Language Model Assisted Fine-Grained Entity Typing
Authors: Tanay Komarlu, Minhao Jiang, Xuan Wang 0008, Jiawei Han 0001
Venue: KDD
Year: 2024

Fine-grained entity typing (FET), which assigns entities in text with context-sensitive, fine-grained semantic types, is a basic but important task for knowledge extraction from unstructured text. FET has been studied extensively in natural language processing and typically relies on human-annotated corpora for training, which is costly and difficult to scale. Recent studies explore the utilization of pre-trained language models (PLMs) as a knowledge base to generate rich and context-aware weak supervision for FET. However, a PLM still requires direction and guidance to serve as a knowledge base as they often generate a mixture of rough and fine-grained types, or tokens unsuitable for typing. In this study, we vision that an ontology provides a semantics-rich, hierarchical structure, which will help select the best results generated by multiple PLM models and head words. Specifically, we propose a novel annotation-free, ontology-guided FET method, OntoType, which follows a type ontological structure, from coarse to fine, ensembles multiple PLM prompting results to generate a set of type candidates, and refines its type resolution, under the local context with a natural language inference model. Our experiments on the Ontonotes, FIGER, and NYT datasets using their associated ontological structures demonstrate that our method outperforms the state-of-the-art zero-shot fine-grained entity typing methods as well as a typical LLM method, ChatGPT. Our error analysis shows that refinement of the existing ontology structures will further improve fine-grained entity typing.
Offline Reinforcement Learning for Optimizing Production Bidding Policies
Authors: Dmytro Korenkevych, Frank Cheng, Artsiom Balakir, Alex Nikulkov, Lingnan Gao, Zhihao Cen, Zuobing Xu, Zheqing Zhu
Venue: KDD
Year: 2024

The online advertising market, with its thousands of auctions run per second, presents a daunting challenge for advertisers who wish to optimize their spend under a budget constraint. Thus, advertising platforms typically provide automated agents to their customers, which act on their behalf to bid for impression opportunities in real time at scale. Because these proxy agents are owned by the platform but use advertiser funds to operate, there is a strong practical need to balance reliability and explainability of the agent with optimizing power. We propose a generalizable approach to optimizing bidding policies in production environments by learning from real data using offline reinforcement learning. This approach can be used to optimize any differentiable base policy (practically, a heuristic policy based on principles which the advertiser can easily understand), and only requires data generated by the base policy itself. We use a hybrid agent architecture that combines arbitrary base policies with deep neural networks, where only the optimized base policy parameters are eventually deployed, and the neural network part is discarded after training. We demonstrate that such an architecture achieves statistically significant performance gains in both simulated and at-scale production bidding environments compared to the default production bidding policy. Our approach does not incur additional infrastructure, safety, or explainability costs, as it directly optimizes the parameters of existing production routines without necessarily replacing them with black box-style models like neural networks.
LeMon: Automating Portrait Generation for Zero-Shot Story Visualization with Multi-Character Interactions
Authors: Ziyi Kou, Shichao Pei, Xiangliang Zhang 0001
Venue: KDD
Year: 2024

Zero-Shot Story Visualization (ZSV) seeks to depict textual narratives through a sequence of images without relying on pre-existing text-image pairs for training. In this paper, we address the challenge of automated multi-character ZSV, aiming to create distinctive yet compatible character portraits for high-quality story visualization without the need of manual human interventions. Our study is motivated by the limitation of current ZSV approaches that necessitate inefficient manual collection of external images as initial character portraits and suffer from low-quality story visualization, especially with multi-character interactions, when the portraits are not well initiated. To overcome these issues, we develop LeMon, an LLM enhanced Multi-Character Zero-Shot Visualization framework that automates character portrait initialization and supports iterative portrait refinement by exploring the semantic content of the story. In particular, we design an LLM-based portrait generation strategy that matches the story characters with external movie characters, and leverage the matched resources as in-context learning (ICL) samples for LLMs to accurately initialize the character portraits. We then propose a graph-based Text2Image diffusion model that constructs a character interaction graph from the story to iteratively refine the character portraits by maximizing the distinctness of different characters while minimizing their incompatibility in the multi-character story visualization. Our evaluation results show that LeMon outperforms existing ZSV approaches in generating high-quality visualizations for stories across various types with multiple interacted characters. Our code is available at https://github.com/arxrean/LLM-LeMon.