OpenRaiser/Intern-Atlas huggingface dataset:
AuditLLM: A Tool for Auditing Large Language Models Using Multiprobe Approach
Authors: Maryam Amirizaniani, Elias Martin, Tanya Roosta, Aman Chadha, Chirag Shah
Venue: CIKM
Year: 2024

As Large Language Models (LLMs) are integrated into various sectors, ensuring their reliability and safety is crucial. This necessitates rigorous probing and auditing to maintain their effectiveness and trustworthiness in practical applications. Subjecting LLMs to varied iterations of a single query can unveil potential inconsistencies in their knowledge base or functional capacity. However, a tool for performing such audits with a easy to execute workflow, and low technical threshold is lacking. In this demo, we introduce "AuditLLM," a novel tool designed to audit the performance of various LLMs in a methodical way. AuditLLM's primary function is to audit a given LLM by deploying multiple probes derived from a single question, thus detecting any inconsistencies in the model's comprehension or performance. A robust, reliable, and consistent LLM is expected to generate semantically similar responses to variably phrased versions of the same question. Building on this premise, AuditLLM generates easily interpretable results that reflect the LLM's consistency based on a single input question provided by the user. A certain level of inconsistency has been shown to be an indicator of potential bias, hallucinations, and other issues. One could then use the output of AuditLLM to further investigate issues with the aforementioned LLM. To facilitate demonstration and practical uses, AuditLLM offers two key modes: (1) Live mode which allows instant auditing of LLMs by analyzing responses to real-time queries; and (2) Batch mode which facilitates comprehensive LLM auditing by processing multiple queries at once for in-depth analysis. This tool is beneficial for both researchers and general users, as it enhances our understanding of LLMs' capabilities in generating responses, using a standardized auditing platform.
Understanding the User: An Intent-Based Ranking Dataset
Authors: Abhijit Anand, Jurek Leonhardt, Venktesh V, Avishek Anand
Venue: CIKM
Year: 2024

As information retrieval systems continue to evolve, accurate evaluation and benchmarking of these systems become pivotal. Web search datasets, such as MS MARCO, primarily provide short keyword queries without accompanying intent or descriptions, posing a challenge in comprehending the underlying information need. This paper proposes an approach to augmenting such datasets to annotate informative query descriptions, with a focus on two prominent benchmark datasets: TREC-DL-21 and TREC-DL-22. Our methodology involves utilizing state-of-the-art LLMs to analyze and comprehend the implicit intent within individual queries from benchmark datasets. By extracting key semantic elements, we construct detailed and contextually rich descriptions for these queries. To validate the generated query descriptions, we employ crowdsourcing as a reliable means of obtaining diverse human perspectives on the accuracy and informativeness of the descriptions. This information can be used as an evaluation set for tasks such as ranking, query rewriting, or others.
Advances in Citation Text Generation: Leveraging Multi-Source Seq2Seq Models and Large Language Models
Authors: Avinash Anand, Ashwin R. Nair, Kritarth Prasad, Vrinda Narayan, Naman Lal, Debanjan Mahata, Yaman Singla, Rajiv Ratn Shah
Venue: CIKM
Year: 2024

Citation Text Generation (CTG) in scientific documents often relies on standard summarization techniques, which may not fully capture the nuanced relationship between the citing and cited papers. To address this, we present a Multi-Source Citation Text Generation (M-CTG) architecture, leveraging a Seq2Seq transformer framework enhanced with keyphrase embeddings, graph embeddings, and text representations. This approach aims to produce more contextually relevant and accurate citation texts by integrating multiple sources of information. Our methodology is tested using the newly created CTG-S2ORC dataset, consisting of English-language computer science research papers. In a comparative analysis, we explore the performance of traditional Language Models (LMs) and demonstrate how Large Language Models (LLMs), particularly when integrated with various prompting techniques and Knowledge Graphs, offer superior capabilities in analyzing and generating citation texts. In addition to traditional evaluation metrics, we introduce a custom metric that emphasizes the overlap of key terms and semantic similarity, providing a more comprehensive assessment of our model's performance. Our code and data are available at https://github.com/midas-research/M-CTG/tree/main.
Out-of-Distribution Aware Classification for Tabular Data
Authors: Amirhossein Ansari, Ke Wang, Pulei Xiong
Venue: CIKM
Year: 2024

Out-of-distribution (OOD) aware classification aims to classify in-distribution samples into their respective classes while simultaneously detecting OOD samples. Previous works have largely focused on the image domain, where images from an unrelated dataset can serve as auxiliary OOD training data. In this work, we address OOD-aware classification for tabular data, where an unrelated dataset cannot be used as OOD training data. A potential solution to OOD-aware classification involves filtering out OOD samples using an outlier detection method and classifying the remaining samples with a traditional classification model. However, seamlessly integrating this approach into downstream optimization tasks is challenging due to the employment of multiple methods. Our approach is turning OOD-aware classification into traditional classification by augmenting the in-distribution training data with synthesized OOD data. This approach continues leveraging traditional classification methods while detecting OOD samples, and the learned model retains the same mathematical properties as traditional classification models, thus, it can be easily integrated into downstream tasks. We evaluate these benefits empirically using real-life datasets. Code is available at https://github.com/ah-ansari/OCT.
Reviewerly: Modeling the Reviewer Assignment Task as an Information Retrieval Problem
Authors: Negar Arabzadeh, Sajad Ebrahimi, Sara Salamat, Mahdi Bashari, Ebrahim Bagheri
Venue: CIKM
Year: 2024

The peer review process is a fundamental aspect of academic publishing, ensuring the quality and credibility of scholarly work. In this talk, we will explore the critical challenges associated specifically with the assignment of reviewers to submitted papers. We will introduce Reviewerly, our innovative solution designed to enhance the efficiency and effectiveness of reviewer assignments by leveraging data from diverse sources, including OpenAlex, PubMed, and DBLP. By modeling the reviewer assignment problem as an information retrieval task, we focus on retrieving a pool of relevant and diverse reviewers for each paper.
iRAG: Advancing RAG for Videos with an Incremental Approach
Authors: Md. Adnan Arefeen, Biplob Debnath, Md. Yusuf Sarwar Uddin, Srimat Chakradhar
Venue: CIKM
Year: 2024

Retrieval-augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for understanding of videos is appealing but there are two critical limitations. One-time, upfront conversion of all content in large corpus of videos into text descriptions entails high processing times. Also, not all information in the rich video data is typically captured in the text descriptions. Since user queries are not known apriori, developing a system for video to text conversion and interactive querying of video data is challenging. To address these limitations, we propose an incremental RAG system called iRAG, which augments RAG with a novel incremental workflow to enable interactive querying of a large corpus of videos. Unlike traditional RAG, iRAG quickly indexes large repositories of videos, and in the incremental workflow, it uses the index to opportunistically extract more details from select portions of the videos to retrieve context relevant to an interactive user query. Such an incremental workflow avoids long video to text conversion times, and overcomes information loss issues due to conversion of video to text, by doing on-demand query-specific extraction of details in video data. This ensures high quality of responses to interactive user queries that are often not known apriori. To the best of our knowledge, iRAG is the first system to augment RAG with an incremental workflow to support efficient interactive querying of a large corpus of videos. Experimental results on real-world datasets demonstrate 23x to 25x faster video to text ingestion, while ensuring that latency and quality of responses to interactive user queries is comparable to responses from a traditional RAG where all video data is converted to text upfront before any user querying.
3DLNews: A Three-decade Dataset of US Local News Articles
Authors: Gangani Ariyarathne, Alexander C. Nwala
Venue: CIKM
Year: 2024

We present 3DLNews, a novel dataset with local news articles from the United States spanning the period from 1996 to 2024. It contains almost 1 million URLs (with HTML text) from over 14,000 local newspapers, TV, and radio stations across all 50 states, and provides a broad snapshot of the US local news landscape. The dataset was collected by scraping Google and Twitter search results. We employed a multi-step filtering process to remove non-news article links and enriched the dataset with metadata such as the names and geo-coordinates of the source news media organizations, article publication dates, etc. Furthermore, we demonstrated the utility of 3DLNews by outlining four applications.
Realistic Synthetic Signed Network Generation and Analysis
Authors: Aikta Arya
Venue: CIKM
Year: 2024

Designing and analyzing network science algorithms, such as node classification, link prediction, and pattern identification (communities, triangles, dense sub-graphs, cliques), require diverse real-world datasets for performance evaluation. However, these datasets are often limited and small due to privacy concerns and platform access policies. This scarcity is even more pronounced for signed networks, as negative relationship data is rarely shared publicly. This PhD thesis aims to address this problem by generating realistic synthetic signed networks using the SNSRM and SISSRM models. Preserving the mesoscopic spectral and structural characteristics of the input signed network is crucial in this process. Additionally, this thesis tackles the challenge of efficiently analyzing elementary network property of triad enumeration by developing an triangle counting algorithm capable of enumerating balanced and unbalanced triads.
Aligning Large Language Model with Direct Multi-Preference Optimization for Recommendation
Authors: Zhuoxi Bai, Ning Wu, Fengyu Cai, Xinyi Zhu, Yun Xiong
Venue: CIKM
Year: 2024

Large Language Models (LLMs) have shown impressive performance in various domains, prompting researchers to explore their potential application in recommendation systems. However, directly applying LLMs to recommendation tasks has proven to be less effective due to the significant gap between the data used for pre-training LLMs and the specific requirements of recommendation tasks. In this study, we propose Direct Multi-Preference Optimization (DMPO), a streamlined framework to bridge this gap and enhance the alignment of LLMs for recommendation tasks. DMPO can be viewed as a pair-wise ranking loss to distinguish between positive and negative samples in recommendation tasks. Furthermore, DMPO improves the performance of LLM-based recommenders by maximizing the probability of positive samples and minimizing the probability of multiple negative samples at the same time. Experimental evaluations are conducted to compare DMPO with traditional recommendation methods and other LLM-based recommendation methods. The results reveal that DMPO significantly enhances the recommendation capabilities of LLMs across three real-world public datasets in few-shot scenarios. Furthermore, the experiments also demonstrate that DMPO exhibits superior generalization ability in cross-domain recommendation. A case study elucidates the reasons behind these consistent improvements and also underscores DMPO's potential as an explainable recommendation system. Our code and data are available at https://github.com/BZX667/DMPO.
Deep Journey Hierarchical Attention Networks for Conversion Predictions in Digital Marketing
Authors: Girim Ban, Hyeonseok Yun, Banseok Lee, David Sung, Simon S. Woo
Venue: CIKM
Year: 2024

In digital marketing, precise audience targeting is crucial for campaign efficiency. However, digital marketing agencies often struggle with incomplete user profiles and interaction details from Advertising Identifier (ADID) data in user behavior modeling. To address this, we introduce the Deep Journey Hierarchical Attention Networks (DJHAN). This novel method enhances conversion predictions by leveraging heterogeneous action sequences associated with ADIDs and encapsulating these interactions into structured journeys. These journeys are hierarchically aggregated to effectively represent ADID's behavioral attributes. Moreover, DJHAN incorporates three specialized attention mechanisms: temporal attention for time-sensitive contexts, action attention for emphasizing key behaviors, and journety attention for highlighting influential journeys in the purchase conversion process. Emprically, DJHAN surpasses state-of-the-art (SOTA) models across three diverse datasets, including real-world data from NasMedia, a leading media representative in Asia. In backtesting simulations with three advertisers, DJHAN outperforms existing baselines, achieving the highest improvements in Conversion Rate (CVR) and Return on Ad Spend (ROAS) across three advertisers, demonstrating its practical potential in digital marketing.
A Learning-based Approach for Explaining Language Models
Authors: Oren Barkan, Yonatan Toib, Yehonatan Elisha, Noam Koenigstein
Venue: CIKM
Year: 2024

We present Learning Attributions (LA), a novel method for explaining language models. The core idea behind LA is to train a dedicated attribution model that functions as a surrogate explainer for the language model. This attribution model is designed to identify which tokens are most influential in driving the model's predictions. By optimizing the attribution model to mask the minimal amount of information necessary to induce substantial changes in the language model's output, LA provides a mechanism to understand which tokens in the input are critical for the model's decisions. We demonstrate the effectiveness of LA across several language models, highlighting its superiority over multiple state-of-the-art explanation methods across various datasets and evaluation metrics.
COSCO: A Sharpness-Aware Training Framework for Few-shot Multivariate Time Series Classification
Authors: Jesus Barreda, Ashley Gomez, Ruben Puga, Kaixiong Zhou, Li Zhang
Venue: CIKM
Year: 2024

Multivariate time series classification is an important task with widespread domains of applications. Recently, deep neural networks (DNN) have achieved state-of-the-art performance in time series classification. However, they often require large expert-labeled training datasets which can be infeasible in practice. In few-shot settings, i.e. only a limited number of samples per class are available in training data, DNNs show a significant drop in testing accuracy and poor generalization ability. In this paper, we propose to address these problems from an optimization and a loss function perspective. Specifically, we propose a new learning framework named COSCO consisting of a sharpness-aware minimization (SAM) optimization and a Prototypical loss function to improve the generalization ability of DNN for multivariate time series classification problems under few-shot setting. Our experiments demonstrate our proposed method outperforms the existing baseline methods. Our source code is available at: https://github.com/JRB9/COSCO.
Covering a Graph with Dense Subgraph Families, via Triangle-Rich Sets
Authors: Sabyasachi Basu, Daniel Paul-Pena, Kun Qian, C. Seshadhri, Edward W. Huang, Karthik Subbian
Venue: CIKM
Year: 2024

Graphs are a fundamental data structure used to represent relationships in domains as diverse as the social sciences, bioinformatics, cybersecurity, the Internet, and more. One of the central observations in network science is that real-world graphs are globally sparse, yet contain numerous "pockets" of high edge density. A fundamental task in graph mining is to discover these dense subgraphs. Most common formulations of the problem involve finding a single (or a few) "optimally" dense subsets. But in most real applications, one does not care for the optimality. Instead, we want to find a large collection of dense subsets that covers a significant fraction of the input graph. We give a mathematical formulation of this problem, using a new definition of regularly triangle-rich (RTR) families. These families capture the notion of dense subgraphs that contain many triangles and have degrees comparable to the subgraph size. We design a provable algorithm, RTRExtractor, that can discover RTR families that approximately cover any RTR set. The algorithm is efficient and is inspired by recent results that use triangle counts for community testing and clustering. We show that RTRExtractor has excellent behavior on a large variety of real-world datasets. It is able to process graphs with hundreds of millions of edges within minutes. Across many datasets, RTRExtractor achieves high coverage using high edge density datasets. For example, the output covers a quarter of the vertices with subgraphs of edge density more than (say) 0.5, for datasets with 10M+ edges. We show an example of how the output of RTRExtractor correlates with meaningful sets of similar vertices in a citation network, demonstrating the utility of RTRExtractor for unsupervised graph discovery tasks.
Fractional Budget Allocation for Influence Maximization under General Marketing Strategies
Authors: Akhil Bhimaraju, Eliot W. Robson, Lav R. Varshney, Abhishek K. Umrawal
Venue: CIKM
Year: 2024

We consider the fractional influence maximization problem, i.e., identifying users on a social network to be incentivized with potentially partial discounts to maximize the influence on the network. The larger the discount given to a user, the higher the likelihood of its activation (adopting a new product or innovation), who then attempts to activate its neighboring users, causing a cascade effect of influence through the network. Our goal is to devise efficient algorithms that assign initial discounts to the network's users to maximize the total number of activated users at the end of the cascade, subject to a constraint on the total sum of discounts given. In general, the activation likelihood could be any non-decreasing function of the discount, whereas, our focus lies on the case when the activation likelihood is an affine function of the discount, potentially varying across different users. As this problem is shown to be NP-hard, we propose and analyze an efficient (1-1/e)-approximation algorithm. Furthermore, we run experiments on real-world social networks to show the performance and scalability of our method.
Hierarchical Graph Latent Diffusion Model for Conditional Molecule Generation
Authors: Tian Bian, Yifan Niu, Heng Chang, Divin Yan, Junzhou Huang, Yu Rong, Tingyang Xu, Jia Li, Hong Cheng
Venue: CIKM
Year: 2024

Recently, generative models based on the diffusion process have emerged as a promising direction for automating the design of molecules. However, directly adding continuous Gaussian noise to discrete graphs leads to the problem that the generated data do not conform to the discrete graph data distribution in the training set. Current graph diffusion models either corrupt discrete data through a transition matrix or relax the discrete data to continuous space for the diffusion process. These approaches make it hard to perform extensible conditional generation, such as adapting to text-based conditions, due to the lack of embedding representations and require significant computation resources due to the diffusion process of the bond type matrix. This paper introduces the Hierarchical Graph Latent Diffusion Model (HGLDM), a novel variant of latent diffusion models that overcomes the problem of applying continuous diffusion models directly to discrete graph data. Meanwhile, based on the latent diffusion framework, HGLDM avoids the issues of computational consumption and lack of embeddings for extensible conditional generation. In addition, by comparing the HGLDM with its variant, the Graph Latent Diffusion Model (GLDM), which only has graph-level embeddings, we validate the advantage of the hierarchical graph structure for capturing the relationship between structure information and molecular properties. We evaluate the performance of our model through various conditional generation tasks, demonstrating its superior performance.
Discovering Denial Constraints Based on Deep Reinforcement Learning
Authors: Lingfeng Bian, Weidong Yang, Jingyi Xu, Zijing Tan
Venue: CIKM
Year: 2024

Numerous algorithms have been proposed for discovering denial constraints (DCs), which are essential and effective for maintaining data consistency. However, existing methods only focus on discovering the complete set of DCs, often resulting in hundreds or even tens of thousands of discovered rules. Such a large number of DCs are impractical for users to verify and utilize. Besides, these methods overlook the intent of users, which requires the discovered DCs to be succinct, relevant, and diverse concurrently. To address these limitations, we introduce DCMiner, a deep reinforcement learning (DRL)-based framework that produces rules satisfying user preferences. Specifically, we first model the discovering process via a kCover Markov decision process to improve efficiency. Then, a graphQ model is introduced to capture the data distribution and facilitate the discovery of DCs. Lastly, we design a reward function that flexibly integrates both objective and subjective criteria to align the discovered rules with user intent, and we propose an efficient training process. Extensive experiments on both real-world and synthetic datasets show that DCMiner can discover succinct, relevant, and diverse rules.
Finding MIDDLE Ground: Scalable and Secure Distributed Learning
Authors: Marco Bornstein, Nawaf Nazir, Ján Drgona, Soumya Kundu, Veronica Adetola
Venue: CIKM
Year: 2024

Edge-computing methods allow devices to efficiently train a high-performing, robust, and personalized model for predictive tasks. However, these methods succumb to privacy and scalability concerns such as adversarial data recovery and expensive model communication. Furthermore, edge computing methods unrealistically assume that all devices train an identical model. In practice, edge devices have varying computational and memory constraints, which may not allow certain devices to have the space or speed to train a specific model. To overcome these issues, we propose MIDDLE, a model-independent distributed learning algorithm that allows heterogeneous edge devices to assist each other in training while communicating only non-sensitive information. MIDDLE unlocks the ability for edge devices, regardless of computational or memory constraints, to assist each other even with completely different model architectures. Furthermore, MIDDLE does not require model or gradient communication, significantly reducing communication size and time. We prove that MIDDLE attains the optimal convergence rate of stochastic gradient descent for convex and non-convex smooth optimization. Finally, our experimental results demonstrate that MIDDLE attains robust and high-performing models without model or gradient communication.