OpenRaiser/Intern-Atlas huggingface dataset:
Veracity Estimation for Entity-Oriented Search with Knowledge Graphs
Authors: Stefano Marchesin, Gianmaria Silvello, Omar Alonso
Venue: CIKM
Year: 2024

In this paper, we discuss the potential costs that emerge from using a Knowledge Graph (KG) in entity-oriented search without considering its data veracity. We argue for the need for KG veracity analysis to gain insights and propose a scalable assessment framework. Previous assessments focused on relevance, assuming correct KGs, and overlooking the potential risks of misinformation. Our approach strategically allocates annotation resources, optimizing utility and revealing the significant impact of veracity on entity search and card generation. Contributions include a fresh perspective on entity-oriented search extending beyond the conventional focus on relevance, a scalable assessment framework, exploratory experiments highlighting the impact of veracity on ranking and user experience, as well as outlining associated challenges and opportunities.
Professionalism-Aware Pre-Finetuning for Profitability Ranking
Authors: Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao
Venue: CIKM
Year: 2024

Opinion mining, specifically in the investment sector, has experienced a significant increase in interest over recent years. This paper presents a novel approach to overcome current limitations in assessing and ranking investor opinions based on profitability. The study introduces a pre-finetuning scheme to improve language models' capacity to distinguish professionalism, thus enabling ranking of all available opinions. Furthermore, the paper evaluates ranking results using traditional metrics and suggests the use of a pairwise setting for better performances over a regression setting. Lastly, our method is shown to be effective across various investor opinion tasks, encompassing both professional and amateur investors. The results indicate that this approach significantly enhances the efficiency and accuracy of opinion mining in the investment sector.
Systems for Scalable Graph Analytics and Machine Learning: Trends and Methods
Authors: Da Yan, Lyuheng Yuan, Akhlaque Ahmad, Saugat Adhikari
Venue: CIKM
Year: 2024

Graph-theoretic algorithms and graph machine learning models are essential tools for addressing many real-life problems, such as social network analysis and bioinformatics. To support large-scale graph analytics, graph-parallel systems have been actively developed for over one decade, such as Google's Pregel and Spark's GraphX, which (i) promote a think-like-a-vertex computing model and target (ii) iterative algorithms and (iii) those problems that output a value for each vertex. However, this model is too restricted for supporting the rich set of heterogeneous operations for graph analytics and machine learning that many real applications demand. In recent years, two new trends emerge in graph-parallel systems research: (1) a novel think-like-a-task computing model that can efficiently support the various computationally expensive problems of subgraph search; and (2) scalable systems for learning graph neural networks. These systems effectively complement the diversity needs of graph-parallel tools that can flexibly work together in a comprehensive graph processing pipeline for real applications, with the capability of capturing structural features. This tutorial will provide an effective categorization of the recent systems in these two directions based on their computing models and adopted techniques, and will review the key design ideas of these systems.
Generating Cross-model Analytics Workloads Using LLMs
Authors: Xiuwen Zheng, Arun Kumar, Amarnath Gupta
Venue: CIKM
Year: 2024

Data analytics applications today often require processing heterogeneous data from different data models, including relational, graph, and text data, for more holistic analytics. While query optimization for single data models, especially relational data, has been studied for decades, there is surprisingly little work on query optimization for cross-model data analytics. Cross-model query optimization can benefit from the long line of prior work in query optimization in the relational realm, wherein cost-based and/or machine learning-based (ML-based) optimizers are common. Both approaches require a large and diverse set of query workloads to measure, tune, and evaluate a query optimizer. To the best of our knowledge, there are still no large public cross-model benchmark workloads, a significant obstacle for systems researchers in this space. In this paper, we take a step toward filling this research gap by generating new query workloads spanning relational and graph data, which are ubiquitous in analytics applications. Our approach leverages large language models (LLMs) via different prompting strategies to generate queries and proposes new rule-based post-processing methods to ensure query correctness. We evaluate the pros and cons of each strategy and perform an in-depth analysis by categorizing the syntactic and semantic errors of the generated queries. So far, we have produced over 4000 correct cross-model queries, the largest set ever. Our code, prompts, data, and query workloads will all be released publicly.
Bridging Knowledge Gaps in LLMs via Function Calls
Authors: Kinjal Basu
Venue: CIKM
Year: 2024

Large Language Models (LLMs) demonstrate impressive abilities across a wide range of NLP tasks. However, their underlying architecture and design come with inherent limitations, which result in issues like hallucinations and constrained reasoning capabilities. Additionally, creating an autonomous AI agent capable of handling complex real-world tasks demands access to real-time information, sensitive data, or external tools-capabilities that most LLMs currently lack. Addressing these issues may require augmenting LLMs with external knowledge through function calling. These function calls serve as an interface between LLMs and the world, enabling access to real-time data, diverse tools, reasoning systems, knowledge graphs, APIs, plugins, code interpreters, and more. The primary objective of this talk is to highlight the significance of function-calling capabilities in bridging the knowledge gap in LLMs, showcase recent research advancements in this area, and discuss existing challenges along with future directions. Also, I will present a training and benchmarking data suite for function calling - API-BLEND and a function calling model - Granite-20B-FunctionCalling.
In Situ Answer Sentence Selection at Web-scale
Authors: Zeyu Zhang, Thuy Vu, Alessandro Moschitti
Venue: CIKM
Year: 2024

Current answer sentence selection (AS2) applied in open-domain question answering (ODQA) selects answers by ranking a large set of candidates, i.e., sentences, extracted from the retrieved text. In this paper, we present Passage-based Extracting Answer Sentence In-place (PEASI), a novel answer selection model optimized for Web-scale setting. This is a Transformer-based network that can jointly (i) rerank passages retrieved for a question and (ii) identify a probable answer from the top passages. We train PEASI with multi-task learning for sharing representations between the passage reranker and answer sentence extractor. We construct a new large-scale QA dataset (WQA) consisting of 800,000+ labeled passages/sentences for 60,000+ questions. The experiment results show that PEASI outperforms AS2 state of the art by 6.51% in accuracy on WQA, from 48.86% to 55.37%. Finally, PEASI is efficient in computing answer sentences, requiring only ~20% of the inferences compared to the standard point-wise setting, which ranks all candidates. We will release WQA and PEASI implementation as we believe that they can contribute to advance the research in QA services at Web scale.
Leveraging Large Language Models for Improving Keyphrase Generation for Contextual Targeting
Authors: Xiao Bai, Xue Wu, Ivan Stojkovic, Kostas Tsioutsiouliklis
Venue: CIKM
Year: 2024

Generating a set of keyphrases that convey the main concepts discussed in a document has been applied to improve various applications including document retrieval and online advertising. The state-of-the-art approaches mostly rely on the neural sequence-to-sequence framework to generate keyphrases. However, training such deep neural networks either requires a significant amount of human efforts in obtaining ground truth keyphrases or suffers from lower quality training data derived from weakly supervised signals. More recently, pre-trained language models are fine-tuned to build more data-efficient keyphrase generation models. Yet, the documents often need to be truncated to adapt to the pre-trained context window. On the other hand, large language models (LLMs) have demonstrated impressive abilities in understanding very long text and generating answers for a wide range of natural language processing tasks, making them great candidates for improving keyphrase generation. There however is a lack of a systematic study on how to use LLMs, especially in an industrial setting that requires low generation latency. In this work, we present an empirical study to facilitate a more informed use of LLMs for keyphrase generation. We compare zero-shot and few-shot in-context learning with parameter efficient fine-tuning using a number of open-source LLMs. We show that using only a handful of well selected human annotated samples, the LLMs already outperform the fine-tuned language model baselines. When thousands of human labeled samples are available, fine-tuned large language models significantly improve the amount and the quality of the generated keyphrases. To enable efficient keyphrase generation at scale, we distill the knowledge from LLMs to a base-size language model. Our evaluation shows significant increase in user reach when the generated keyphrases are used for contextual targeting at Yahoo.
Sequential Optimum Test with Multi-armed Bandits for Online Experimentation
Authors: Fang Kong, Penglei Zhao, Shichao Han, Yong Wang, Shuai Li
Venue: CIKM
Year: 2024

In large-scale online experimentation platforms, experimenters aim to discover the best treatment (arm) among multiple candidates. Traditional A/B testing and multi-armed bandits (MAB) algorithms are two popular designs. The former usually achieves a higher power but may hurt the customers' satisfaction when always recommending a poor arm, while the latter aims at improving the customers' experience (collecting more rewards) but faces the loss of testing power. Recently, [26] combine the advantage of A/B testing and MAB algorithms to maximize the testing power while maintaining more rewards for experiments with two-arm and Bernoulli rewards. However, in practice, the number of arms is usually larger than two and the reward type also varies. In multi-arm experiments, the required sample size to find the optimal arm blows up to guarantee a false discovery rate with the increase of arm numbers, bringing high opportunity costs to experimenters. To save the cost during the long experimental process, we propose a more efficient sequential test framework named Soptima that can work with general reward types. Inspired by the design of traditional MAB algorithms in chasing rewards and A/B testing in maximizing power, we propose an Elimination-type strategy adapted to this framework to dynamically adjust the traffic split on arms. This strategy cooperating with Soptima simultaneously maintains the advantage of the A/B testing in maximizing the testing power, the sequential test methods in saving the sample size, and the MAB algorithms in collecting rewards. The theoretical analysis gives guarantees on the Type-I, Type-II, and optimality error rates of the proposed approach. A series of experiments from both simulation and industrial historical data sets are conducted to verify the superiority of our approach compared with available baselines.
CheckGuard: Advancing Stolen Check Detection with a Cross-Modal Image-Text Benchmark Dataset
Authors: Fei Zhao, Jiawen Chen, Bin Huang, Chengcui Zhang, Gary Warner
Venue: CIKM
Year: 2024

The prevalence of check fraud, particularly with stolen checks sold on platforms such as Telegram, creates significant challenges for both individuals and financial institutions. This underscores the urgent need for innovative solutions to detecting and preventing such fraud on social media platforms. While deep learning techniques show great promise in detecting objects and extracting information from images, their effectiveness in addressing check fraud is hindered by the lack of comprehensive, open-source, large training datasets specifically for check information extraction. To bridge this gap, this paper introduces "CheckGuard," a large labeled image-to-text cross-modal dataset designed for check information extraction. CheckGuard comprises over 7,000 real-world stolen check image segments from more than 15 financial institutions, featuring a variety of check styles and layouts. These segments have been manually labeled, resulting in over 50,000 samples across seven key elements: Drawer, Payee, Amount, Date, Drawee, Routing Number, and Check Number. This dataset supports various tasks such as visual question answering (VQA) on checks and check image captioning. Our paper details the rigorous data collecting, cleaning, and annotation processes that make CheckGuard a valuable resource for researchers in check fraud detection, machine learning, and multimodal large language models (MLLMs). We not only benchmark state-of-the-art (SOTA) methods on this dataset to assess their performance but also explore potential enhancements. Our application of parameter-efficient fine-tuning (PEFT) techniques on the SOTA MLLMs demonstrates significant performance improvements, providing valuable insights and practical approaches for enhancing model efficacy on this task. As an evolving project, CheckGuard will continue to be updated with new data, enhancing its utility and driving further advancements in the field. Our PEFT-based MLLM code is available at: https://github.com/feizhao19/CheckGuard. For data access, researchers are required to contact the authors directly.
UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models
Authors: Qi Liu, Yongyi He, Tong Xu, Defu Lian, Che Liu, Zhi Zheng, Enhong Chen
Venue: CIKM
Year: 2024

Multimodal Entity Linking (MEL) is a crucial task that aims at linking ambiguous mentions within multimodal contexts to the referent entities in a multimodal knowledge base, such as Wikipedia. Existing methods focus heavily on using complex mechanisms and extensive model tuning methods to model the multimodal interaction on specific datasets. However, these methods overcomplicate the MEL task and overlook the visual semantic information, which makes them costly and hard to scale. Moreover, these methods cannot solve the issues like textual ambiguity, redundancy, and noisy images, which severely degrade their performance. Fortunately, the advent of Large Language Models (LLMs) with robust capabilities in text understanding and reasoning, particularly Multimodal Large Language Models (MLLMs) that can process multimodal inputs, provides new insights into addressing this challenge. However, how to design a universally applicable LLMs-based MEL approach remains a pressing challenge. To this end, we propose UniMEL, a unified framework which establishes a new paradigm to process multimodal entity linking tasks using LLMs. In this framework, we employ LLMs to augment the representation of mentions and entities individually by integrating textual and visual information and refining textual information. Subsequently, we employ the embedding-based method for retrieving and re-ranking candidate entities. Then, with only ~0.26% of the model parameters fine-tuned, LLMs can make the final selection from the candidate entities. Extensive experiments on three public benchmark datasets demonstrate that our solution achieves state-of-the-art performance, and ablation studies verify the effectiveness of all modules. Our code is available at https://github.com/Javkonline/UniMEL.
PARs: Predicate-based Association Rules for Efficient and Accurate Anomaly Explanation
Authors: Cheng Feng
Venue: CIKM
Year: 2024

While new and effective methods for anomaly detection are frequently introduced, many studies prioritize the detection task without considering the need for explainability. Yet, in real-world applications, anomaly explanation, which aims to provide explanation of why specific data instances are identified as anomalies, is an equally important task. In this work, we present a novel approach for efficient and accurate model-agnostic anomaly explanation for tabular data using Predicate-based Association Rules (PARs). PARs can provide intuitive explanations not only about which features of the anomaly instance are abnormal, but also the reasons behind their abnormality. Our user study indicates that the anomaly explanation form of PARs is better comprehended and preferred by regular users of anomaly detection systems as compared to existing model-agnostic explanation options. Furthermore, we conduct extensive experiments on various benchmark datasets, demonstrating that PARs compare favorably to state-of-the-art model-agnostic methods in terms of computing efficiency and explanation accuracy on anomaly explanation tasks. The code for our experiments is available at https://github.com/cfeng783/PARs.
General Time Transformer: an Encoder-only Foundation Model for Zero-Shot Multivariate Time Series Forecasting
Authors: Cheng Feng, Long Huang, Denis Krompass
Venue: CIKM
Year: 2024

We present General Time Transformer (GTT), an encoder-only style foundation model for zero-shot multivariate time series forecasting. GTT is pretrained on a large dataset of 200M high-quality time series samples spanning diverse domains. In our framework, we consider multivariate time series as a distinct category of images characterized by varying number of channels, and represent each time series sample as a sequence of non-overlapping curve shapes (patches) within an unified numerical magnitude. Furthermore, we formulate the task of multivariate time series forecasting as a problem of predicting the next curve shape based on a window of past curve shapes on a channel-wise basis. Experimental results demonstrate that GTT exhibits superior zero-shot multivariate forecasting capabilities on unseen time series datasets, even surpassing state-of-the-art supervised baselines. Additionally, we investigate the impact of varying GTT model parameters and training dataset scales, observing that the scaling law also applies in the context of zero-shot multivariate time series forecasting. The codebase of GTT is available at https://github.com/cfeng783/GTT.
Towards Efficient Temporal Graph Learning: Algorithms, Frameworks, and Tools
Authors: Ruijie Wang, Wanyu Zhao, Dachun Sun, Charith Mendis, Tarek F. Abdelzaher
Venue: CIKM
Year: 2024

Temporal graphs capture dynamic node relations via temporal edges, finding extensive utility in wide domains where time-varying patterns are crucial. Temporal Graph Neural Networks (TGNNs) have gained significant attention for their effectiveness in representing temporal graphs. However, TGNNs still face significant efficiency challenges in real-world low-resource settings. First, from a data-efficiency standpoint, training TGNNs requires sufficient temporal edges and data labels, which is problematic in practical scenarios with limited data collection and annotation. Second, from a resource-efficiency perspective, TGNN training and inference are computationally demanding due to complex encoding operations, especially on large-scale temporal graphs. Minimizing resource consumption while preserving effectiveness is essential. Inspired by these efficiency challenges, this tutorial systematically introduces state-of-the-art data-efficient and resource-efficient TGNNs, focusing on algorithms, frameworks, and tools, and discusses promising yet under-explored research directions in efficient temporal graph learning. This tutorial aims to benefit researchers and practitioners in data mining, machine learning, and artificial intelligence.
Improving Adversarial Transferability via Frequency-Guided Sample Relevance Attack
Authors: Xinyi Wang, Zhibo Jin, Zhiyu Zhu, Jiayu Zhang, Huaming Chen
Venue: CIKM
Year: 2024

Deep neural networks (DNNs) are known to be vulnerable to adversarial examples. To facilitate model safety, transfer-based attacks employ surrogate models to craft adversarial examples. In this work, we firstly study the intricate mechanisms of such attacks. We observe a correlation between the sharpness of decision boundaries in model sensitive regions and overfitting during adversarial training, which hampers the adversarial examples' transferability. To address this issue, we propose a novel approach termed Frequency-Guided Sample Relevance Attack (FGSRA). Specifically, we leverage frequency information to explore similar sensitive regions across different models, thereby generating neighborhood samples. Additional similarity weights are subsequently introduced to assess the adversarial contribution of the neighborhood samples. A hybrid gradient is then obtained to thoroughly exploit neighborhood information within input samples. Extensive experiments demonstrate the prominent performance of our approach. Compared to other state-of-the-art benchmarks on surrogate model Inc-v3, our method has an average improvement of 27.21% for normally trained CNNs and 42.1% for adversarially trained CNNs. Moreover, we achieve an average improvement of 24.6% for ViTs. Our code is available at:https://github.com/LMBTough/FGSRA
Hypergraph Hash Learning for Efficient Trajectory Similarity Computation
Authors: Yuan Cao, Lei Li, Xiangru Chen, Xue Xu, Zuojin Huang, Yanwei Yu
Venue: CIKM
Year: 2024

Trajectory similarity computation is a fundamental problem in various applications (e.g., transportation optimization, behavioral study). Recent researches learn trajectory representations instead of point matching to realize more accurate and efficient trajectory similarity computation. However, these methods can still not be scaled to large datasets due to high computational cost. In this paper, we propose a novel hash learning method to encode the trajectories into binary hash codes and compute trajectory similarities by Hamming distances which is much more efficient. To the best of our knowledge, this is the first work to conduct hash learning for trajectory similarity computation. Furthermore, unlike the Word2Vec model based on random walk strategy, we utilize hypergraph neural networks for the first time to learn the representations for the grids by constructing the hyperedges according to the real-life trajectories, resulting in more representative grid embeddings. In addition, we design a residual network into the multi-layer GRU to learn more discriminative trajectory representations. The proposed Hypergraph Hash Learning for Trajectory similarity commutation is an end-to-end framework and named HHL-Traj. Experimental results on two real-world trajectory datasets (i.e., Porto and Beijing) demonstrate that the proposed framework achieves up to 6.23% and 15.42% accuracy gains compared with state-of-the-art baselines in unhashed and hashed cases, respectively. The efficiency of trajectory similarity computation based on hash codes is also verified. Our code is available at https://github.com/caoyuan57/HHL-Traj.