AnoLLM: Large Language Models for Tabular Anomaly Detection

Authors: Che-Ping Tsai, Ganyu Teng, Phillip Wallis, Wei Ding
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Anomaly detection, tabular data, large language models

Abstract:
We introduce AnoLLM, a novel framework that leverages large language models (LLMs) for unsupervised tabular anomaly detection.  By converting tabular data into a standardized text format, we further adapt a pre-trained LLM with this serialized data, and assign anomaly scores based on the negative log likelihood generated by the LLM. Unlike traditional methods that can require extensive feature engineering, and often lose textual information during data processing, AnoLLM preserves data integrity and streamlines the preprocessing required for tabular anomaly detection. This approach can effectively handle mixed-type data, especially those containing textual features. Our empirical results indicate that AnoLLM delivers the best performance on six benchmark datasets with mixed feature types. Additionally, across 30 datasets from the ODDS library, which are predominantly numerical, AnoLLM performs on par with top performing baselines.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 5

Reviews and discussion:
<thread>
creation date:
1730436657739

thread_type:
official_review

<thread_object>
creator: reviewer_sz6k

<content>
summary:
This paper proposes AnoLLM, a new framework that uses large language models (LLMs) for unsupervised tabular anomaly detection. AnoLLM assigns anomaly scores based on the negative log likelihood for anomaly detection. The authors claim that AnoLLM detects anomal samples in raw features and can deal with textual and categorical features. Experimental results show that AnoLLM achieves good performance on six benchmark datasets with mixed feature types. AnoLLM is also competitive with KNN.

strengths:
It’s good to consider different types in tabular data, such as textual, numerical and categorical columns.

Experiment results show good potential for AnoLLM.

weaknesses:
The method seems too simple and lack novelty. It’s easy and direct to consider to use the negative log likelihood for anomaly detection. Even though authors consider different types of columns in tabular data, the process seems to be the same.

I think it’s inappropriate to claim that you are the first to apply LLMs to tabular anomaly detection. There are many works in this area, such as “Anomaly detection of tabular data using llms”, and other corresponding works such as “LLMClean: Context-Aware Tabular Data Cleaning via LLM-Generated OFDs”, “Enhancing Anomaly Detection in Financial Markets with an LLM-based Multi-Agent Framework”.

The backbone LLM is not so famous such as Llama, Qwen, Mistral, etc. Why authors do not use these LLMs?

I don’t understand what the differences in contribution 3 and 4. It seems they are all talking about the experiment.

In experiments, only one of the datasets here has text columns. More such datasets and the dataset with more attributes (features) should be considered.

More ablation studies should be provided such as the impact of Random column permutations. Case study is also missing, I cannot recognize which samples are anomaly.

By the way, the formatting on line 249 could use some improvement.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624640899

thread_type:
official_review

<thread_object>
creator: reviewer_eu7r

<content>
summary:
This paper presents a new framework, AnoLLM, for unsupervised anomaly detection by fine-tuning a pretrained large language model (LLM). The authors use a predefined template to serialize, i.e., convert tabular data into text for the LLM, along with preprocessing to mitigate limitations related to the model's autoregressive nature. They employ the negative log-likelihood across different column permutations to compute an anomaly score for each sample in the test set. The method is compared against various classical and deep learning methods on the ODDS datasets and six new datasets featuring mixed types of attributes. Overall, the approach demonstrates strong performance against baselines, particularly for datasets containing text features.

strengths:
- The paper introduces a novel model type for anomaly detection using a large language model.
- The method provides an effective way to handle text and categorical data as features for anomaly detection, which are typically challenging to manage. To do so, they proposed a method to mitigate length-bias in the LLM’s output probabilities and theoretically validated it.
- The authors demonstrate a preprocessing technique for tabular data, facilitating effective LLM fine-tuning.
- The work is easy to follow and the motivation is clear.

weaknesses:
The paper claims to outperform certain deep learning methods; however, in my experience, some of these methods perform similarly or even better than KNN (which is reported to have results comparable to the proposed method). For example, ICL outperforms KNN on the ODDS benchmark (Shenkar and Wolf, 2022), as does DTE (Livernoche et al., 2024), which was cited but not included as a baseline. The use of column permutations in the paper can be seen as a sort of ensemble strategy, a technique known to slightly improve anomaly detection performance. To ensure a fair comparison, the baselines should also be evaluated using these same permutations, as Appendix C suggests that this step may not be critical, or specific, to AnoLLM. In a small test I conducted, implementing this strategy led to performance improvements in other deep learning methods as well. Including F1-score or AUC-PR results as supplemental material would be helpful, as these metrics are more sensitive to class imbalances, which are common in anomaly detection. Scoring metrics can influence the relative ranking of methods on benchmarks. This claim that AnoLLM outperform deep learning methods should be more cautiously framed.

One key limitation mentioned at the end of the paper is the computational expensiveness of the proposed method. 7 A100 GPUs were used for LLM fine-tuning, this makes it difficult for others to access this model or replicate results of the paper. Since no code is provided, it is even more challenging to verify the reported results. Most anomaly detection methods can run on basic GPUs, or only on CPUs, a significant contrast with AnoLLM. A section discussing inference and training times would help clarify this limitation. I consider this to be the paper’s biggest weakness: its most significant limitation is not addressed at all.

In anomaly detection literature, a clear distinction exists between unsupervised and semi-supervised (or uncontaminated unsupervised) anomaly detection. While we can call them unsupervised methods, since they can be applied in both context, it should be noted that the experiments were conducted in a semi-supervised setting. The distinction lies in whether the training set contains anomalies (unsupervised) or not (semi-supervised). Section 2.1 should be revised to clarify this distinction.

**Minor Comments:**

- In the introduction's first line, "specicious" should be corrected to "specious."
- There is a double colon on line 213 ("equation::").
- Figure 2’s title is missing a space between "yellow" and the parentheses.
- Please use conference or journal citations rather than arXiv versions where possible. Below is a list of those I identified:

  - Liron Bergman and Yedid Hoshen. Classification-based anomaly detection for general data. (ICLR 2020)
  - Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. (ICLR 2023)
  - Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister. Large language models can automatically engineer features for few-shot tabular learning. (ICML 2024)
  - Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. (ICLR 2022)
  - Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. (ICLR 2024)
  - Xuannan Liu, Peipei Li, Huaibo Huang, Zekun Li, Xing Cui, Jiahao Liang, Lixiong Qin, Weihong Deng, and Zhaofeng He. Fakenewsgpt4: Advancing multimodal fake news detection through knowledge-augmented lvlms. (MM2024)
  - Victor Livernoche, Vineet Jain, Yashar Hezaveh, and Siamak Ravanbakhsh. On diffusion modeling for anomaly detection. (ICLR 2024)
  - Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. (ICLR 2019)
  - Tomás Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient estimation of word representations in vector space. (ICLR Workshop 2013)
  - Hu Wang, Guansong Pang, Chunhua Shen, and Congbo Ma. Unsupervised representation learning by predicting random distances. (AJCAI'20)
  - Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. (ICLR 2024)
  - Tianping Zhang, Shaowen Wang, Shuicheng Yan, Jian Li, and Qian Liu. Generative table pretraining empowers models for tabular prediction. (EMNLP 2023)
  - Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. (ICML 2023)
  - Yaqi Zhu, Shaofeng Cai, Fang Deng, and Junran Wu. Do LLMs understand visual anomalies? uncovering LLM capabilities in zero-shot anomaly detection. (MM2024)

questions:
- Did the other baseline also use the ensemble of permutations at inference time?
- What explains the discrepancy in the results of ICL in this paper vs the original paper, which was also tested on OODS?
- How did you choose hyperparameters for the baselines?
- Given the observed trend that larger pretrained models do not seem to benefit AnoLLM, this raises the question: what would happen if we trained a model from scratch using the AnoLLM framework? This feels like a natural question that should have been explored. Did you try this? This might weaken the understanding of the text feature of the model, but it would be interesting to see the impact it has on numerical values.
- What is the total computational cost of the experiments?
- Was experimenting with contaminated training data (the truly unsupervised setting) considered? I ask this because reproducing the paper is not easy, and this is also an important task for anomaly detection.
- What steps are you taking to ensure reproducibility? Will the code be released?
- *See weaknesses*.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625104120

thread_type:
official_review

<thread_object>
creator: reviewer_wusv

<content>
summary:
The paper proposes an innovative use of LLMs, that of detecting anomalies from tabular data. It is well written, and gives good results.

strengths:
S1. The problem statement is innovative.

S2. The paper is quite well written. It is easy to follow and logical.

S3. The results are good.

S4. Instead of simply using large LLMs, small variants are explored, and it is shown that they are no less better.

weaknesses:
W1. The effect of number of decimal digits should have been explored in greater detail.

W2. Similarly, the effect of normalization could have been explored in more detail. Although, the effect of raw numbers is seen, how about simply rounding raw number to x number of decimal digits (and not normalizing) to reduce effect of long decimal numbers, and then using them directly?

W3. What is the effect of not permuting the column names, and having a canonical ordering? Are they not supposed to give even better results?

W4. It will be good to highlight some failure cases, both false positives and false negatives, and try and analyze why that happened.

questions:
W1, W2, W3
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707138658

thread_type:
official_review

<thread_object>
creator: reviewer_18uy

<content>
summary:
The paper proposes AnoLLM, a large language model (LLM) based framework for unsupervised tabular anomaly detection. It utilizes serialization of tabular data into sentences, and finetunes an LLM with the causal language modeling loss. The model have been applied to numerous datasets and shows competitive performances compared to other baselines.

strengths:
- The paper is generally well-written and easy to follow.
- The paper provides theoretical grounds on their model and shows competitive performances compared to other baselines.
- The paper provides some interesting insights on anomaly detection on tabular data with large-language models.

weaknesses:
- An example for anomaly detection (in Figure 1) would be good for reader's understanding of what is an anomaly detection for tabular data. (Possibly an example explicitly included in the dataset used for the experiments)
- From the results, it is difficult to determine which model performs the best without explicit standard deviation (which is provided in the appendix). It would be better to have some simple plots (e.g., critical difference plots) that incorporates some statistical testing to determine which model is performing better.
- While the authors state there is a similar trends for other metrics, it would be good to see the actual results (at least in the appendix), since there is a high imbalance of normal/anomaly.
- The time comparison between the models and the discussion on the trade-off would be very interesting.
- It is confusing to see "50% of normal samples for training" in the evaluation protocol. Clearer description would help better understanding of the experiments.
- One interesting direction for the future works might be to pretrain a model suited for the anomaly detection. Moreover, more elaboration on impacts for the proposed model would be interesting for the conclusion.

questions:
- What is an example of an anomaly detection in tabular data? 
- How does the time comparison look for the baselines? 
- What is "50% of normal samples for training"? It seems that the proposed model is unsupervised and 50% is used to train the unsupervised loss, but what does it mean for other models?
- Have the authors considered dealing numerical columns separately? If LLM falls behind in modeling numerical values, it would be interesting to a simple concatenation of numerical values with the output of 'serialization + LLM' and applying the proposed loss.
- Have the authors considered other encoding methods for categorical variables? (e.g., Target Encoder or the package of Skrub)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732312574302

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for dedicating their time and effort to evaluating our paper. Your valuable comments and suggestions are greatly appreciated. We have revised the draft accordingly, highlighting the updated text in red, and provided a summary of the changes below:


**Additional evaluation metrics:** We have included other evaluation metrics such as AUC-PR and F1 scores in Section H in the Appendix. We also add critical difference diagrams for statistical ranking comparison (Figure 6 and Figure 7). We observe the same trends in these metrics.

**Modifications to baseline models:** We found that there is a bug in the public implementation of the ICL methods.  To address this, we adapted the original code provided by the authors and re-ran the experiments. The updated results show that ICL performs worse than AnoLLM on the mixed-type benchmark but is on par with KNN and AnoLLM on the ODDS benchmark. These corrected results are reflected in the current version.

**Adding diffusion time estimation (DTE) as baselines:** As pointed out by reviewer 3, we also compare the performance of the DTE.  The updated results show that DTE performs worse than AnoLLM on the mixed-type benchmark but is on par with ICL, KNN, and AnoLLM on the ODDS benchmark.

**Adding runtime comparison:** We provide a compute efficiency analysis in Section F of the Appendix, comparing the runtime of various methods. While AnoLLM is slower due to its large backbone LLM, its performance on datasets with mixed-type data is significantly superior, often outperforming other approaches by a considerable margin. This highlights an interesting trade-off for practitioners to consider: balancing computational efficiency with model efficacy. Future work could explore improving AnoLLM's efficiency through techniques like model distillation or by leveraging it as a feature extractor for classical anomaly detection methods.

**Adding an ablation study - effect of random permutation:** As pointed out by reviewers 2 and 4, we also study the performance of AnoLLM without doing random permutations. The updated results in the Section D show that random permutation does provide significant performance improvements on the mixed typed benchmark. 

**Update to the overview figure:** In response to reviewer 1's suggestion, we have added an anomaly example (Figure 1) to better highlight our anomaly detection task.

**Correct typos and arxiv references:** We corrected typos and replaced arxiv references with conference or journal citations where available.

**Code release:** Since we are part of an industry lab, we must adhere to company policies. Therefore, the code will be released on the company's official github once we obtain approval from the legal department.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732760647606

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**Additional appendix section:** we have included an overview of the Appendix in Section A.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a contribution for anomaly detection based on Large Language Models (LLMs) tailored for unsupervised tabular anomaly detection: AnoLLM. A predefined template is used to serialize, i.e., convert tabular data into text for the LLM, along with a preprocessing to mitigate length bias due to the model's autoregressive nature. A negative log-likelihood across different column permutations is used to compute an anomaly score.  A large experimental evaluation is proposed.

Strengths: 
-paper well-written, 
-novel model based on LLMs, 
-solid work, 
-competitive performances, 
-mitigation of length-bias, 
-capacity to deal with tabular data. 

Weaknesses: 
-some elements need more precisions, 
-the impact of normalisation and not permute columns should have been assessed,  
-costly method, 
-experimental evaluation could be enlarged (e.g. include other strategies like column permutations and other ablation studies),

During the rebuttal, authors have answered to the issues raised by the reviewers, they provided additional experiments including ablation studies and updated their paper with most to the new elements. During the discussion, if it was raised that the explanation about the behavior 7B LLMs was not sufficient for some reviewers the general trend is that the work is interesting and of interest quality. Globally, the answers provided convinced the reviewers and have helped to strengthen the paper.  
There is a general consensus for acceptance.  

I propose then accept.  
I encourage the authors to include the additional elements they have committed to adding.