AnoLLM: Large Language Models for Tabular Anomaly Detection
Authors: Che-Ping Tsai
Ganyu Teng
Phillip Wallis
Wei Ding
Venue: ICLR 2025 Poster (2025)
Keywords: Anomaly detection
tabular data
large language models
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We introduce AnoLLM, a novel framework that leverages large language models for unsupervised tabular anomaly detection.

Abstract: We introduce AnoLLM, a novel framework that leverages large language models (LLMs) for unsupervised tabular anomaly detection.  By converting tabular data into a standardized text format, we further adapt a pre-trained LLM with this serialized data, and assign anomaly scores based on the negative log likelihood generated by the LLM. Unlike traditional methods that can require extensive feature engineering, and often lose textual information during data processing, AnoLLM preserves data integrity and streamlines the preprocessing required for tabular anomaly detection. This approach can effectively handle mixed-type data, especially those containing textual features. Our empirical results indicate that AnoLLM delivers the best performance on six benchmark datasets with mixed feature types. Additionally, across 30 datasets from the ODDS library, which are predominantly numerical, AnoLLM performs on par with top performing baselines.

Published as a conference paper at ICLR 2025
ANOLLM: LARGE LANGUAGE MODELS FOR TABU- LAR ANOMALY DETECTION
Che-Ping Tsai, Ganyu Teng, Phil Wallis, Wei Ding Amazon
(chepingt, tenganyu, phwallis,
ABSTRACT
We introduce AnoLLM, novel framework that leverages large language mod- els (LLMs) for unsupervised tabular anomaly detection. By converting tabular data into a standardized text format, we further adapt a pre-trained LLM with this serialized data, and assign anomaly scores based on the negative log likeli- hood generated by the L.M.. Unlike traditional methods that can require extensive feature engineering, and often lose textual information during data processing, AnoLLM preserves data integrity and streamlines the preprocessing required for tabular anomaly detection. This approach can effectively handle mixed-type data, especially those containing textual features. Our empirical results indicate that AnoLLM delivers the best performance feature types. Additionally, across 30 datasets from the ODDS library, which are predominantly numerical, AnoLLM performs par with top performing base- lines.
six benchmark datasets with mixed
INTRODUCTION
Anomaly detection (AD) seeks examine specific data points to identify rare, specious oc- currences that deviate from established behavior patterns. AD has been applied to a wide range of applications spanning computer vision (Chandola al. 22099) natural language processing (NLP) (Schölkopf et al. 2001) and tabular data (Hawkins) 1980). Among these, tabular anomaly detection is particularly crucial, as tabular data is fundamental format in machine learning which has been used extensively in applications related to cyber-attack prevention (Landauer et al. (2223), detecting fraudulent financial transactions (Dornadula & Geetha 2019), and diagnosing medical conditions (Fernando et al. 2021).
Conversely, as large language models (LLMs) have shown remarkable performance on various NLP tasks (Radford et al. 2018 Chung al. 2024 Wei et al. 2022). researchers are keen to understand the capabilities of LLMs applied to other modalities, including tabular data (Fang et al., 2024).
While previous studies have demonstrated the proficiency of LLMs in prediction (Dinh al. 2022
table understanding (Sui et al., 2004).. and data generation (Borisov et al.) (203),, their effectiveness in handling the task of tabular anomaly detection remains largely unexplored.
Applying L.Ms to tabular anomaly detection presents several challenges. First, tabular data is inherently structured in two dimensions, which does not align well with the linear and sequential nature of LLM inputs. Second, unlike traditional tabular classification tasks, unsupervised anomaly detection lacks labels, making the in-context learning framework unfeasible. Third, while existing studies demonstrate that LLMS can be fine-tuned as generative models for tabular data, naively using output probabilities as anomaly scores introduces length- issues as the token-level probabilities are aggregated.
In this paper we introduce AnoLLM, a novel framework of using LLMs for unsupervised tabular anomaly detection. AnoLLM is comprised of three phases. In the initial phase, we serialize each row of tabular dataset into standardized text format. During the training phase, pretrained LLM is fine-tuned with the before mentioned serialized tabular data via eext-token-predictien where the LLM learns to be a sequential tabular data generator that models the underlying data distribution.
Work completed during an internship Amazon

---

Published as a conference paper at ICLR 2025
Training Data
AnoLLM Training
Transsction valion
Transsction
transsctiar value 6.2, user
San Innn,
13.00
Palo
Surnyyvuu,
Binning Permute
rroncaetiar valian is 8-1, voar in it.
1000.00
Orlando,
Test Data
AnoLLM Inference
Nagative log- likelihood
$2.58
Pormune
times
Iwn Fromeettin Twatten
Figure 1: Overall framework AnoLLM. During the preprocessing stage, numerical columns are binned into groups, and each data row is transformed into natural language sequence with ran- domly shuffled order of columns. In the training stage, pretrained LLM is fine-tuned using the preprocessed tabular data. During inference, anomaly scores are determined by averaging the nega- tive likelihoo across random permutations of the test data.
In the inference phase, anomaly scores are determined using the negative log likelihood produced by the LLM for the test samples provided, with higher scores indicating greater surprise by the model when encountering the inputs. Since naively multiplying all token probabilities prone to length-bias issue, propose normalize probabilities for each textual column individually, while retaining the raw probabilities for numerical and categorical columns, as supported by theoretical validation.
AnoLLM has the following advantages over existing tabular anomaly detection approaches. First, existing approaches often discard feature names and categorical values when transforming data into vectors, lcading to a potential loss of valuable information. Second, these methods rely heavily on feature engineering to handle mixed-type tabular data (Borisov et al. 2022), such as text data, which cannot be easily transformed into numerical and categorical features. For instance, system log anomaly detection requires extensive feature engincering to transform logs into real-valued vec- tors (He et al. 2017). Moreover, existing approaches often require the use of imputers handle missing values effectively (Emmanuel et al., 202I). contrast, using text representations as inputs offers more flexible and comprehensive representation of data compared to traditional anomaly detection models.
Our contributions summarized below.
We propose AnoLLM, novel framework for adapting LLMs for unsupervised tabular anomaly detection. To the best of our knowledge, AnoLLM the first tabular AD approach that is capable of handling raw textual features without pre-processing.
To mitigate the length- bias in the LLM's output probabilities, we propose a novel method for computing anomaly scores based on LLM-generated probabilities, backed by theoreti- cal validation.
Empirically, we demonstrate that AnoLLM outperforms all existing methods, including four classical approaches and eight deep learning-basse approaches, across six datasets containing mixed tabular data types.
Despite known limitations of LLMs in performing basic arithmetic calculations (Shen et al., 2023 Lee et al., 2023a), our empirical evidence shows that the overall performance of AnoLLM matches best- pers-peerooming baselines, k-nearest neighbors, internal contrastive learning, and diffusion time estimation, on 30 datasets in the ODDS library (Rayana 2016), which primarily contain numerical features.
We also conduct extensive analysis the key design choices in AnoLLM, including various feature binning methods and the impact of different LLM sizes. To the best our knowledge, we are the first apply L.MSs tabular anomaly detection, achieving state-of-the-art results.
2

---

Published as a conference paper at ICLR 2025
2 ANOLLM: LARGE LANGUAGE MODELS FOR MIXED-TYPE TABULAR ANOMALY DETECTION
This section introduces the A approach for detecting anomalies in tabular data, which com- prises three main stages. An overview of the AnoL.MM approach is shown in Figure [ In the first stage, tabular data is serialized into text format. During the training stage, the serialized data used to fine-tune pretrained large language models through next token prediction. In the inference stage, anomaly scores are assigned based on the negative log-likelihood generated by the LLMs for the given test samples. Each stage described in detail below.
2.1 PROBLEM FORMULATION
We focus uncontaminated, unsupervised anomaly detection using tabular data, represented as X = [x1,- ... x, where x; E X' is the 7th row of the data. The objective is to train an anomaly detector, f: X + R. solely on the normal training samples X. The anomaly detector assigns scores to each sample as fx)), with higher scores indicating a higher likelihood of being an anomaly.
During the test stage, we are provided with a labeled test set, D' {(x1 v1).... (xn, ).) where each X E X' and y E {0, 1 with anomalies being labeled as y 1. After applying the anomaly detector f to each test sample, we obtain anomaly scores {f(x)), ... f(x,,, threshold, or probability region, can be employed to classify whether a sample is an anomaly based on these
scores.
Suppose that we have tabular dataset with rows and d columns (features). We denote the column (feature) names as c = [ci..... cal. with each o being a natural language text sequence such as "age" or "job description". Let the value of each entry be Xi, with i {1,.... n) and j e {1. [1.....d] .d.. We note that x, can represent different types of features, such as numerical, categorical or textual features.
2.2 SERIALIZATION OF TABULAR DATA
For serialization, follow previous tabular data serialization methods (Hegselmann et al. 2023 Borisov et al. 2023) by simply concatenating column names and column values. Specifically, we use the following template: column name is column value. Formally, the jth column of the ith sample x; is transformed into the text sequence (x;x; = "cy is F(x.,)", where F.(.) is text encoder that pre-process the feature into the text. form, which introduced below.
Feature preprocessing: Since LLMs only take text as inputs, we use a text encoder E(-) to trans- form every row of data into text format. Specifically, for textual columns and categorical columns, we directly use their original form as they are already represented natural language. For numeri- cal columns, do standard rescaling and then round them to single-digit decimals. This operation effectively reduces the number of digits when the inputs are very large, or have high precision. Formally, apply the following affine transformation to each element Xi.j-
E(ris)
10 10
1
X1, -mj
if ey is numerical column.
(1)
2j
Here, my and 2j are the mean and the standard deviation of the jth column in the training set, i.e. my Xx., .x.j). and 2, Xtd. ,,,,,, 1.7 is a rounding-to- integer operator. This affine transformation normalizes the numerical values and converts them into decimal number with a single digital place. This operation effectively converts large or long-digit numbers into decimal numbers containing only a few digits. This step is crucial because obtain the anomaly scores by computing negative log- likelioo,ood, which sensitive to token lengths.
This standard rescaling operation can be viewed a feature binning method since most of the numerical values fall into a range centered around the mean 0.0, containing a finite number of single-digit decimal numbers. In this approach, we categorize numerical values and allow AnoLLM to treat these numerical attributes as categorical features. The categories are represented by single- digit decimals that retain information related to magnitude. We choose standard rescaling for
3

---

Published as a conference paper at ICLR 2025
binning method since it outperformed other feature binning methods during experimentation, such as equal-width binning and percentile binning, as shown in Section 3.3
Handling missing values and column names: For all missing feature values, the text en- coder E treats them as separate category and maps them to the word "Unknown." For tabu- lar data with missing column names, we manually assign names in alphabetical order follows: "A", "B", ..... "Z", "AA", "AB", ..... "AZ", "BA", "BB",. etc. As we will demonstrate in the ex- periments, A remains effective even the column names are not provided.
Random column permutations: Since there is no inherent ordering among columns, we permute the d serialized text sequences corresponding to each column using a randomly selected permutation Sa. where Sa is symmetric group of d elements, containing all d! possible permutations. The resulting text sequence representing the row x; is as follows:
Te xT) = " (1) is E(x,,01)) Cx(2) is ((x))
(r( is ((x))= "
(x))
(xx)
(xta))
which contains d permuted columns names and their corresponding values on the ith row. During training, we randomly select permutation ~ Sa at each gradient step to ensure that the LLMs do not depend on the feature order. For inference, we fix the permutation for all test samples to maintain consistency.
2.3 FINE-TUNING LARGE LANGUAGE MODELS
Finally, we describe the fine-tuning process of a pretrained LLM with serialized tabular data T = (T, xx)
Sa, E f1. n)}. For each T, the serialized tabular data is first transformed into sequence of tokens, i.e. Tokenize(s) = (W1,... ... ([(()) of length ((s) using the tokenizer of the given LLM. We then fine-tune the pretrained LLM in auto-regressive manner using the causal language modeling loss:
La
1e (0,s) SET
E
E SET
Σ log po (we (0i, WK--1
where 0 is the parameters of the LLM. By fine-tuning a pretrained LLM with causal language mod- eling loss, it can learn to predict column values based on the columns that have already been seen. Given that pretrained LLMs have already acquired semantic understanding of the features, it can leverage its extensive contextual knowledge to predict the subsequent column values. For example, the features age of vehicle and vehicle price have clear coherence since vehicle price decreases as it ages. Furthermore, by representing numerical feature values with decimal numbers, the pretrained L.MMs can utilize their basic arithmetic abilities to encode quantity of numerical val- ues (Hanna et al., 2024, Stoofo et al. 2023). This fine-tuning process not only reinforces the LLMs' comprehension the specific formatting specified by our serialization method but also enhances their ability to learn the dependencies between different features.
2.4 CALCULATION OF ANOMALY SCORES
Anomaly scores are calculated using the negative log-likelihood of the fine-tuned LLM,, as this score reflects the (negative) probability of sample being generated by the LLM. A higher negative log-likelihood indicates greater surprise from the LLM when predicting the inputs. Assuming that the majority of the training samples are normal, the fine-tuned LLM should predict normal inputs more accurately and assign higher probabilities to them. Since we found that directly computing the negative ligliiill for textual features, which vary greatly length, could introduce length bias, we employ different methods to calculate anomaly scores.
Case 1: Datasets comprised of only numerical and categorical columns: We first discuss how to obtain probabilities of all categories in the categorical column cj. Given the prefix sequence of the test sample x/, "c, is E(x;,) , 02 is 2xx ey is ", our objective is to obtain the conditional probabilities of all possible categories, given this prefix.
4

---

Published as a conference paper at ICLR 2025
Theorem 1. Given prefix token sequence Su. assume that there are only q possible subsequent
token sequences, which are s!2), s((), Let each sequence s have probability P(s (so)
to be sampled. Let each sequence ş consisting lf tokens, denoted w Wi2 ..... w (3) Then
(8)
the optimal language model pe- that minimizes the causal language modeling loss in Eqn should align with the true distribution to satisfy the following equation:
4.
for all i E ... ,9.. 10-00-(800) (5)) = II po- (wk (so, 1 (8)
W x0-11 = P((s())s))) (4)
s ((2) 9(9) with 9 class of one category and (s)())))) with their true conditional probabilities given the previously observed features, the above theorem shows that the product of all token probability resembles the probability of being in the corresponding category even though categorical names may have different lengths for the optimal LLMs. As LLMs have been fine-tuned with extensive tabular data using the same template as in Eqn they should inher- ently recognize when to generate categorical data, the L.MSs should meet the optimality conditions required for the above theorem to be applicable. Since we treat numerical columns as categorical, the theorem should also apply to numerical columns.
When we instantiate ,(1),
Note that we apply the same template, "c1 is E(x;,) ), 02 is E(x',)) Cj is ", across all test sam- ples. Therefore, the token probabilities associated with the feature names, such as "c, is ", should be consistent across all test samples. As a result, we can compute anomaly scores by simply summing the log probabilities of every token in the sequence.
Next, given that the fine-tuned LLM has learned to generate tabular data with arbitrary column orders, we run the LLM inference r times with different column orders in the test stage to reduce variance. Specifically, we compute anomaly scores using the following formula: given test samples X = (x1.''' xm ), the anomaly score of the sample x is given as follows:
((Te(T,)())
score(x!) - : Σ leim Te(T,, Σ j=1
Σ Σ
- log loog po w6-1), (5)
j=1
Sa are different permutations over {1,2, ... .d), TT. x()) is the token sequence of the serialized tabu- lar data x/ with permutation πj. and this sequence has length (TT x()). For consistency, we apply the same permutations Ti, TT, Tr to all test samples. The impact of the number of permutations r' analyzed in the experimental section.
where T1, "2, Tr (wi,-.., (((T.((,)xx)))
For datasets containing textual features, we ob- serve that 5 heavily influenced by text lengths. Longer texts naturally yield higher anomaly scores due to the summation over more words. To address this, we normalize each textual column by its length separately and sum the scores up. Specifically, we compute anomaly scores using the formula, score(x')= = 1 Σド= En nnor Ty, (7) (1) (x()). with enor defined below:
Case 2: Datasets containing textual columns:
Ty.
-1 log po (((x(,,(()) "t (1)(()((X),,-) ... (,,,,(,--1((xx))) (x) is is*)
Enor (0, Ty, (1) (1)((((()))) (xx) g(Cr,(00)
(6)
When (z)(k) is a textual column, we set (k)) to the number of tokens in E(x When (x) is not textual column, we (()x,())) 1 so that it is normalized by the number of tokens. Different from Eqn we compute the negative log-likelihood only when predicting column values (k))), ). and normalize the loss only for textual columns. We exclude the likelihho of the column name, (A) is " since the fine-tuned LLM recognizes is as part of the serialization template and therefore should not be included when counting the text length. Normalization is done separately for each column to ensure that columns with longer texts do not disproportionatlly influence the anomaly scores.
3 EXPERIMENTS
Datasets: Since popular anomaly detection benchmarks, such as ADBench (Han et al., 2022) and the ODDS library (Rayanal 2016). mainly consist of numerical features, we manually col-
5

---

Published as a conference paper at ICLR 2025
lect six datasets that contain mixed types of features. The six datasets derived from ODDS library (Rayanal 2016), the fraud dataset benchmarks (Grover et al. 2022) and Kaggle. The dataset statistics are described in Table [ To demonstrate the ability of Ano to accommodate numeri- cal columns, we also evaluate the approach on 30 datasets from the ODDS library, which are mainly composed of numerical features. The ODDS library is collected from various domains, such as chemistry, healthcare, and astronautics. Due to space constraints, we include dataset statistics with respect the ODDS library Table of the Appendix.
Datasets
# Samples
# text
# num
category
anomaly (%)
Fake job posts Grover et al. 2022)
17,880
3
866 (4.84%)
Fraud ecommerce Grover et al. 2022)
151,112
0
1
6
14,151 (9.36%)
Lymphography (Rayana 2016
148
0
3
15
6 (4.05%)
Seismic [Rayanal 2016
2,584
0
14
4
170 (6.58%)
Vehicle insurance (Kaggle)
15,420
0
8
24
923 (5.99%)
newsgroup
11,905
0
0
591 (4.96%)
Table I: Dataset statistics for the six datasets from the mixed-type benchmark. # text, # num, and # category stand for the numbers textual, numerical, and categorical features in each dataset.
Baselines: We compare against 11 prominent methods in the field of tabular anomaly detection.
For classical approaches, we compare against IForest (Liu et al. 2008). PCA (Shyu et al.) 2003).
KNN (Ramaswamy et al., 2000) and ECOD (Li al. 2022). These
classical approaches that
are still competitive against deep-learning haa--lannnnn-beeee based methods (Han et al. 2022). For deep-lcarning based approaches, we compare against DeepS (Ruff et al.] 2018), RCA (Liu et al. (02T), and self-supervised learning based approaches including SLAD (Xu et al. (023b). GOAD (Bergman &
Hoshen 2020), NeuTral (Qiu et al. 2021), ICL (Shenkar & Wolf 2022), DTE (Livernoche al.)
2024) and REPEN (Pang et al. 2018). ) All methods are implemented with identical dataset parti- tioning.
Implementation details: We use SmolLM- 135MMM and SmolLM 360M (Allal al. 2024 as our backbone LLM since they represent state-of-the-art small models among open-weights LLMs. We choose small models (135M and 360M parameters) since the experimental results in Section 3.4 suggest that increasing model sizes does not provide much improvements. Fine-tuning is conducted for 2,000 steps with an Adam optimizer (Loshchilov & Hutter 2019) with learning rate 5 x 10-5 across all datasets as the training loss converges uniformly. Batch sizes are adjusted for each dataset to accommodate the varying lengths of scrialized data. During inference, we select the number of permutations = 21 since further increasing does not result in any observed improvement. Fine- tuning and inference are performed on seven Nvidia A 110 40GB GPUs hosted on Amazon EC2 P4 Instances. Detailed hyperparameters are shown in Table of the Appendix.
For baseline implementation, we use PyOD library (Zhao et al. 2019) for shallow methods and Dee- pOD library (Xu et al.) 2023a) for weee-lennnig--ee based approaches. We standardize all numerical features to have a mean zero and standard deviation of one as AnoL.MM does. For categorical features, we group all rare classes with less than 1% of samples together and use one-hot encoding For textual columns, we use averaged word2vec embeddings (Mikolov et al. (201)) across all words in each column. The word2vec embeddings are 300-dimensional and are trained from the Google news dataset For each method, we used the best best-performing set of hyperparameters reported in their original paper. For others not specified, we use the default hyperparameters as suggested by DeepOD and PyOD toolkits.
we con- duct experiments an uncontaminated, unsupervised setting. The training set consists of a random sample of 50% from the pool of normal examples, with the test set comprising the remaining normal examples, along with all anomalies. We randomly split each dataset using different random seeds
Evaluation protocols: Following prior works (Shenkar & Woli 2022 Xu et al. 2023b
'For the fake job post dataset, the model is fine-tuned for 10,000 steps due longer convergence time. FFor SLAD, we use ordinal encoding for categorical features as it does not support one-hot encoding. https::/code.googlee wwwgge/coo//aaaai////ltttttttooo

---

Published as a conference paper at ICLR 2025
Fake job posts
Fraud ccommerce
Lympho- graphy
Vehicle insurance
20news groups
Methods
Datasets
Seismic
Average
Classical methods
Iforest
0.755
0.501
0.673
0.692
0.496
0.623
0.623
PCA
0.724
0.647
0.826
0.692
0,509
0.623
0.670
KNN
0,636
0.860
0.738
0,524
0.605
0.727
ECOD
0.512
0.755
0.830
0.692
0.509
0.62
0.653
Deep learning based methods
DeepSVDD
0.561
0.899
0.713
0.505
0.597
0.713
RCA
0.629
0.919
0.727
0.531
0.546
0.725
SLAD
0.603
0.998
0,964
0.714
0.556
0.64
0.746
GOAD
0,566
0.998
0.817
0.717
0.512
0.707
NeuTral
0,548
0.847
0.681
0.507
0.658
0.707
ICI.
0.699
0.827
0.719
0.501
0.671
0.736
DTE
0.548
I
0.909
0.714
0.512
0.6
0.714
REPEN
0.653
0.808
0.724
0,513
0.574
0.712
AnoLLM
SmoILM-135M
0.800 0.814
0.968 0.995
0.712 0.746
0.569 0.555
0.766 0.752
0.803
SmoILM-360M
0.810
Table 2: AUC-ROC scores for all methods on the six datasets containing mixed types of features.
and reported the averaged results. We employ Area Under the Receiver Operating Characteristic Curve (AUC-ROC) as our primary evaluation metric. Other metries, e.g. area under the precision- recall curves (AUC-PRC) and FI scores, reflect similar trends in experimental results and are shown in Section H of the Appendix. Moreover, a runtime comparison shown in Section of the Appendix.
3.1
MAIN RESULTS ON MIXED-TYPE TABULAR DATASETS
Table 2 shows the overall performance of different methods the six mixed-type datasets. As shown in the table, AnoLLM consistently delivers the best re- sults all datasets. Specifically, AnoLLM with the SmolLM-360M backbone exhibits least a 6.4%
0.85
AUC-OC
improvement the baseline methods. This per- formance especially notable on datasets with tex-
tual columns, such as the fake job posts and 20 news- groups datasets, where AnoLLM substantially out- performs other methods. This highlights the ad- vantage of utilizing LLMs for anomaly detection, as they can process raw text inputs effectively and do not require the extensive feature engineering for textual inputs that is necessary for baseline meth- ods. We also note that although AnoLLM's effi- ciency may concern due to its large model size, observations indicate that, with the utilization of large GPU memory, the training time of AnoLLM- 135M can be comparable other deep learning- based methods, as discussed in Section F of the Appendix.
0.75
Bornst
Plad
KIW
ECOO
Despsipe
St AD
G04D
Heuttlls
DTE
REPEN
(SOO)
Figure 2: Averaged AUC-ROC scores with standard error bars for all methods over 30 datasets in ODDS. Colour scheme: yellow (shallow methods), purple (deep learning methods), dark blue (ours).
3.2 RESULTS THE ODDS BENCHMARK
Figure presents preseetes the overall performance of these different methods on 30 tasks in the ODDS library. The results for each dataset and standard error are provided in T0 Table Appendix. A critical difference diagram showing statistical significance is included Figure in the Appendix.

---

Published as a conference paper at ICLR 2025
Methods
AUC-ROC
LLM sizes
Mix-typed
ODDS
Equal-width
0.865
1355
0.803
0.884
Quantile
0.752
360M
0.811
0.865
Language
0.863
1.7B
0.812
0.861
No binning 0.800
(a) Comparison of different LLM sizes actoss the mixed-type tabular benchmarks and the ODDS benchmark, with the numbers representing AUC- ROC scores.
Standard 0.884
(b) Comparison on different binning methods numerical columns on ODDS library.
Table 3: Comparison of different LLM sizes and binning methods.
We note that in ODDS benchmarks, over 98.5% of columns are numerical, 0% of columns are tex- tual, and only 10 out of 30 datasets contain human-undee uamaan-underttadaalle column names, which diminishes the strengths of LLMs. However, by using pseudo feature names and feature binning, AnoLLM with SmolLM- 35M backbone performs comparably to the competitive baselines, KNN, ICL, and DTE. These results highlight AnoLLM's ability effectively handle numerical data and its robustness to missing column names. These findings are consistent with previous studies on tabular anomaly detection which indicates that KNN, ICL, and DTE yield top performance (Livernoche al. 2024
3.3 EFFECTS OF FEATURE BINNING
In this experiment, we systematically study different options for feature binning strategies for numer- ical columns. Specifically, we choose to use the 30 datasets in ODDS library since they are primarily comprised of numerical columns. All methods use the same pretrained LLM, smo 1 LL-MMMM 35M. We compare the standard rescaling method as in Eqn with four alternative methods listed below.
Equal-width binning:
The range of each column is divided into 10 intervals, each with the same width. The resulting bins are labeled with single-digit decimal numbers, ranging from 0.1 to 1.0.
Quantile binning: All numerical columns divided into 10 bins with each bin contain approxi- mately 10% of samples. The bins are labeled by percentiles, from "the 10th percentile" to "the 100th percentile".
Language: This method follows the same approach as equal-width binning, but the bin labels are re- placed with 10 adjectives in natural language, representing magnitudes: "minimal", "slight", "mod- erate", "noticeable", "considerable", "significant", "substantial", "major", "extensive". and "maxi- mum".
No binning: Numerical features are used in their raw and unaltered form.
Tablo[3 the averaged performance of each method over 30 datasets in the ODDS library. The results show that the standard rescaling method delivers the best performance, followed by equal- width binning and its language-based variant. contrast, both no binning and quantile binning show comparatively lower performance, highlighting the importance of linear rescaling and rounding. This evident linear rescaling methods, such as equal-width binning and standard rescaling, outperform approaches that rely raw numerical values, which may include long digit numbers that negatively impact overall performance. Moreover, replacing bin labels with adjectives has minimal effect on overall performance. Detailed comparison of each dataset can be seen in Table 16 in Appendix.
3.4 IMPACT OF LLM SIZES
In this experiment, discuss the impact of L.LM sizes the A framework. We compare LLMs in the Smoll LM family as they share the same pre-training corpus and tokenizer, with the only difference being the number of model parameters Due to GPU memory constraints, we use LoRA adapter (Hu et al. 2022 for the 1.7B model instead of full fine-tuning.
WWe note that Smol 1 L.M- 78 uses 1T tokens and 135M and 360M models only use 600M tokens for pre-training.
8

---

Published as a conference paper at ICLR 2025
Table 3a shows the averaged performance for all LLM sizes the mixed-type benchmark and the ODDS library. Detailed performance on each individual dataset is provided in Table [7 in the Appendix. The results suggest that using the 1.7B model does not provide much performance boost. This could be because larger models are trained on text data that are not relevant to our tabular tasks. As result, the increased model capacity and contextual knowledge do not lead to any improvements. Therefore, we recommend that practitioners use smaller models, which offer a better balance of efficiency and effectiveness.
More ablation studies on the effects of random permutations and pretrained weights are provided in Appendix D
4 RELATED WORKS
Unsupervised Anomaly Detection for Tabular data: Tabular anomaly detection is a long- standing problems and number of approaches, including classic methods and deep-learning based methods, are proposed over past several decades. One line of classical methods compute anomaly scores based on density, e.g. kernel density estimators (Latecki et al. 2007). local density (Bre-) unig et al. 2000), and gaussian mixture models (Yang et al.) 2009). Othe methods include isola- catpriaaacuuuaavvv--iisrbbbbuuuu
outlier detector (ECOD) (Li et al. 2022), One-class support vector machine (Schölkopf et al. 1999), and k-nearest neighbors (KNN) (Ramaswamy et al. 2000). Notably, KNN remains a strong baseline even when numerous deep learning based approaches have been proposed (Livernoche et al. 2024).
tion forest (Luu et al., 2008),
Deep-learning-base based approaches can be categorized into two groups, margin-based approaches and self-supervised learning (SSL) based approaches. Margin- -raged approaches, e.g. DeepSVDD (Ruff) ct al. 2018) and DROCC (Goyal et al. 2020). employ neural networks to map normal data into hy- perspace with minimal volume. SSL-based approaches typically define pseudo-tasks where normal data is expected to perform better than abnormal data. As result, these methods often incorporate SSL loss functions into their objectives. For example, NeuTral (Qiu al. [02T] and REPEN (Pang iet al., 2018) employ contrastive loss to bring the embeddings of similar data closer, while distancing the embeddings of dissimilar data. SLLAD (Xu et al., 2023b) and ICI. (Shenkar & Wolf 2022) both construct pseudo tasks by splitting the input vectors. Please refer to Section Appendix for more
details. We note that all the above methods are not able to handle raw texts and focus on numerical features only. Their applications to categorical data are not thoroughly discussed (Taha & Hadi, 2019
LLMs for Anomaly Detection: With the rising popularity of LLMs, recent studies have ex- plored their application in various anomaly detection (AD) tasks. In time-series AD, Liu et al. (2024a) demonstrate that LLMs can deliver accurate and interpretable results through well-designed prompts. AAlnegheimish et al. (2024) investigate the zero-shot performance of LLMs in time-series AD. For image data, several studies (Gu et al.] 2024 Cao et al. 2023 Zhu et al. 2024: Yang et al.] 2024cb examine the performance of visual-linguistic models in visual and video AD across differ- ent settings. Additionally, ] et al.] (2024) and Liu et al. (2024b) show that large visual-linguistic models can effecttivey detect fake news using multi-modal inputs. For system log data, several stud-
ies (Han et al. 20233 Yamanaka et al. 2024 Lee et al. 2023b Hadadi et al. 2024) have fine-tuned pretrained LLMs detect anomalies in system logs.
For tabular data, a concurrent study (Li et al. 2024) examines the zero-shot performance of LLMs. Their approach differs from ours in two ways. First, they focus on zero-shot performance and report weaker results on the ODDS benchmark, whereas we fine-tune LLMs on tabular data, achieving better performance. Second, their method prompts L.LMs to detect anomalies for each dimension separately and then aggregates the scores. Their performance lags behind ECOD, which also com- putes per-feature anomaly scores but does so by computing empirical cumulative distribution. In contrast, our methods are able to leverage contextual dependencies between features and achieve better results than ECOD. Additionally, Biester et al. (2024) and Park (2024) utilize LLMs as agents to generate domain-specific contexts to format data, such as for financial data processing and data cleaning. Their methods require additional modules and, in some instances, need human interven- tion process the outputs produced by the L.MMs.

---

Published as a conference paper at ICLR 2025
LLMs for Tabular Data: Recent advances in LLMs have spurred extensive exploration of their use in various tabular data tasks, including prediction (Dinh et al. 2022 Hegselmann et al. 2023] Manikandan et al. 2023), synthesis (Borisov al., 2023 Zhang et al. 2023), feature engineer- ing (Han et al., 2024), and table understanding (Sui et al. 2024). Additionally, recent studies have investigated pre-training foundation models specifically for tabular data (Zhu et al. [0233] Yan al.,
2024| Ye et al. 20224; Yang et al. 2024bba) a). For a comprehensive overview, we refer readers to
survey paper (Fang et 2024).
CONCLUSION AND FUTURE WORK
We introduce AnoLLM, a novel framework for adapting pretrained LLMs to unsupervised anomaly detection for tabular data. AnoLLM is robust tabular anomaly detection method that in- get raw textual features without pre-processing- Empirical evaluations demonstrate that AnoLLM achieves state-of-the-arr performance across six benchmark datasets containing mixed feature types, and matches the performance of the leading methods, KNN, ICL., and DTE, on 30 datasets predom- inantly containing numerical attributes.
One future direction for this work could be reducing the computational overhead associated with AnoLLM, given that LLMs much less efficient compared traditional anomaly detection meth- ods. Furthermore, considering AnoLLM's ' superior capability in modeling categorical and textual features, an interesting direction for future work could involve utilizing LLMs extract salient representations from such features, which could then be leveraged by classical anomaly detection methods to improve efficiency. Furthermore, since LLMs are known to exhibit limitations in numer- ical reasoning, it is of interest enhance AnoLLM's performance numerical data. One potential improvement could involve the development of specialized tokenization strategies and encoding tailored to numerical attributes.
Finally, as building foundation models for tabular data is an ongoing trend (Van Breugel & Van Der Schaar 2024 Gardner et al. 2024), AnoLLM represents an initial step in this direction by demonstrating that LLMs can be effectively adapted for tabular anomaly detection. While numer- other tabular tasks remain to be explored, an important avenue for future work is the creation of a ceneral-purposs tabular foundation model capable addressing a wide range of tabular data challenges.
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm blazingly fast and remarkably powerful, 2024.
Sarah Alnegheimish, Linh Nguyen, Laure Berti- Berti-Equille, and Kalyan Veeramachaneni. Large language models be zero-shot anomaly detectors for time series? arXiv preprint 1rX1:5240S.14755. 2024.
Liron Bergman and Yedid Hoshen. Classification-bass anomaly detection for general data. In International conference on learning representations, 2020.
Fabian Biester, Mohamed Abdelaal, and Daniel Del Gaudio. LImelean: Context-aware tabular data cleaning via -lmegenertted ofds. arXiv preprint wxXiv2240.11868/ 2024.
Vadim Borisov, Tobias Leemann, Kathrin SeBler, Johannes Haug. Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems, 2022.
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. In International conference learning representa- fions, 2023.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93-104, 2000.
10

---

Published as a conference paper at ICLR 2025
Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen. Towards generic anomaly detection and understanding: Large-scale visual- Ssual-linguistie model (gpt-4v) takes the lead. arXiv preprint arXiv:2311.02782. 2023.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM com- puling surveys (CSUR), 41(3):1-58, 2009.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finetuned lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: -anguage-interfaceed fine-tuning for non- language machine learning tasks. Advances in Neural Information Processing Systems, 17631 11784, 2022.
Vaishnavi Nath Dornadula and Sa Geetha. Credit card fraud detection using machine learning algo- rithms. Procedia computer science, 165:631-641 2019.
Tlamelo Emmanuel, Thabiso Maupong, Dimane Mpoeleng, Thabo Semong, Banyatsang Mphago, and Oteng Tabona. A survey on missing data in machine learning. Journal of Big data, 8:1-37, 2021.
Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Jane Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, Christos Faloutsos, et al. Large language models (Ims) tabular data: Prediction, generation, and understanding-a survey. Transactions Machine Learning Research(TMLBB) 2024.
Tharindu Fernando, Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Deep learning for medical anomaly detection a survey. ACM Computing Surveys (CSUR), 54(7): 1-37, 2021.
2021.
Joshua P Gardner, Juan Carlos Perdomo, and Ludwig Schmidt. Large scale transfer learning for tab- ular data via language modeling. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024. U http://pwneevveeeeeeeeeterrrrrrr
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. Droce: Deep robust one-class classification. In International conference on machine learming, pp. 3711- 3721. PMLR, 2020.
Prince Grover, Julia Xu, Justin Tittelfitz, Anqi Cheng. Zheng Li, Jakub Zablocki, Jianbo Liu, and Hao Zhou. Fraud dataset benchmark and applications. arXiv preprint arXiv:2208. /4417, 2022.
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anoma- lygpt: Detecting industrial anomalies using large vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1932-1940, 2024.
Fatemeh Hadadi, Qinghua Xu, Domenico Bianculli, and Lionel Briand. Anomaly detection on unstable logs with gpt models. arXiv preprint (.XXi:240600746.. 2024.
Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Advances in Neural Information Processing Systems, 35:32142-32159, 2022.
Sungwon Han, Jinsung Yoon, Sercan Ö. Arik, and Tomas Pfister. Large language models can automatically engineer features for few-shot tabular learning. In ICML, 2024. URL https:
//openreview.o http::/.ww//ffuummooeeemm
ERG45xx11
Xiao Han, Shuhan Yuan, and Mohamed Trabelsi. Loggpt: Log anomaly detection via gpt. In 2023 IEEE International Conference Big Data (BigData), pp. 1117-1122. IEEE, 2023.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-thant than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2025
Douglas M Hawkins. Identification of outliers, volume 11. Springer, 1980.
Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R Lyu. Drain: An online log parsing approach with fixed depth tree. In 2017 IEEE international conference on web services (ICWS), pp. 33-40. IEEE, 2017.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 1549-5581 PMLR, 2023.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International confer- ence on learning representations, 2022.
Ruihan Jin, Ruibo Fu, Zhengqi Wen, Shuai Zhang, Yukun Liu, and Jianhua Tao. Fake news detection and manipulation reasoning via large vision- languaee models. arXiv preprint arXiv:240..02042, 2024.
Max Landauer, Sebastian Onder, Florian Skopik, and Markus Wurzenberger. Deep learning for anomaly detection in log data: A survey. Machine Learning with Applications, 12: 12:100470, 2023.
Longin Jan Latecki, Alcksandar Lazarevic, and Dragoljub Pokrajac. Outlier detection with kernel density functions. In International workshop on machine learning and data mining in pattern recognition, pp. 61-75. Springer, 2007.
Nayoung Lec, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic small transformers. arXiv preprint arXiv:2307.03381, 2023,
Yukyung Lee, Jina Kim, and Pilsung Kang. Lanobert: System log anomaly detection based on bert masked language model. Applied Soft Computing, 146:1 146:1, 2023b.
Aodong Li, Yunhan Zhao, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt. Anomaly detection of tabular data using llms. arXiv preprint arXiv:2406. 16308, 2024.
Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, and George H Chen. Ecod: Unsu- pervised outlier detection using empirical cumulative distribution functions. IEEE Transactions Knowledge and Data Engineering, 12181111118112299 2022.
Boyang Liu, Ding Wang, Kaixiang Lin, Pang-Ning -ing Tan, and Jiayu Zhou. Rea: A deep collaborative autoencoder approach for anomaly detection. In IJCAI: proceedings of the conference, volume 2021, pp. 1505. NIH Public Access, 2021.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pp. 413-422. IEEE, 2008
Jun Liu, Chaoyun Zhang, Jiaxu Qian, Minghua Ma, Si Qin, Chetan Bansal, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Large language models deliver accurate and interpretable time series anomaly detection. arXiv preprint arXiv:2405. 15370, 2024a.
Xuannan Liu, Pei Pei Li, Huaibo Huang, Zekun Li, Xing Cui, Mahaooliang lixiong Qin, Weihong Deng, and Zhaofeng He. FKA-owl: Advancing multimodal fake news detection through kmowledge-augmentdd LVLMs. In ACM Multimedia 2024, 2024b. URL https:
//openzeview. wttp::wwww.ee..ee.eeeeeoee d- 944
Victor Livernoche, Vineet Jain, Yashar Hezaveh, and Siamak Ravanbakhsh. On diffusion modeling for anomaly detection. In International conference on learning representations, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International confer- ence on learning representations, 2019.
Hariharan Manikandan, Yiding Jiang, and J Zico Kolter. Language models weak learners. Ad- vances in Neural Information Processing Systems, 36:50907-50931, 2023.
Tomas Mikolov, Kai Chen, G.s Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. Proceedings of Workshop at ICIR, 2013, 01 2013.
12

---

Published as a conference paper at ICLR 2025
Guansong Pang, Longbing Cao, Ling Chen, and Huan Liu. Learning representations of ultrahigh- dimensional data for random distance- based outlier detection. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 2041-2050, 2018.
Taejin Park. Enhancing anomaly detection in financial markets with an Ilm-based multi-agent frame- work. arXiv preprint arXiv:2403. 19735, 2024.
Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. Neural transformation learning for deep anomaly detection beyond images. In International conference on machine learning, pp. 8703-8714. PMLR, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language under- standing by generative pre-training. 2018.
Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 427-438. 2000.
Shebuti Rayana. Odds library. http:::///dddss..sss.....a..eee cs.
2016.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 4393-4402. PMLR, 2018.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe- Taylor, and John Platt. Support vector method for novelty detection. Advances in neural information processing systems, 12, 1999.
Bernhard Schölkopf, John C Platt, John Shawe- Taylor, Alex J Smola, and Robert Williamson. Estimating the support of high-dimensional distribution. Neural computation, 13(7):14331141, 2001.
Ruogi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. Positional description matters for transformers arithmetic, arXiv preprint arXiv:2311.14757 2023.
Tom Shenkar and Lior Wolf. Anomaly detection for tabular data with internal contrastive learning- In International conference learning representations, 2022.
Mei-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakom, and LiWu Chang. A novel anomaly detection scheme based on principal component classifier. In Proceedings of the IEEE foundations and new directions of data mining workshop, pp. 172-179. IEEE Press Piscataway, NJ, USA. 2003.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arith- metic reasoning in language models using causal mediation analysis. Empirical Methods Nat- ural Language Processing, 2023.
Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. Table meets Ilm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp- 645-654, 2024.
Ayman Taha and Ali S Hadi. Anomaly detection methods for categorical data: A review. ACM Computing Surveys (CSUR), 52(2):1-35, 2019.
Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, and Bich-Liên Doan. Beyond individual input for deep anomaly detection tabular data. In Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 48097- 48123. PMLR, 21-27 Jul 2024.
13

---

Published as a conference paper at ICLR 2025
Boris Van Breugel and Mihaela Van Der Schaar. Position: Why tabular foundation models should be a research priority. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 48976-48993. PMLR, 21-27 Jul 2024. URL ptttps://proceedingg ml p.ess// wwwaaanbreeeeeaaa..a... html
Hu Wang. Guansong Pang. Chunhua Shen, and Congbo Ma. Unsupervised representation learning by predicting random distances. In Proceedings of the Twenty-Ninth International Joint Confer- ence on Artificial Intelligence, IJCAT'20, 20, 2021. ISBN 7780999241165.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. Transactions on Machine Learning Research (TMLR), 2022.
Hongzuo Xu, Guansong Pang, Yijie Wang, and Yongjun Wang. Deep isolation forest for anomaly detection. IEEE Transactions on Knowledge and Data Engineering, 35(12): 2023a.
Hongzuo Xu, Yijie Wang, Juhui Wei, Songlei Jian, Yizhou Li, and Ning Liu. Fascinating supervisory signals and where to find them: Deep anomaly detection with scale learning. In International Conference on Machine Learning, pp. 38655-38673, PMLR, 2023b.
Yuuki Yamanaka, Tomokatsu Takahashi, Takuya Minami, and Yoshiaki Nakajima. Logelectra: Self- supervised anomaly detection for unstructured logs. arXiv preprint (XXix:24022.1397 2024.
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. In International confer- ence on learning representations, 2024.
Xingwei Yang, Longin Jan Latecki, and Dragoljub Pokrajac. Outlier detection with globally optimal exemplar-based gmm. In Proceedings of the 2009 SIAM international conference on data mining, pp. 145-154. SIAM, 2009.
Yazheng Yang. Yuqi Wang, Guang Liu, Ledell Wu, and Qi Liu. Unitabe: universal pretraining protocol for tabular foundation model in data science. In The Twelfth International Conference on Learning Representations, 2024a.
Yazheng Yang, Yugi Wang, Sankalok Sen, Lei Li, and Qi Liu. Unleashing the potential of large language models for predictive tabular tasks in data science. arXiv preprint arXiv:2403.20208 2024b.
Yuchen Yang. Kwonjoon Lee, Behzad Dariush, Yinzhi Cao, and Shao- Yuan Lo. Follow the rules: Reasoning for video anomaly detection with large language models. arXiv preprint arXiv:2407.10290 2024c.
Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Towards cross- table masked pretraining for web data mining. In Proceedings of the ACM Web Conference 2024, pp. 4449-4459, 2024.
Jiaxin Yin, Yuanyuan Qiao, Zitang Zhou, Xiangchao Wang, and Jie Yang. Mem: Masked cell modeling for anomaly detection in tabular data. In The Twelfth International Conference on Learning Representations, 2024.
Tianping Zhang, Shaowen Wang, Shuicheng Yan, Li Jian, and Qian Liu. Generative table pre- training empowers models for tabular prediction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pp. 14836-14854, Singapore, December 2023. Association for Computational Linguistics. URL htt.hhtps::/aac wattt//aaaaaaacccaaaoaaa/ccoooaa ogy..g...o...................o.. org. 2023 emn 1ammnlpmman po-main.... 917
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detec- tion. Journal of Machine Learning Research, 20(96):1-7, 2019. URL http:/ /jmrr. orr/
papers//200/99001 .html
14

---

Published as a conference paper at ICLR 2025
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: cross-table pretraining for tabular transformers. In Proceedings of the 40th International Confer- ence on Machine Learning, ICML'23. JMLR.org, 2023.
Jiaqi Zhu, Shaofeng Cai, Fang Deng, Beng Chin Ooi, and Junran Wu. Do Ilms understand visual anomalies? uncovering Ilm's capabilities in zero-shot anomaly detection. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, pp. 48-57, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400706868. URL https://dot org/10. 114111111116366444444 000./4677.6661118 3681190
15

---

Published as a conference paper at ICLR 2025
A OVERVIEW OF THE APPENDIX
In Section we present detailed information about the datasets used in experiments. Section C analyzes AnoLLM's sensitivity to the number of permutations r. Section includes more ablation studies on the effects of random pertmuations and pretrained weights. The proof of Theorem [ is provided in Section E Section [F] discusses the computational efficiency our approach. com- prehensive review of related work tabular anomaly detection is included in Section G Finally, Section H details the hyperparameters of A and provides full experimental results, including additional performance metries.
B DETAILED DATASET INFORMATION
In this section, we provide detailed dataset information that we used in our experiments. The datasets statistics of the seven main datasets are Table
Fake job posts (Grover et al., 2022): This Kaggle dataset comprises 18,000 job descriptions, approximately 800 of which are traudulent. The dataset has 5 textual features that related to job postings, which are title, company profile, job description, requirements, and benefits. The objective develop classification model capable of identifying fraudulent job postings. We download the dataset using the provided API
Fraud ecommerce (Grover et al., 2022): This dataset comprises 150,000 e-commerce transac- tions, featuring data such as sign-up time, purchase time, purchase value, device IDD browser, and IP address. We use the provided table to convert IP addresses into the countries and discard the device feature. We also follow (Grover et al., 2022) to use the time difference between sign-up and purchase. We download the dataset using the provided API.
Lymphography (Rayana 2016): We collect this dataset from the ODDS library (Rayana 2016), but it was originally obtained from the University Medical Centre, Institute of Oncology. Ljubljana. The dataset features 18 attributes with the primary objective of predicting whether a lymphography result is normal. The objective is to predict whether high-energy seismic bump will occur in the next shift.
Seismic (Rayana 2016): The dataset addresses the issue of forecasting seismic bumps with high energy levels (greater than 10a J) in coal mine. The data were collected from two longwalls in Polish coal mine.
Vehicle insurance The dataset comprises 32 metadata of claimants and their insurance details, including 6 ordinal and 25 categorical. It contains 15,420 records, with only 6% (923 records) identified as fraudulent. The objective is to predict whether a claim is fraudulent. The dataset is publically available and is provided by Angoss Knowledge Seeker. We collect the dataset from Kaggle 8
20 newsgroups:
The dataset comprises news articles from 20 distinct newsgroups Following the configuration outlined in ADBench (Han et al. 2022), we use six top-level categories: com- puter, recreation, science, miscellaneous, politics, and religion. We conduct six experiments, each time designating one the six classes as the normal class and treating the remaining classes as anomalies. The anomalous classes downsampled to represent 5% of the total instances. The average performance the six experiments reported.
Outlier Detection DataSets (ODDS) (Rayana 2016): The ODDS provides datasets from differ- domains, e.g. healthcare, image, finance, botany, etc, each with varying numbers of features and
"https//www.kkge"""" wwt.a//wwww......ss/coccccoooonn catprrrraannnfaaadddddttttttttoi http:://cdaieoa.ccccctaaaataaaas wtw.e//eeeeaaaaac//oo//ooooo///d
16

---

Published as a conference paper at ICLR 2025
samples. The statisties provided in Table 4 We note that ODDS is mainly comprised of numer- ical columns as over 98.5% of columns are numerical. Also, for datasets lacking explicit column names, we use pseudo column names, c.g. "A", "B" z", AA""
Dataset
points
text
num category
Has column names
outliers (%)
Annthyroid
7,200
No
534 (7.42%)
Arthythmia
452
0
274
No
66 (15%)
BreastW
683
0
9
Yes
239 (35%)
Cardio
1,831
0
21
Yes
176 (9.6%)
Ecoli
336
0
7
Yes
9 (2.6%)
ForestCover
286,048
0
10
Yes
2,747 (0.9%)
Glass
214
9
No
9 (4.2%)
Heart
224
44
No
10 (4.4%)
Http (KDDCUP99)
567,479
3
No
2,211 (0.4%)
lonosphere
351
33
No
126 (36%)
Letter Recognition
1,600
32
No
100 (6.25%)
Lymphography
148
3
Yes
(4.1%%
Mammography
11,183
6
No
260 (2.32%)
Muleross
262,144
4
No
26,214 (10%)
Musk
3,062
166
No
97 (3.2%)
Optdigits
5,216
64
No
150 (3%)
Pendigits
6,870
16
No
156 (2.27%)
Pima
768
8
No
268 (35%)
Satellite
6,435
36
No
2,036 (32%)
Satimage-2
5,803
36
No
71 (1.2%)
Seismic
2,584
Yes
170 (6.5%)
Shuttle
49,097
9
No
3,511 (7%)
Smtp (KDDCUP99)
95,156
3
No
30 (0.03%)
Speech
3,686
0
400
No 61 (1.65%)
Thyroid
3,772
6
No
93 (2.5%)
Vertebral
240
6
Yes
30 (12.5%)
Vowels
1,456
No
50 (3.4%)
WBC
278
0
30
No
21 (5.6%)
Wine
129
13
Yes
10 (7.7%)
Yeast
1,364
0
8
Yes
64 (4.7%)
Table 4: Summary of datasets in Outlier Detection DataSets (ODDS) |Raaan 2016).
17

---

Published as a conference paper at ICLR 2025
C SENSITIVITY ANALYSIS OF THE NUMBER OF PERMUTATIONS
In this section, we explore the sensitivity of AnoLLMMs performance to the number permutations r during inference. Figure EequreBilluste iilustrates how performance changes with varying values of r. The blue shaded area indicates the standard deviation, calculated from bootstrapping with 100 trials for each The AnoLLM uses SmolLM- 135M the LLM backbone.
As shown in the figure, the average performance plateaus after = 10 and it has roughly 0.8% gain compared
Additionally, the standard deviation decreases increases. The results indicate that utilizing multiple permutations during inference enhances performance and reduces variance as more anomaly scores are ensembled from different permutation functions. Practitioners can select the number of permutations r by considering the tradeoff between effectiveness and ef- ficiency, as illustrated in this figure. We choose 21 our experiments as provides the best performance.
0.882
0.880
Averaged AUC-ROC C-CC
0.878
0.876
0.874
0.872
of permutations
Figure 3: The average change of AnoLLM in AUC-ROC with respect to the number of permutations on 30 datasets from the ODDS library. The blue shaded area represents the standard deviation. The AnoLLM uses SmoILM-135M as the LLM backbone.
18

---

Published as a conference paper at ICLR 2025
D MORE ON ABLATION STUDIES
In this section, we study the effect of random permutation and the effect of pretrained weights of LLMs.
D.1 EFFECT OF RANDOM PERMUTATION
To study the effect of random permutations to the final performance, we train another SmolLM- 13SM on the five datasets containing mixed-typed features I using the same hyperparameters. We maintain the original column order from the dataset during both the training and inference stages. As demonstrated in Table 5 removing random permutations leads to a significant decline in AnoLLM's performance. This finding underscores the importance of random permutations in reducing the model's ' sensitivity to column ordering.
Fake job posts
Fraud ecommerce
Lympho- graphy
Vehicle insurance
Methods
Datasets
Seismic
Average
0.800 Smoll M-oll.M-1550 M-135M (w.o. permutation)
SmolLM- 135M (w permutation)
0.968 0.656
0.712 0.438
0.569 0.577
0.809
0.565
0.995
0.646
Table 5: AUC-ROC scores for AnoLLMs with and without random permutation on five datasets containing mixed typed features.
D.2 EFFECT OF PRETRAINED WEIGHTS
To study the performance of AnoLLMs without LLM pretrained weights, we evaluate the perfor- mance of randomly initialized transformers on the ODDS benchmark. To ensure fair comparison, we use same model architecture, SmolI.M-135M, with identical hyperparameters. The results are summarized in Table
As shown, AnoLLM with pretrained weights slightly outperforms its randomly initialized counter- part in terms of overall average performance. It achieves better performance on 16 out of 30 datasets and matches performance on datasets. Additionally, a visual inspection of the training curves re- veals that AnoLLM with pretrained weights converges approximately twice as fast most datasets. This faster convergence can be attributed to the pretrained LLM providing a better initialization for fine-tuning. In contrast, AnoLLM without pretrained weights not only converges also more susceptible to overfitting, as evidenced by its significantly lower training loss. While overfitting may not be major concern in uncontaminated, unsupervised settings, it could present challenges in contaminated scenarios, where the model risks memorizing anomalous samples.
slowly but
Given that the performance gap between variants is relatively modest, the randomly initialized AnoLLM can serve as viable alternative for datasets primarily composed of numerical attributes in uncontaminated settings, particularly when smaller pretrained models are unavailable. An inter- esting direction for future work would be exploring the trade-off between efficiency and accuracy.
'e exclude the 20 newsgroups dataset hecause it includes only single text column, making difference for random column permutation.
19

---

Published as a conference paper at ICLR 2025
Pretrained L.M weights
Randomly initialized weights
Annthyroid
0.927
0.93
Arrhythmia
0.825
0.827
BreastW
0.992
0.993
Cardio
0.94
0.935
Ecoli
0.777
0.778
ForestCover
0.881
0.853
Glass
0.819
0.816
Heart
0.82
0.825
Htp (KDDCUP99)
1
lonosphere
0.909
0.89
Letter Recognition
0.967
0.907
Lymphography
0.968
0.997
Mammography
0.915
0.878
Mulcross
Musk
1 0,983
Optdigits
0.897
Pendigits
0,971
0.988
Pima
0.663
0.649
Satellite
0.902
0.86
Satimage-2
1
0.998
Seismic
0.712
0.737
Shuttle
1
1
Smtp (KDDCUP99)
0.927
0.926
Speech
0.47
0.459
Thyroid
0,975
0.984
Vertebral
0.565
0.415
Vowels
0,982
0.895
WBC
0,964
0,953
Wine
0,909
0.884
Yeast
0.744
0.749
Average
0.884
0.867
Table 6: Comparison of AnoLLMs with randomly initialized weights and pretrained LLM weights (SmolL 1-M35M)) on the ODDS benchmark.
20

---

Published as a conference paper at ICLR 2025
E PROOF OF THEOREM
Proof.
For all W1, w2..., Wks we have
po- (WI, .... Wk 80)
pe- (wkl so. wi.... wk-1)
pa- (W1, wk- 80)
E, BP [ (the first l: tokens of 8 are wi,..., Wkk ) |80]
E gP
[1 [the first 1 tokens of g(() are Wi,..., wx-1) 30
where the second equation since the optimal language model po- aligns with the true distribution of (()) being sampled. Then, for all i (1,2, .g). we have
4
II Pa- (w 1=1
(1)
18, w( w 12
BR (the first k words of s(() are w (6) ..... wx ((²))ss)) |so
II k=1 Excoap
[the first 1 words of s()) are wf" ...., w 12 1s00
(the first 4 words of (()) are w
(6) , ...., w/ (1)) }/50]
EP P(s((||s))
21

---

Published as a conference paper at ICLR 2025
F COMPUTE EFFICIENCY ANALYSIS
The total compute required to train A 135M across all datasets with five seeds, including six datasets from the mixed-type benchmark and 30 datasets from the ODDS benchmark, is approxi- mately 90 GPU hours on a single RTX-A6000 GPU with 48 GB of memory.
Figure 4 and Figure 5 illustrate the training and inference times for all methods, averaged five datasets (fake job posts, fraud ecommerce, lymphography, seismic. and vehicle insurance). As anticipated, deep cearning-based methods exhibit significantly higher training times and slightly higher inference time compared to classical methods. One exception is that KNN is much slower during inference since fraud ecommerce contains over 150, 000 training samples, making naive search computationally slow. Notably, AnoLLM-133MM achieves comparable training times due to the large GPU memory of the RTX-A6000, which enables efficient batch processing. However, considerably slower during inference due to its large model size. We note that we use r 21 different permutations throughout the paper, but the number can reduced by selecting a smaller The trade-off between number of permutations and performance is shown in Fig
10
Training time (s) log
10a
10a
10e
Iforedt
PCA
KNNN
ECOD
DeepSvngg
SLAD
GOAD
Neuttal
IC
DTE
REPEN
AnGLLM(133M)
Figure 4: Mean training time over 5 datasets, fake job posts, fraud ecommerce, lymphography. seismic, and vehicle insurance. Colour scheme: yellow (shallow methods), purple (deep learning methods), dark blue (ours).
Inference time (s) scale
10a
10a
10
Iforast
PCA
KNW
ECOD
DespSVRRA
SLAD
GOAD
NeuTral
KCL
DTE
REPEN
A3SLLM(135M)
Figure 5: Mean inference time over 5 datasets. fake job posts, fraud ecommerce, lymphography, seismic, and vehicle insurance. Colour scheme: yellow (shallow methods). purple (deep learning methods), dark blue (ours).
22

---

Published as a conference paper at ICLR 2025
G MORE RELATED WORK IN UNSUPERVISED ANOMALY DETECTION FOR TABULAR DATA
Tabular anomaly detection is a long-standing problems and number of approaches, including clas- sic methods and deep-learning based methods, are proposed over past several decades. One line of classical methods computes anomaly scores based on density, c.g. kernel density estimators (Late- cki et al. 2007), local density (Breunig et al., 2000), and gaussian mixture models (Yang ct al., 2009). Isolation forest (Liu al. (008) uses recursive partitioning with binary trees identify out- liers. cumpirical-cumulatin distribution-bass tail probabilities per dimension by computing empirical distributions and aggregates these proba- bilities to detect anomalies. One-class support vector machine (Schölkopf et al., 1999) finds the best boundary that separates data points into normal and anomalous classes. The k-nearest neigh- bors (KNN) (Ramaswamy et al. 2000) uses the distance to the A neighbors as its anomaly scores, which remains a strong baseline (Livernoche et al. 2024).
outlier detector (ECOD) (Li et al. (2022) estimates
Deep-learming-aasee based approaches can be categorized into two groups, margin-based approaches and self-supervised learning (SSL) based approaches. Margin- approaches, e.g. DeepSVDD (Ruff et al. 2018 and DROCC (Goyal et al., [020), employ neural networks map normal data into hyperspace with minimal volume. Deep isolation forest (Xu et al.] 2023a) introduces a new rep- resentation scheme learned from neural networks perform data partition. SSL-based approaches typically define pseudo-task where normal data is expected to perform better than abnormal data. As a result, these methods often incorporate SSL loss functions into their objectives. For exam- ple, NeuTTaal (Qiu et al. (02T) employs contrastive loss to bring the embeddings of augmented data closer to those of the original samples, while simultaneously pushing away the embeddings of augmented data derived from the same inputs. GOAD (Bergman & Hoshen 2020) predefines a set of fixed transformations and trains a classifier to differentiate between them. REPEN (Pang ee al. 2018) uses triplet loss to encourage normal samples to be closer than outlier samples that are generated by classical approaches. RCA (Liu et al. 202I) uses two autodens to filter training samples with high loss and uses reconstruction loss its anomaly scores. SLAD (Xu et al. 2023b) and ICI. (Shenkar & Wolf, 2022) both split the input vectors. SLAD learns a model to predict the number of dimensions given the split vectors and ICI. leams embedding model map vectors from the same input to nearby regions. MCM (Yin et al. (024) utilizes masked prediction loss with learnable masks detect anomalies. RDP (Wang al. 2022 trains neural networks predict the random distance of samples. DTE (Liverocche et al. 2024) estimates diffusion time and uses it as anomaly scores. Parammrtetei Transformers (Thmmonier et al., 2024) (NPEs) detect anomalies in tabular data by reconstructing masked features of normal samples. We note that all the above methods are not able to handle raw textual columns and focus on numerical features only. Their applications to categorical data are not thoroughly discussed (TTaha & Hadi 2019).
23

---

Published as a conference paper at ICLR 2025
H FULL RESULTS
In this section, we present the hyperparameters we used for AnoLLMs and the detailed perfor- mance of each method across all datasets. The hyperparameters AnoLLMs are shown in Table 7 For mixed-typed benchmark, critical difference diagram computed based on AUC-ROC scores shown Fig 6
Table X and Table 9 shows the Fl and area under the precision-recat curve (AUC-PR) scores for the Six datasets containing mixed types of features.
For the ODDS benchmark, the detailed AUC-ROC scores, their standard errors, and critical differ- ence diagrams for all methods are shown from Table 10 to Table TT and Fig 7 The F1 and AUC-PR scores and their standard errors for all methods are shown in Table 12 and Table 14 Table [6 and Table 17 present the detailed performance of experiments in comparing different binning methods and LLM sizes. These results complement those presented in the experiments.
SmoILM-1333
SmolLM-360M
SmoILM-1.7B
Datasets Hyperparameters
batch size
Use LoRA
batch size
Use LoRA
batch size
Use LoRA
Mix-type Benchmark
fake job post fraud Lymphography Seismic
3
Yes
12
Yes
4
Yes
32
No
32
No
16
Yes
No
Yes
No
Vehicle insurance 20 newsgroups
32
No
32
No
16
Yes
16
No
16
No
4
Yes
ODDS Benchmark
Annthyroid
16
No
16
No
4
Yes
Arrhythmia
2
Yes
2
Yes
1
Yes
BreastW
32
No
32
No
Yes
Cardio
No
Yes
Ecoli
32
No
No
Yes
ForestCover
48
No
48
No
12
Yes
Glass
32
No
32
No
8
Yes
Heart
32
No
32
No
8
Yes
Http (KDDCUP99)
128
No
128
No
32
Yes
lonosphere
32
No
32
No
Yes
Letter Recognition
32
No
Lymphography
No
Mammography
32
No
32
No
8
Yes
Mulcross
96
No
96
No
24
Yes
Musk
8
Yes
8
Yes
2
Yes
Optdigits
16
No
16
No
4
Yes
Pendigits
32
No
32
No
Yes
Pima
No
Satellite
No
Satimage-2
32
No
32
No
8
Yes
Scismic
16
No
16
No
8
Yes
Shuttle
32
No
32
No
8
Yes
Smtp (KDDCUP99)
128
No
128
No
32
Yes
Speech
2
Yes
Yes
Yes
Thyroid
32
No
Vertebral
No
Vowels
32
No
32
No
8
Yes
WBC
32
No
32
No
8
Yes
Wine
32
No
32
No
8
Yes
Yeast
32
No
32
No
Yes
Table 7: Hyperparameters used the experiments for AnoLLMs with different backbones. For 32 and apply to all modules
LoRA (Hu et al. 2022), we use the default parameters, r = 16 and
including all attention and MLP layers.
24

---

Published as a conference paper at ICLR 2025
Fake job posts
Fraud ecommerce
Lympho- graphy
Vehicle insurance
20news groups
Methods
Datasets
Scismic
Average
Classical methods
fooest
0.274
0.173
0.233
0.251
0.137
0.196
PCA
0.256
0.209
0.567
0.266
0.124
0.133
0.259
KNN
0.163
0.667
0,291
0.135
0.156
0.402
ECOD
0.165
0.408
0.400
0,282
0.112
0.132
0.250
Deep learning hased methods
DeepSVDD
0.136
0.567
0,258
0.115
0.152
0.371
RCA
0.137
0.667
0.32
0.135
0.129
0.398
SLAD
0.175
0.988
0.667
0.285
0.155
0.159
0,405
GOAD
0.129
0.92
0.667
0.295
0.119
0.136
0.378
NeuTral
0.115
0.633
0.195
0.12
0.195
0.376
ICL
0.245
0.667
0.298
0.108
0.19
0.418
DTE
0.107
0.667
0.239
0.121
0.185
0.387
REPEN
0.164
0.667
0,306
0.126
0.124
0.398
AnoLLM
SmolLM-135h 0.325
0.767
0.279
0.162
0.241
0,462
SmolLM-3660
0.343
0.992
0.8
0.336
0.174
0.22
0.478
Table 8: Detailed Fl scores for all methods the six datasets containing mixed types of features.
This table complements the results shown Section 3.T dataset splits.
The scores are averaged over random
Fake job posts
Fraud ecommerce
Lympho- graphy
Vehicle insurance
20news groups
Methods
Datasets
Seismic
Average
Classical methods
Iforest
0.227
0.172
0.232
0,235
0.112
0.146
0.187
PCA
0.194
0.238
0.624
0.216
0.117
0.143
0.255
KNN
0.138
0.720
0.256
0.123
0.148
0.398
ECOD
0.13
0.39
0.365
0.244
0.116
0.141
0.231
Deep learning based methods
DeepSVDD
0.12
0.680
0.226
0.114
0.135
0.379
RCA
0.134
0.783
0.25
0.124
0.136
0,405
SLAD
0.15
0.992
0.795
0.241
0.14
0.159
0,413
GOAD
0.117
0.979
0.697
0,239
0.116
0.144
0,382
NeuTral
0.108
0.681
0.193
0.117
0.176
0.379
ICI.
0.193
0.718
0.251
0.115
0.175
0.452
DTE
0.106
0.747
0.224
0.116
0.157
0.392
REPEN
0.146
0.697
0.249
0.118
0.389
AnoLLM
SmolL.M-1350 SmolL.M3360
0.286 0.304
0.999 0.972
0.856
0.236 0.281
0.141
0.223
0.457
0.938
0.143
0.214
0.475
Table 9: Detailed AUC-PR scores for all methods on the six datasets containing mixed types of features. This table complements the results shown in Section 3.1 random dataset splits.
The scores are averaged over
25

---

Published as a conference paper at ICLR 2025
Critical difference diagram average ranks
8
10
12
Mm36LMM330M (2.2)
(12) ECOD
SmoILM-135M (3.5)
[11] ffores
SLAD (6.2)
9933) PCA
KNN (6.3)
(9.2) DeepSVDD
RCA (6.5)
(8.9) NeuTral
ICL
(8) GOAD
REPEN
17.9) DTE
Figure 6: Critical difference diagram of average ranks for various methods. The ranks are computed over AUC-ROC scores. The x-axis shows the average ranks across datasets, with smaller values indicating better performance. Horizontal bars connect groups of methods that are not significantly different in performance according to statistical test. We note that AnoLLMs are not significantly better than SLAD in terms of ranking since the ranking score is only averaged over six datasets. However, it outperforms it by a large margin as can be shown in Table 2
Clusical Metoods
Deep learning based methods
AnoLLM
Datosets |Methods
Mfrrett
PCA
KNN
ECOD
DeeSVDD
SL.AD
GOAD
NeuTral
IC1.
DITE
REPEN
135M
Ammbyrood
0.922
0.839
0.811
0.79
0.743
0.718
0.76)
0.572
0.813
0.842
0.977
0.736
0.927
01931
Arthyaamia
0,827
0.796
0.786
0.811
0.768
0.785
0.781
0.681
0.75
0.785
0.771
0.644
0.825
0,822
BreadW
0.994
0.963
0.992
0.992
0.911
0.967
0.985
0.994
0.983
0.992
0,982
0.955
0.992
0.993
Cardio
0.948
0.966
0.921
01955
0.842
0.948
0,859
0.44
Booli
0.856
0.877
0.887
0.777
0. 0.804
FovestCover
0,915
0.985
0.921
0.533
0.977
0000
0,80
Glas
0.802
0.713
0.849
0.693
0.824
0.719
0.79
0.574
0,933
0.887
0.799
0755
0,11
0.797
Heaut
0,819
0.842
0.114
01639
0.773
0.785
0.822
0,838
0.111
0.787
0.831
1149
020
0.799
Hipp (KDDCUP)))
0.992
0.909
1
0.979
0.99
0.905
0.999
0.996
0.973
1
0.995
0.94
1
lonoophere
0.891
0.834
0.96
0.734
0.93
0.916
0:36
0.95
0.95
0.97
0.964
03545
0.909
0.924
Leuer Recogmition
0.631
0.529
0.865
0.567
0.776
0.716
0,908
0.811
0.929
0.959
0.872
0.597
0.967
0.667
Lymphoggagby
0.673
0.826
0.86
0.899
0.847
0,993
Manmoggrphhy
0.9
0.872
0,857
0.756
0.69
0.782
0.915
Mulcross
0.999
0.96
0,969
0.968
Musk
0.97]
0.956
0.723
Opidigits
0.834
0.574
0.94
0.606
0.751
0.8
0.735
0.847
0.979
0.955
0.888
0.607
0,983
0.930
Pendigits
0.971
0.942
0.999
0.928
0.838
0.967
0.932
0.22
0.963
0.971
0,993
0.922
0.971
0.964
Pina
0.72
0.711
0.741
01587
0.599
0.704
0.584
0.665
0.763
0.607
0.60
13688
0666
B.654
Satellite
0.807
0.662
0.874
0.582
0.8
0.734
0.864
0.795
0.859
0.854
0,760
0.74
0.902
0,877
Satimag-22
0.993
0.979
0.999
Seismic
0.692
0.692
0.738
0.713
0.719
0.724
0.712
0,746
0.99
0.934
0.999
0.997
0,995
0.99
0.997
0.999
0,995
0992
Soap (KDUCUPS))
0,901
0.809
0.936
0.88
0.78
0.845
0.926
0.911
0.89
0.885
0,953
11894
0,927
0,929
Spooch
0.478
0.471
0.486
0.471
0.561
0.472
0.511
0.55
0.609
0.582
0.511
0.53)
0.47
0.47
Thyroid
0,989
0.951
0.976
0.978
0.869
0.969
0.958
0.689
0.886
0.987
0.99
0,904
0.975
0.983
Vertebral
0.466
0.494
0.406
0.474
0.478
0.478
0,483
0,516
0.545
0.543
0.57
0.247
0.565
0.408
Vowels
0.779
0.644
0,895
0.988
0,983
0.982
WBC
0.949
0.004
Wine
0.927
0.952
0.720
0.854
0.964
0.961
0.96
0.944
0.974
0.909
Yeast
0.863
0.85
0.802
0.787
0.706
0.816
0,416
0.235
0.629
0.713
0.781
0.699
0.744
0.73
Average
0.847
0.826
0.879
0.79
0.818
0.848
0.841
0.745
0.855
0.877
0.879
0.766
0.884
01865
Table 10: Detailed AUC-ROC scores for all methods over 30 datasets in ODDS. This table comple- ments the results shown in Section 3.2 The scores are averaged over 5 random dataset splits. The highest number and second highest number in each row are highlighted in red and blue, respectively. A with SmolI.M MM-13MM 135M backbone outperforms all other methods in terms of averaged per- formance.
26

---

Published as a conference paper at ICLR 2025
Chasical Medhods
Deep-leurning based methods
AnoLLM
Datnsets |Metheds
Morest
PCA
KNN
ECOD
DesgSVDD
RCA
SLAD
GUAD
NeuTral
K.1.
DTE
REPEN
135M
300M
Anathgroid
0.003
0.013
0.002
0.001
0.01
0.001
0.005
0.009
0.000
0.002
6011
0.001
0.001
Arbychmia
0.002
0.031
0.004
0.004
0.035
0.008
0.004
BreastW
0.001
0.001
0.002
0.0112
0.002
0001
0.301
0.901
Cardio
0.004
0
0.004
0.012
0.001
0.003
0.018
0.013
0.015
0.005
0,02
0.044
0.004
Ecoli
0.000
0.004
0.001
0.004
0.021
0.003
0,002
0.003
0.004
0.004
0.005
0.15
000
0.014
Fover
0.006
0
0
0.025
0.001
0.027
0.017
0,018
0.004
0.002
0.007
0021
0.008
Glass
0,007
0,003
0.008
0.009
0,017
0.008
0,013
0,021
0,011
0,016
0,905
10019
0016
0.014
Heart
0.009
0,006
0.007
0.006
0.008
0.005
0,009
0.007
0.01
0.034
0.042
0.006
0.009
Hup (DDDCUP9)
0
0
0
0.005
Tonoophere
0.008
0.005
0.006
0.005
0.005
0.003
0,05
0.006
Letter Recegnition
0.003
0.001
0.033
8.006
0,003
0.003
0.006
0.001
0.044
0.001
0018
0.033
B.0
Lomphugraphy
0.037
0.007
0.015
0.013
0.01
0.008
0:01
0.014
0.015
0.018
0.006
0,05S
0.088
0.002
Manuzography
0.002
0.001
0.001
0.011
0.001
0.006
0.821
0L009
0.0044
0.006
0002
0.001
Mulcross
0
0
0
0
0,034
0,021
0.07
Musk
0.005
0
0
0
0
D4%
Opidigiis
0.013
0,002
0.001
0.002
0.048
0.004
0.04
0.018
0.002
0.007
0.013
0.052
0.001
0.005
Pendigits
0.003
0
0.03
0.002
Pima
0.005
0.004
0.003
0.007
0.008
0.012
0.017
0.005
0.099
0.004
0.007
0
Sateilite
0.007
0.001
0.001
0.001
0.008
0.001
0.001
0.004
0.002
0.002
0.003
0.004
Satimuge-2
0
0
0
0.004
0.009
Seismic
0.005
0.005
0.003
0.004
0.000
0.003
0.004
0.005
0,003
0.004
0.003
0.008
0.003
0.002
Shuttle
0
0
0
0
0
0,001
0.000
0
Sonp (EDICUPS)
0.003
0
0.002
0.001
0.012
0.012
0,002
0,017
0L002
13001
0.003
B4001
Speech
0.0K7
0.003
0.004
CLOK3
01009
B.016
0.012
0.005
0.04
0
0
0.025
0.005
0.001
Vertebral
0.019
0.031
0.199
0.003
0.032
0.023
0.166
0.038
0.024
Vowels
0.01]
0.005
0.002
0.003
0.008
0.004
0,00]
0.004
0.001
0.006
0.001
0.025
0.004
0.006
WBC
0.003
0.KK2
0001
01003
0.0044
0.002
0,005
0.067
0.002
0.063
0.005
0.017
0.00
0.094
Wine
0.008
0.008
0.08
0.007
0.013
0.006
0.036
0.02
0.007
0.007
0.001
0.058
0.047
0.016
Yoast
0.004
0.005
0.005
01003
0.011
0.084
0.046
0.03
0,019
0.013
0.021
0.019
0.019
0.01
Average
0.002
0.001
0.001
0,000
0.001
0.004
0.004
0,003
0.002
0,002
0
0.002
0
Table 11: Standard errors of AUC-ROC scores for all methods over 30 datasets in ODDS. This table
complements the results shown in Section 3.2 The standard errors are computed over 5 random dataset splits.
Classical Mithods
Despleamiing bised methods
A
Datasets |Methods
Hoorest
PCA
KNN
ECUD
DDD
RCA
SLAD
GUAD
Neutral
K.L
DTE
REPEN
135M
30UM
Antthyroid
0.574
0.487
0.44
0.388
01436
0.367
0,418
0.257
0,468
0.501
0.789
0.338
0.584
0.597
Anhyrhmia
0.612
0.542
0.554
0.591
0.531
0.536
0.515
0.531
0.422
BreasW
0.969
0.959
0.963
0.916
0.967
0.959
0.935
0.588
0.966
Candio
0.715
0.808
0.676
0.666
0.564
0.726
0.602
0.294
0.568
0.689
0.644
0.561
0.734
0734400000
0.665
Bcoll:
0.756
0.778
0.778
01311
0.533
0.778
0.778
0.778
0,511
0.111
0.66
0,756
0
0,533
FovesCCovrr
0.109
0.158
0.745
0.238
0.035
0.189
0.136
0.001
0.426
0.769
0.778
0.04
0256
0213
Giass
0,156
0.133
0.178
0.156
0..44
0,156
0.178
0,133
0.422
0.280
0.133
1369
1.178
Heart
0.917
0.922
0.906
0.893
0.906
0 0.908
0,913
0.91
0.904
0.91
0.914
0356
0912
0.907
Hup (DDCUP99)
0.107
0.926
0.994
0.459
0,929
0.193
0.993
0,349
0.304
0.989
Tonesphose
0.797
0.78)
0.804
0.895
0.894
0.854
0.882
0.821
Latter Recognition
0.176
0.136
0.434
0.346
0.004
0.29
0,548
0.404
0.636
0.722
0.588
0.164
0.734
0.4%6
Empphugmphy
0.233
0.567
06667
0.400
0.567
0.667
0.667
0.667
0.633
0,667
0.667
0.667
9-767
0.8
Manezography
0,413
0.674
0.409
0.535
0.443
0.358
0.138
0,267
0,135
0.29%
0,364
0.294
0,551
0,428
Mulcruss
0.995
0.747
0,999
0,76
0.852
0.996
0,816
Musk
0,616
1
1
01546
1
0.15
1
Optdight
0.159
0.001
0.784
0.007
0.297
0.021
0.04
0,165
0.639
0:471
0.164
0.02
0.72
0.443
Pendight
0.551
0.442
0.445
0.467
0.559
0.550
0.505
Pima
0.672
0.688
0.692
0.578
0.57
0.672
0.565
0.695
0.668
0626
0.62
Sateilite
0.695
0.614
0.762
0.538
0.71
0.685
B.76
0.693
0.751
0.757
0.725
D691
0.798
0,774
Satimage 2
0.873
0.843
0.935
0.718
0.854
0.939
0.825
0.955
0,051
0918
0,501
0,91
0952
0,944
Seismic
0.251
0.266
0.291
012822
0.25%
0.32
0.285
0.205
0.195
0.209
0.250
11306
0229
1.316
Shuttl
0.964
0.958
0.977
0.917
0.983
0,969
0,975
0.965
0.982
0.963
0,974
0.936
0.983
0,984
Snop (KDDCUPP9)
0
0.667
0.667
(1667
01002
0.683
0.667
0.667
0L6607
0.453
0.667
13607
0.667
0.667
Speech
0,03
0.049
0.056
0.019
0.036
0.029
0,039
0.039
0,049
0,075
0,052
0.066
0,062
Thymid
0.789
0.723
0.643
0.557
D.346
0.682
0.725
Vertebral
0.187
0.207
0.14
0.213
0.28
0.327
0.193
0.287
0.18
Vowels
0.348
0.2
0.684
0.22
0.548
0.424
6.716
0.46
0.768
0.74
0.78
0.244
0.76
0.5%6
WBC
0.819
0.79
0.714
0.562
0.705
0.724
0.667
0.381
0,181
0.733
0.619
0.219
0.79
0.762
Wine
0.7
0.62
0.7
01.88
0.42
0..6
0.72
0.76
0.7
0.62
0.72
01.4
0.5
0.52
Yeast
0.486
0.421
0.337
0.335
0.31
0,352
0.084
0.01
0.181
0.331
0.325
0.158
0.32
0.31
Average
0.519
0.572
0.648
0.460
0.535
0.562
0,566
0.508
0.535
0.651
0,612
0.430
0.647
0.61]
Table 12: Detailed F1 scores for all methods over 30 datasets in ODDS. This table complements the results shown in Section 3.2 The scores are averaged over random dataset splits. The highest number and second highest number in each row are highlighted in red and blue, respectively.
27

---

Published as a conference paper at ICLR 2025
Chasical Methods
Deep-leurning based methods
AnoLLM
Datnsets |Metheds
Morest
PCA
KNN
ECOD
DesgSVDD
RCA
SLAD
GOAD
NeuTral
K.1.
DTE
REPEN
135M
300M
Anathgroid
0.01
0.013
0.002
0.001
0,005
0.001
0.006
0.008
0.015
0.006
6011
0.005
0.002
Arbychmia
0.009
0.000
0.033
0.007
0.009
0.007
0.003
0.003
BreastW
0,002
0,008
0.022
0,002
0.044
0.811
0.014
0.003
0.04
0.013
0005
0.003
Cardio
0,016
0,002
0.011
0.005
0.015
0.008
0.008
0.006
0.028
0.017
0.009
0,021
0000
0.004
Ecoli
0.02
0
0
0.02
0.08
0.051
0.024
0/2
1800
0.051
Foves
0.003
0.001
0.022
0.001
0.01
0.003
0.033
0.027
0.024
0.007
000
0.008
Glass
0,024
0.02
0.024
0.024
0,003
0.024
0.024
0.02
0,015
0.074
0.02
0.34
0051
0
Heart
0.001
0,002
0.004
0.002
0,005
0.013
0.012
0.004
0.001
0.005
0.002
0,000
0.002
0.005
Hup (DDCCUP9)
0.083
0.001
0
0.32
0.001
0.0%
0,02
0.013
Tonoophere
0.009
0.007
0.009
0.008
0.002
0.003
0.06
Letter Recegnition
0.009
0.004
0.004
0.002
0.007
0.066
6.011
0.004
0.009
0.16
0.023
B018
0010
B.IIT
Lomphugraphy
0.03
0.056
0
0.056
0
0.03
0:03
0.036
0
0.03
0,101
0.06
0,03
Manuzography
0.012
0.IK5
0.004
01002
0,0027
0.004
0.006
0.001
0.007
CLOO8
0.011
1.009
0,01
1LA0G
Mulcross
0,001
0
0
0
0.028
0,077
0.35
0
Musk
0.055
0
0
0
0
DA12
Opidigiis
0.023
0.001
0.012
0,071
0.001
0.031
0.04
0.029
0.0418
0.021
0.012
0.006
0.024
Pendigits
0.02
0.005
0.004
0.002
0,063
0.024
0.024
0.045
0,066
Pima
0.005
0.004
0.007
0,005
0.006
0.004
0.011
0012
0.004
0.008
0.004
0:003
0.01
Satellite
0.003
0.002
0.001
0.008
0.012
0.001
BAKE
0.001
BANE
Satimage22
0.007
0.005
0.008
0.01
0,085
0.016
0,005
0.003
0.003
0.016
0.009
0.006
0.011
Seismic
0.013
0.004
0.01
0.003
0.008
0.008
0.011
0.006
0.007
0.007
0.01
0.008
0.008
0.003
Shuttle
0,005
0.001
0.001
0,001
0.001
0.001
0,001
0.001
0,001
0.0266
Sonp (EDICUPS)
0
0
0
0.056
0,007
0,015
0.034
0.34
Speech
0.005
0
0.004
9:01
0.016
0.006
0.004
0.01
0.048
0.008
0.013
0.007
0,113
Vertebral
0.022
0.024
0,022
0.007
0.041
0.026
0.016
0.026
0.044
0.012
0.035
Vowels
0.017
0.01
0.017
0.017
0.013
0.018
0.008
0.02
0.036
0.011
0.04
0.045
0.017
WBC
0.016
0.017
0
01016
0,073
0.008
0.019
0.075
0.941
001
0.013
0,032
0002
0.025
Wine
0.04
0.044
0.04
0.052
0.087
0.057
0.044
0.078
0.04
0.44
0.018
0.112
0.028
0.052
Yoas
0.013
0.014
0.011
0.002
0.002
0.013
0.015
0,021
0.009
0.058
0.021
0.025
0.014
Average
0.003
0,002
0.002
0.002
0.001
0.003
0.003
0.002
0,005
0.004
0,003
0.007
0.006
0.004
Table 13: Standard error of F1 scores for all methods over 30 datasets in ODDS. This table com-
plements the results shown in Section 3.2 The standard errors are computed over 5 random dataset splits.
Classical Mithods
Despleamiing bised methods
A
Datasets |Methods
Horrest
PCA
KNN
ECUD
DDD
RCA
SLAD
GUAD
Neutral
K.L
DTE
REPEN
135M
30UM
Antthyroid
0.646
0.55
0.463
0.406
0.441
0.383
0.46]
0.286
0.435
0.555
0.835
0.343
0.631
0.648
Anhyrhmia
0.662
0.617
0.556
0.603
0.561
0.5%
0.507
0.666
BreasW
0.994
0.985
0.992
0.966
0,983
0.97
0.925
0.911
Candio
0.786
0.844
0.737
0.712
0.806
0.745
0.667
0.325
0.61
0.75
0.678
0.567
0,11
18110
0.726
Bcoll:
0.086
0.16
0.739
0.189
0.004
0.176
0.11
0.012
0.43
0.064
0.777
0,03
0206
0.127
FovesCCover
0.649
0.71
0.786
0.306
0.621
0.739
0.728
0.796
0.475
0,807
0.583
D662
0419
0.417
Giass
0,195
0.167
0.242
01.42
0.363
0.187
0.208
0:15
0,484
0.374
0.236
1166
0347
0.254
Heart
0.972
0.976
0.972
0.94
0.964
0.966
0,973
0.977
0.972
0.963
0,975
0384
0972
0.960
Hup (DDCUPP9)
0.445
0.911
0.995
0.588
0.364
0.995
0,53
097
Tonesphose
0.898
0.912
0.967
0.769
0.677
0.959
0.514
0.33
Recognition
0.168
0.143
0.426
0.341
0.122
0.578
0.39
0.703
0.773
0.55
0.51
0.797
0.191
Empphugmphy
0.232
0.624
0.720
0.365
0.680
0.783
0.795
0.697
0.68]
0.718
0.747
0.607
9.856
0,938
Manezography
0,392
0.643
0.399
0.548
0.447
0.312
0.126
0,292
0,044
0.287
0,378
0.268
0.592
0,364
Mulcruss
0.989
1
0.722
0,768
0,816
0.998
0.782
Musk
0.666
1
1
(1627
1
1.175
1
Optdight
0,166
0.059
0.314
0.065
0.232
0.122
0.109
0.178
0.645
0.414
0,222
0.076
0.75
0,398
Pendight
0.544
0.376
0.95N
0.305
0.416
0.292
0.408
0.656
0.500
0.554
Pima
0.714
0.696
0.716
0.606
0.746
0.673
0,677
0.674
Sateilite
0.845
0.709
0.889
0.58
0.842
0,8066
8.866
0.00
0.86
0.887
0.843
1.80
0,891
Satimage 2
0.93
0.901
0.98
0.745
0.905
0.977
0.913
0.98
0,082
0.967
0.526
0,952
0888
8988
0,974
Seismic
0.255
0.216
0.256
0.344
0.226
0.25
0.241
0.239
0.193
0.25
0.234
11240
0236
0.281
Shuttl
0.984
0.962
0.972
0.946
0.957
0.96
0.968
0,949
0.994
0.945
0,946
0.928
0.997
0.996
Snop (KDICUPP9)
0,01
0.454
0.439
(1008
11.05
0.4:1
0,469
0.4:1
0,582
0.377
0,407
13400
0.658
0.645
Speech
0.035
0.037
0.028
0.01
0,02
0.037
0,036
0,052
0.057
0,035
0.036
0,037
Thymid
0.783
0.791
0.696
0.56
0.401
0.585
0.6%
0.74
Vertebral
0.252
0.192
0.253
0.281
0.15L
0.289
0.181
Vowels
0.229
0.162
0762
0.151
0.601
0.455
0.765
0.544
0.861
0.804
0831
0201
0.830
0.590
WBC
0.842
0.876
0.814
01586
0.747
0.808
0.711
0.408
0.226
0.714
0.61
0.235
0.873
0.753
Wine
0.672
0.659
0.711
01321
0.512
0.517
0.782
0.789
0.779
0.734
0.873
0.484
0.522
0.529
Yeast
0.44
0.366
0.294
0.323
0.299
0,313
0.196
0.076
0.168
0.262
0.282
0.187
0.301
0,302
Average
0.549
0,586
0.668
0.480
0.561
0.573
0,587
0.526
0.558
0.677
0.635
0.444
0.682
0.623
Table 14: Detailed AUC-PR scores for all methods over 30 datasets in ODDS. This table comple- ments the results shown in Section 3.2 The scores averaged over 5 random dataset splits. The highest number and second highest number in each row are highlighted in red and blue, respectively.
28

---

Published as a conference paper at ICLR 2025
Clasical Methods
Deep learning based methods
AnoLLM
Dutnsers |Motheds
Moorett
PCA
KNN
ECOD
DeeSSVD
RCA
SL.A
GOAD
NeuTral
KT1
DTE
REPEN
135M
360M
Antrhyroid
0.019
0.002
0.044
0.007
0.003
0.022
0011
Anhythmia
0013
0.011
0.91
0.15
Breast
0.001
0:008
0
0.001
0.005
0.005
0002
0
0.001
Cardio
0.01
0,005
0.007
0.002
0.015
0.004
0.008
0.005
0,034
0.013
0.008
0022
0005
0,003
Ecoli
0.044
0.027
0011
0.028
0,072
0,025
0.024
0.005
0.026
0.028
0.013
0033
0086
0.044
Pover
0.004
0.001
0.005
0.001
0.003
0.002
0.03
0.035
0.022
0.012
0,000
0084
0.007
Class
0.007
0.008
0.11
0.005
0.015
0
0.002
0,006
0.063
0.066
0.018
0013
0022
0.039
Heart
0.002
0.001
0.001
0.001
0.001
0.002
0.007
0.001
Hup (DDCCUP9)
0,023
0.001
0
0.005
0.102
0.009
Tonoophere
Letter Recegnition
0.001
0.007
0.007
0.001
0.006
0.005
0.016
0.008
0.006
0.16
0.07
0,07
0015
0.003
Lamphugrapby
0.016
0,074
0.01
0.018
0.008
0.019
0.018
0.021
0.007
0.023
0.001
0,105
0.066
0
Manenography
0.016
0.003
0.004
0.004
0.031
0.007
0.006
0.009
0.007
0.012
0.2077
B011
00007
0.007
Muleross
0,001
0
0
0.016
0,088
Musk
0.063
0
0
0.005
0
0
D.73
Opidigits
0.013
0.004
0,065
0.002
0.022
0,022
0.023
0.034
0.015
0.01
0.166
0.021
Pendigits
0.029
0.016
0,33
Pima
0.007
0.004
0003
8.00
2012
0.011
0008
0.01
Satellite
0.004
0.001
0.01
0.001
8.007
0.001
0.001
0,003
0.001
0.001
0.001
B00S
0001
0.00
Satimage-2
0.004
0.001
0.02
0.004
0.00%
0.001
6.015
0.004
0.001
0/123
0.01
0,02
0.044
Seismic
0.008
0.005
0.033
0.005
0.002
0.001
0.005
0.004
0.004
0.005
0,000
0.007
0.003
0.001
Stuttle
0,002
0,002
0.002
0.001
0.002
0,001
0.003
0,002
0.001
0.001
0,302
0,027
0
0.001
Smip (KDDCUP)))
0
0.015
0.015
0.014
0,076
0.016
0.039
0,012
0,013
0.051
0.307
D421
0.003
0.009
Specch
0
01001
0.024
0,058
Vertebral
0.018
0.099
0,023
0.016
0019
0.026
0.01
Vowels
0.023
0.015
0.16
0.011
0.011
0.018
0.022
0.018
0.016
0.034
0.014
0,021
0.04
0.025
WBC
0.157
0.022
0.177
0.016
0.089
0.017
0,029
0.077
0.019
0.03
0.127
0.017
0.14
0AE
Wine
0.061
0.034
0.43
0.018
0.085
0.029
0.04]
0.105
0.024
0.04
0.016
0.113
0.055
0.042
Yous
0.009
0.01
0.006
0.005
0.036
0.007
0.011
0.083
0.011
0.013
0.026
0.01
0.021
0.013
0.005
0,003
0.002
0.001
0.004
0.002
0.002
0.003
0.004
0,003
Table 15: Standard error of AUC-PR scores for all methods over 30 datasets in ODDS. This table
complements the results shown in Section 3.2 The standard errors are computed over random dataset splits.
Critical difference diagram average ranks
5
10
11
AnoLLM(135M)
REPEN
(11)
ICL (5.0)
99.4) DeepSVDD
DTE (5.5 S
(9) GOAD
MnoLLM(360M) (6. 1)
17.9) PCA.
Iforest (7.3)
17.8) SLAD
RCA (7.6
(7.8) NeuTral
Figure 7: Critical difference diagram of average ranks for various methods on ODDS benchmark.
The ranks are computed over AUC-ROC scores. The x-axis shows the average ranks across datasets, with smaller values indicating better performance. Horizontal bars connect groups of methods that are not significantly different in performance according to a statistical test. While AnoLLM(135MM achieves the best average rank (4.8), it is not significantly better than the best-performing baselines, KNN, ICL, and DTE.
29

---

Published as a conference paper at ICLR 2025
Dataset
Methods
Equal-width
Quantile
Language
No binning
Standard
Annthyroid
0.684
0.85
0.63
0.896
0.927
Arrhythmia
0.827
0.832
0.824
0.814
0.825
Breastw
0.991
0.982
0.991
0.991
0.992
Cardio
0.903
0.882
0.875
0.912
0.94
Covertype
0.937
0,442
0,936
0.643
0.881
Ecoli
0.795
0.831
0.854
0.729
0.777
Glass
0.879
0.401
0.835
0.392
0.819
Heart
0.808
0.804
0.799
0.827
0.82
Http (KDDCUP99)
1.000
0.166
1.000
0.999
1.000
Tonosphere
0.912
0.763
0.941
0.164
0.909
Letter Recognition
0,893
0.888
0.861
0.892
0.967
Lymphography
0.997
0.993
0.998
0.988
0.968
Mammography
0.855
0.853
0.849
0.828
0.915
Muleross
1.000
1.000
1.000
0.996
1.000
Musk
1.000
0.691
1.000
1.000
1.000
Optdigits
0.912
0.691
0,939
0.942
0.983
Pendigits
0.958
0.909
0.958
0.794
0.971
Pima
0.714
0.609
0.712
0.615
0.663
Satellite
0.843
0.892
0.842
0.888
0.902
Satimage-2
0.992
0.83
0.996
0.999
1.000
Seismic
0.733
0.698
0.74
0.732
0.712
Shuttle
0.994
0.991
0.987
1.000
1.000
Smtp(KDDCUP99)
0.89
0.501
0.816
0.909
0.927
Speech
0.456
0.479
0.46
0.545
0.47
Thyroid
0.951
0.812
0.963
0.97
0.975
Vertebral
0.461
0.54
0,461
0.471
0.565
Vowels
0.946
0.855
0.957
0.555
0.982
WBC
0.938
0.84
0.969
0.857
0.964
Wine
0.885
0.845
0.895
0,907
0.909
Yeast
0.794
0.697
0.804
0.744
0.744
Average
0.865
0.752
0.863
0.800
0.884
Table 16: Comparison of different binning methods. The table shows AUC-ROC scores of different binning methods for all datasets in ODDS library. This table complements the results shown in Section 3.3 As can be seen from the table. AnoLLM with standard rescaling performs the best and achieve the best performance on out of 30 datasets from ODDS.
30

---

Published as a conference paper at ICLR 2025
Mixed-type benchmark
Datasets
LLM backbone
Smo5LM-133M
SmolL
SmolLM-1.1 M-1.7B
fake job post fraud ecommerce Lymphography Seismic
0.800
0.814
0.802
1
0.999
0.999
0.968
0.995
0.995
0.712
0.746
0.74
Vehicle insurance
0.569
0.557
0.56
20 newsgroups
0.766
0.752
0.774
Average (Mixed-type)
0.803
0.811
0.812
ODDS benchmark
Datasets
LLM backbone
SmoIL.M-335M
SmoLLMM3600M
SmoIL.--1.7B
Annthyroid
0.927
0.931
0.93
Arrhythmia
0.825
0.822
0.824
BreastW
0,992
0.993
0.991
Cardio
0.94
0.873
0.867
Ecoli
0.777
0.804
0.791
ForestCover
0.881
0.835
0.887
Glass
0.819
0.797
0,818
Heart
0.82
0.799
0.803
Http (KDDCUP99)
1
1
Ionosphere
0.909
0.924
0.918
Letter Recognition
0.967
0.867
0.772
Lymphography
0.968
0.993
0.995
Mammography
0.915
0.876
0.874
Mulcross
1
Musk
1
1
Optdigits
0.983
0.939
0.888
Pendigits
0.971
0.964
0.93
Pima
0.663
0.654
0.653
Satellite
0.902
0.877
0.858
Satimage-2
0.999
0.999
Seismic
0.712
0.746
0.74
Shuttle
1
1
0.999
Smtp (KDDCUP99)
0.927
0.929
0.924
Speech
0.47
0.47
0.47
Thyroid
0,975
0,983
0,991
Vertebral
0.565
0.408
0.392
Vowels
0.982
0.938
0.933
WBC
0.964
0.952
0.957
Wine
0.909
0.851
0.876
Yeast
0.744
0.73
0.754
Average (ODDS)
0.884
0.865
0.861
Table 17: Comparison of different LLM sizes in the AnoLLM framework. The table shows AUC- ROC scores of different backbone LLMs of the AnoLLM from the mixed-type benchmark and ODDS library. This table complements the results shown in Section 3.4
31


Reviews:
[{"review_id": "6yv68NZU45", "reviewer": "Reviewer_18uY", "summary": "The paper proposes AnoLLM, a large language model (LLM) based framework for unsupervised tabular anomaly detection. It utilizes serialization of tabular data into sentences, and finetunes an LLM with the causal language modeling loss. The model have been applied to numerous datasets and shows competitive performances compared to other baselines.", "questions": "- What is an example of an anomaly detection in tabular data? \n- How does the time comparison look for the baselines? \n- What is \"50% of normal samples for training\"? It seems that the proposed model is unsupervised and 50% is used to train the unsupervised loss, but what does it mean for other models?\n- Have the authors considered dealing numerical columns separately? If LLM falls behind in modeling numerical values, it would be interesting to a simple concatenation of numerical values with the output of 'serialization + LLM' and applying the proposed loss.\n- Have the authors considered other encoding methods for categorical variables? (e.g., Target Encoder or the package of Skrub)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well-written and easy to follow.\n- The paper provides theoretical grounds on their model and shows competitive performances compared to other baselines.\n- The paper provides some interesting insights on anomaly detection on tabular data with large-language models.", "weaknesses": "- An example for anomaly detection (in Figure 1) would be good for reader's understanding of what is an anomaly detection for tabular data. (Possibly an example explicitly included in the dataset used for the experiments)\n- From the results, it is difficult to determine which model performs the best without explicit standard deviation (which is provided in the appendix). It would be better to have some simple plots (e.g., critical difference plots) that incorporates some statistical testing to determine which model is performing better.\n- While the authors state there is a similar trends for other metrics, it would be good to see the actual results (at least in the appendix), since there is a high imbalance of normal/anomaly.\n- The time comparison between the models and the discussion on the trade-off would be very interesting.\n- It is confusing to see \"50% of normal samples for training\" in the evaluation protocol. Clearer description would help better understanding of the experiments.\n- One interesting direction for the future works might be to pretrain a model suited for the anomaly detection. Moreover, more elaboration on impacts for the proposed model would be interesting for the conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XupryZ68Vm", "reviewer": "Reviewer_wUsV", "summary": "The paper proposes an innovative use of LLMs, that of detecting anomalies from tabular data. It is well written, and gives good results.", "questions": "W1, W2, W3", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1. The problem statement is innovative.\n\nS2. The paper is quite well written. It is easy to follow and logical.\n\nS3. The results are good.\n\nS4. Instead of simply using large LLMs, small variants are explored, and it is shown that they are no less better.", "weaknesses": "W1. The effect of number of decimal digits should have been explored in greater detail.\n\nW2. Similarly, the effect of normalization could have been explored in more detail. Although, the effect of raw numbers is seen, how about simply rounding raw number to x number of decimal digits (and not normalizing) to reduce effect of long decimal numbers, and then using them directly?\n\nW3. What is the effect of not permuting the column names, and having a canonical ordering? Are they not supposed to give even better results?\n\nW4. It will be good to highlight some failure cases, both false positives and false negatives, and try and analyze why that happened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ySZb1MvgWy", "reviewer": "Reviewer_Eu7r", "summary": "This paper presents a new framework, AnoLLM, for unsupervised anomaly detection by fine-tuning a pretrained large language model (LLM). The authors use a predefined template to serialize, i.e., convert tabular data into text for the LLM, along with preprocessing to mitigate limitations related to the model's autoregressive nature. They employ the negative log-likelihood across different column permutations to compute an anomaly score for each sample in the test set. The method is compared against various classical and deep learning methods on the ODDS datasets and six new datasets featuring mixed types of attributes. Overall, the approach demonstrates strong performance against baselines, particularly for datasets containing text features.", "questions": "- Did the other baseline also use the ensemble of permutations at inference time?\n- What explains the discrepancy in the results of ICL in this paper vs the original paper, which was also tested on OODS?\n- How did you choose hyperparameters for the baselines?\n- Given the observed trend that larger pretrained models do not seem to benefit AnoLLM, this raises the question: what would happen if we trained a model from scratch using the AnoLLM framework? This feels like a natural question that should have been explored. Did you try this? This might weaken the understanding of the text feature of the model, but it would be interesting to see the impact it has on numerical values.\n- What is the total computational cost of the experiments?\n- Was experimenting with contaminated training data (the truly unsupervised setting) considered? I ask this because reproducing the paper is not easy, and this is also an important task for anomaly detection.\n- What steps are you taking to ensure reproducibility? Will the code be released?\n- *See weaknesses*.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel model type for anomaly detection using a large language model.\n- The method provides an effective way to handle text and categorical data as features for anomaly detection, which are typically challenging to manage. To do so, they proposed a method to mitigate length-bias in the LLM’s output probabilities and theoretically validated it.\n- The authors demonstrate a preprocessing technique for tabular data, facilitating effective LLM fine-tuning.\n- The work is easy to follow and the motivation is clear.", "weaknesses": "The paper claims to outperform certain deep learning methods; however, in my experience, some of these methods perform similarly or even better than KNN (which is reported to have results comparable to the proposed method). For example, ICL outperforms KNN on the ODDS benchmark (Shenkar and Wolf, 2022), as does DTE (Livernoche et al., 2024), which was cited but not included as a baseline. The use of column permutations in the paper can be seen as a sort of ensemble strategy, a technique known to slightly improve anomaly detection performance. To ensure a fair comparison, the baselines should also be evaluated using these same permutations, as Appendix C suggests that this step may not be critical, or specific, to AnoLLM. In a small test I conducted, implementing this strategy led to performance improvements in other deep learning methods as well. Including F1-score or AUC-PR results as supplemental material would be helpful, as these metrics are more sensitive to class imbalances, which are common in anomaly detection. Scoring metrics can influence the relative ranking of methods on benchmarks. This claim that AnoLLM outperform deep learning methods should be more cautiously framed.\n\nOne key limitation mentioned at the end of the paper is the computational expensiveness of the proposed method. 7 A100 GPUs were used for LLM fine-tuning, this makes it difficult for others to access this model or replicate results of the paper. Since no code is provided, it is even more challenging to verify the reported results. Most anomaly detection methods can run on basic GPUs, or only on CPUs, a significant contrast with AnoLLM. A section discussing inference and training times would help clarify this limitation. I consider this to be the paper’s biggest weakness: its most significant limitation is not addressed at all.\n\nIn anomaly detection literature, a clear distinction exists between unsupervised and semi-supervised (or uncontaminated unsupervised) anomaly detection. While we can call them unsupervised methods, since they can be applied in both context, it should be noted that the experiments were conducted in a semi-supervised setting. The distinction lies in whether the training set contains anomalies (unsupervised) or not (semi-supervised). Section 2.1 should be revised to clarify this distinction.\n\n**Minor Comments:**\n\n- In the introduction's first line, \"specicious\" should be corrected to \"specious.\"\n- There is a double colon on line 213 (\"equation::\").\n- Figure 2’s title is missing a space between \"yellow\" and the parentheses.\n- Please use conference or journal citations rather than arXiv versions where possible. Below is a list of those I identified:\n\n  - Liron Bergman and Yedid Hoshen. Classification-based anomaly detection for general data. (ICLR 2020)\n  - Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. (ICLR 2023)\n  - Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister. Large language models can automatically engineer features for few-shot tabular learning. (ICML 2024)\n  - Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. (ICLR 2022)\n  - Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. (ICLR 2024)\n  - Xuannan Liu, Peipei Li, Huaibo Huang, Zekun Li, Xing Cui, Jiahao Liang, Lixiong Qin, Weihong Deng, and Zhaofeng He. Fakenewsgpt4: Advancing multimodal fake news detection through knowledge-augmented lvlms. (MM2024)\n  - Victor Livernoche, Vineet Jain, Yashar Hezaveh, and Siamak Ravanbakhsh. On diffusion modeling for anomaly detection. (ICLR 2024)\n  - Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. (ICLR 2019)\n  - Tomás Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient estimation of word representations in vector space. (ICLR Workshop 2013)\n  - Hu Wang, Guansong Pang, Chunhua Shen, and Congbo Ma. Unsupervised representation learning by predicting random distances. (AJCAI'20)\n  - Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. (ICLR 2024)\n  - Tianping Zhang, Shaowen Wang, Shuicheng Yan, Jian Li, and Qian Liu. Generative table pretraining empowers models for tabular prediction. (EMNLP 2023)\n  - Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. (ICML 2023)\n  - Yaqi Zhu, Shaofeng Cai, Fang Deng, and Junran Wu. Do LLMs understand visual anomalies? uncovering LLM capabilities in zero-shot anomaly detection. (MM2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1r4S0HkC2W", "reviewer": "Reviewer_Sz6K", "summary": "This paper proposes AnoLLM, a new framework that uses large language models (LLMs) for unsupervised tabular anomaly detection. AnoLLM assigns anomaly scores based on the negative log likelihood for anomaly detection. The authors claim that AnoLLM detects anomal samples in raw features and can deal with textual and categorical features. Experimental results show that AnoLLM achieves good performance on six benchmark datasets with mixed feature types. AnoLLM is also competitive with KNN.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It’s good to consider different types in tabular data, such as textual, numerical and categorical columns.\n\nExperiment results show good potential for AnoLLM.", "weaknesses": "The method seems too simple and lack novelty. It’s easy and direct to consider to use the negative log likelihood for anomaly detection. Even though authors consider different types of columns in tabular data, the process seems to be the same.\n\nI think it’s inappropriate to claim that you are the first to apply LLMs to tabular anomaly detection. There are many works in this area, such as “Anomaly detection of tabular data using llms”, and other corresponding works such as “LLMClean: Context-Aware Tabular Data Cleaning via LLM-Generated OFDs”, “Enhancing Anomaly Detection in Financial Markets with an LLM-based Multi-Agent Framework”.\n\nThe backbone LLM is not so famous such as Llama, Qwen, Mistral, etc. Why authors do not use these LLMs?\n\nI don’t understand what the differences in contribution 3 and 4. It seems they are all talking about the experiment.\n\nIn experiments, only one of the datasets here has text columns. More such datasets and the dataset with more attributes (features) should be considered.\n\nMore ablation studies should be provided such as the impact of Random column permutations. Case study is also missing, I cannot recognize which samples are anomaly.\n\nBy the way, the formatting on line 249 could use some improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)