FoMo-0D: A Foundation Model for Zero-shot Tabular Outlier Detection
Authors: Yuchen Shen
Haomin Wen
Leman Akoglu
Venue: Accepted by TMLR ()

Abstract: Outlier detection (OD) has a vast literature as it finds numerous real-world applications. Being an unsupervised task, model selection is a key bottleneck for OD without label supervision. Despite a long list of available OD algorithms with tunable hyperparameters, the lack of systematic approaches for unsupervised algorithm and hyperparameter selection limits their effective use in practice. In this paper, we present FoMo-0D, a pre-trained Foundation Model for zero/0-shot OD on tabular data, which bypasses the hurdle of model selection altogether. Having been pre-trained on synthetic data, FoMo-0D can directly predict the (outlier/inlier) label of test samples without parameter fine-tuning—requiring no labeled data, and no additional training or hyperparameter tuning when given a new task. Extensive experiments on 57 real-world datasets against 26 baselines show that FoMo-0D is highly competitive; outperforming the majority of the baselines with no statistically significant difference from the 2nd best method. Further, FoMo-0D is efficient in inference time requiring only 7.7 ms per sample on average, with at least 7x speed-up compared to previous methods. To facilitate future research, our implementations for data synthesis and pre-training as well as model checkpoints are openly available at https://github.com/A-Chicharito-S/FoMo-0D.

Published in Transactions on Machine Learning Research (09/2025)
FoMo-OD: A Foundation Model for Zero-shot Tabular Outlier Detection
Yuchen Shen Cornegie Mellon University
yuckensi3anddrew. edu
Haomin Wen
Cornegie Mellon University
Leman Akoglu Carnegie Mellon University
lakoglu www.:/wwwwwwwwwww.w...oommmmm... crna, edu
Reviewed on OpenReview: https: /ttp://oppp openreview. net/ forun? id= XCQeupR9
Abstract
Outlier detection (OD) has a vast literature as it finds numerous real-world applications. Being an unsupervised task, model selection a key bottleneck for OD without label supervision. Despite a long list of available OD algorithms with tunable hyperparameters, the lack of systematic approaches for unsupervised algorithm and hyperparameter selection limits their effective use in practice. In this paper, present FoMo-OD, pre-trained Foundation Model for zero/O-shot OD on tabular data, which bypasses the hurdle of model selection aer. Capitalizing on synthetic pre-training, FoMo-OD can directly predict the coutlee//inieer label of test samples without parameter fine-tuning. Even more importantly, requires no labeled data, and no additional Iraining or hyperparameter buning when given new task. Extensive experiments on 57 real-world datasets against 26 baselines show that FoMo-OD is highly competitive; outperforming the majority of the baselines with statistically significant difference from the 2nd best method. Further, FoMo-OD is efficient in inference time requiring only 7.7 ms per sample on average, with at least. 7x speed- up compared to previous methods. To facilitate future research, our implementations for data synthesis and pre-training as well as model chockpoints are openly available at http://ttt..ao////ccccccarraaata
1 Introduction
Outlier detection (OD) in tabular data finds numerous applications in critical domains such as security, environmental monitoring, finance, and medicine, to name a few. This popularity brings along a large literature with plethora detection algorithms to choose from given a new OD task. These algorithmss, however, exhibit several hyperparamoters (HPs) that need carcful tuning (Ma et al., 2023). Since most OD tasks are unsupervised what makes elfective detection notoriously difficul is unsupervised model selection (both algorithm and HP selection) in the absence of labels.
While deep learning has revolutionized many areas of machinc learning (ML), it is not quite the case for OD. This is mainly because compared to classical methods, deep OD models Pang et al. 2021) have many more HPs to which the detection performance is sensitive (Ding et al. 2022). making model selection even more challenging. While the recent success large foundation models, pre-trained on massive amounts of data, offers new opportunities for zero-shot OD, thus far the most notable progress has been in NLP and computer vision (Brown al. 2020 Touvron et 2023 Radford al. 2021 This thanks to the admirable
"hiile supervised OD exists, unsupervised setting preferred most domains detect novel, emergent anomalies.

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: p-values of the ono-sided Wilcoxon signed rank test, comparing FoMo-OD (with D 100) to top 10 baselines with default hyperparaneteers (HPs), and top 4avg baselines with avg. performance over varying HPs (denoted w/ = over All (57) datasets, those (42) w/ d S 100 and (46) w/ d I 500 dimensions, and those (47) excluding NLP, CV datascts. FoMo-OD shows no statistically significant difference from the 2nd best model (kNN, w/ p= 0.106) over All datasets, and none of the differences are significant when embedded, i.e., NLP and CV datasets are excluded, while it is comparable to (p > a) or significantly better than (p - a) all 26 original + 4 baselines over datasets w/ d d 100 (aligned w/ pretraining where D= 100) as well as on datascts w/ d < 500 (generalizing beyond pretraining). We use underline and bold to indicate p < a and p p>1 a. Rank, avg. 'ed over all 57 datasets by AUROC. (setting: D 100, R 500, train/inferent context 5K, w/ quantile transform, a = 0.05) (See Tables 17.1 17.2 for full results.)
FoMo-oD
DTE-NP
ANN ICI, DTEEC L.F
CBLOF Feat. Bag. SLAAD DDPM OCSVM D'TENPe ANN ICI,
D
S 100
0.415
0.700
0.949
0.953
0.970
0.971
0.996
0.876
0.980
0.978
0.752
0.860
0.958
1.000
d S 500
0.220
0.569
0.827
0.894
0.960
0.968
0.994
0.910
0.960
0.979
0.607
0.756
0.846
1.000
All
0.016
0,106
0.462
0.454
0.585
0.750
0.823
0.759
0.901
0.895
0.112
0.315
0.670
1.000
(NI, CV)
0.164
0.476
0.757
0.832
0.934
0.945
0.988
0.867
0.938
0.965
0.515
0.683
0.777
1.000
Rank(avg)
11.886
7.553
9.018
10.851
11:36
12.316
13.312
13.386
12,982
14.061
13.851
9.079
11,105
12.901
22,263
quantity and quality of public text and image datasets. In comparison, public tabular OD benchmarks remain
minuscule (Han et al. 2022 Zhao et al. 2021 Steinbuss & Böhm 2021).
Recently, Prior-data Fitted Networks (PFNs) has marked a milestone in ML as a new approach to learning on tabular data (Müleer et al. 2022). The core idea is to compute a posterior predictive distribution (PPD) for a test point given training data as context. To approximate the PPD, a Transformer (Vaswani et al, 2017) is pre-trained on a large set of synthetic datasets drawn from pre-defined data priors. At inference, the pre-trained PFN is fed with lest. samples along with some training samples as context for zero-shot prediction, requiring no parameter fine-tuning model selection on new datasets. Variants of PFN are shown to match tree-based models in performance on small classification datascts (Hollmann et al. 2023 as well as time series forecasting (Dooley el al. 2023
In this paper, we capitalize on these ideas and introduce FoMo-OD: a prior-data fitted Foundation Model for zero/O-shot OD. Once pre-trained on synthetic datasots, FoMo-OD unlocks zcro-shot OD on a new datasot where the (unlabeled) input data is fed only as context. As such, FoMo-OD bypasses not only model (parameter) training, but more importantly, the nontrivial task of unsupervised model (algorithm and HP) selection without labeled data. Figure [ jillustrate the new FoMo-OD paradigm versus the typical OD setting. To our knowledge, FoMo-OD is the first pre-trained foundation model for tabular OD.
In designing FoMo-OD, we use Gaussian mixture models as a simple yet effective tabular data prior for inlier data distributions (Hollmann et al. 2023 Zhao et al.) 2021 which are also employed to simulate outlier types common in the real world; namely, local and global subspace outliers (Steinbuss & Böhm 2021). While the data prior can be extended to comprise more complex data distributions (Hollmann et al. 2023) (e.g.
Bayesian Neural Networks [Ncal 2012) and Structural Causal Models (Pearl 2009)), and additional outlier types can be included (e.g. dependency, conlextual, etc.), as we show in the experiments, even with its relatively straightforward prior, FoMo-OD achieves remarkable performance: As shown in Table I FoMo-OD, which is pro-trained on datasets with d S 100 dimensions, shows no statistically significant difference from all 26 statc-of-tho-ae baselines (all p-values > 0.4) on 42 benchmark datasets with dimensionality d < 100 (aligned with pre-training), while our method consistently ranks among the top and outperforms a majority of the baselines with p-value 0.95. (See Appendix Tables [7. 17.1 for full results.) The results remain consistent on 46 benchmarks with d s 500 dimensions. FoMo-OD is also competitive across all 57 datasets, effectively generalizing beyond ils pre-training distributions, with no statistically significant difference from the 2nd best bascline. When intrinsically non-tabular datasets are removed (i.c., datascts from NLP, CV domains embedded with pre-trained encoders), there is no statistical evidence to suggest a performance difference between FoMo-OD and any baseline on the remaining 47 datasets. Further, FoMo-OD takes a mere 7.7 ms to infer a test sample on average with no extra training or tuning overhead on the new dataset. We summarize the main contributions of our work as follows.
2

---

Published in Transactions on Machine Learning Research (09/2025)
A Foundation Model for Tabular OD:
We present FoMo-OD, the first foundation model for zero- shol OD on unseen tabular datasets, with no additional training or hyperparameter tuning, backed by Transformer-based in-context learning (ICL), synthetic data pre-training, and food-forward inference.
Model Selection Made Obsolete: FoMo-OD is designed for zero-shot inference given new dataset, fully abolishing not only model training on the new dataset, but also the notorions task of algorithm selection and hyperparameter tuning in the absence of labeled data.
Scalable Pro-training: To enable pre-training many large datasets, we propose (t) new mechanism to reduce sample-to-sample attention from quadratic to linear time enabling larger datasets, as well as (ii) on-the-fly data synthesis through data transformations eenioormaaiiioo-eebinnnnnn more diverse datasets in less time.
Fast OD at Inference: Given new dataset, FoMo-OD bypasses both model training and selection, both of which can be slow for modern deep OD models with many hyper/ ppper/praameers. Rather, it takes fraction of second to label test point through single forward pass. Such speedy inference also unlocks the potential for deploying FoMo-OD in real time on data streams.
Effectivencss: On a large benchmark of 57 datascts (Han et al. 2022 from diverse domains and against 26 baselines ranging from classical to modern OD models (Livernoche et al. 2024), FoMo-OD out putporforms the majority of the baselines, with no statistical evidence for performance difference from the 2nd best. baseline, while operating fully zero-shot on real-world datasets out-of-the-box.
As the first foundation model for OD, FoMo-OD presents a paradigm shift in how to perform OD in practice, while offering new directions to explore as well open questions to investigate. From the algorithmic perspective, how can we understand what (OD) algorithm, if any, has the Transformer- and ICL-based FoMo-OD learned? How can we interpret (mechanistically or otherwise) how the pretraining achieves zero-shot and out-of-distribution (OOD) generalization? From a data perspective, what prior distributions for pretraining are suitable for downstream real-world tasks? What the role of prior data diversity and complexity in the generalization ability of the model? In summary, FoMo-OD paves the path towards a better understanding of ICL as well as building more powerful future foundation models for OD.
New directions:
2
Problem and Preliminaries
2.1 Outlier Detection Problem and Setting
Outlier detection (OD) methods can be categorized based on the availability of labeled data. In supervised OD, the lask is similar to binary classification with imbalanced classes (as outliers typically make up only a small portion of the overall data). The more difficult unsupervised setting assumes that the "contaminated" training data contains both inliers and outlicrs, but without any labels. This is also the transductive sotting where training and lest data are the same. The one-class setting lies between these two extremes, where the "clean" training data consists only of inliers, while the unlabeled test data contains the outliers to be detected. Here, training and test sets disjoint and thus the setting is inductive. Note that there exist no labeled outliers in both settings, making model selection challenging.
We remark that some OD literature refers to the latter setting as semi-supervised OD, which is a misnomer from the supervised ML perspective where semi-supervised classification assumes the presence of some labeled instances from all classes in the training data. In the rest of text, we adopt the terminology unsupervised OD for both settings, and specify "clean" inlier-only versus "contaminated" mixed training data to differentiate them. Then, our work considers the unsupervised OD problem under "clean" inlier-only training data.
Formally, let Di,- = {(x1. v1).... (Xn, bn)} denote the inlier-only input data x; E Rd, where = 0 vi E [n]. and Drest depicts the unlabeled test data comprising both inliers and outliers. Note that Din n Drest = 0, i.e. train/test split is disjoint. The task is to assign labels to x; E Drest given the inlicr-only input Din.
2.2 Background on Paior-data Fitted Networks
In the Bayesian framework for supervised learning, the prior defines hypothesis space & which expresses our beliefs about the data distribution before seeing any data.
Posterior Predictive Distribution (PPD):
3

---

Published in Transactions on Machine Learning Research (09/2025)
Typical OD Setting
FoMo-OD Setting
Train on the same, single dataset
Pre-train once offline, on many synthetic datasets
xN
Data prior
Simulated data
x
Real-world dataset
(I)
Train
D (i) Irain
Model training
fo Data-specifie OD model
PPretra
GMM
sim
Model training
fg
Simulated dataset
D sim
(2)
Input OD dataset
Gaussian
SCM
FoMo-OD
mixture model
GMM
1: Train & Test on each dataset
1: Many
Pre-train once, Test many times
Structural
causal
xN
Single forward process
x
NN
SCM
model
Data-
Test
fe
Outlier score
Test
D
fa
Outlier score
score
f
specific
test
Test
OD model
Score (a new part of) same dataset
Zero-shot score on unseen, new real-world dataset
FoMo-OD
Figure 1: (best in color) Comparison of typical OD the FoMo-OD settings. Given new unlabeled OD dataset, FoMo-OD not only eliminates the need for model (parameter) training, but most importantly, also abolishes the oncrous task of unsupervised model selection (algorithm and hyperparameters)
Each hypothesis + describes mechanism by which the data is generated. The posterior predictive distribution p(-|Xceat, Perain) provides a framework for making prediction on new, unseen test data Xtest conditioned on observed training data Derain = ((x1111)))) .i).... (x......,,..)..))) Based on Bayes' Theorem, the PPD can be derived by the integration over the space of hypotheses $:
P( Xtest; , Derain) = p(yheest Xtost, (Diiinn p)p(d)dé,
(1)
where p(ő) denotes the prior probability and p(D(Ø) is the likelihood of the data D given d.
PFNs and PPD Approximation: As obtaining the above PPD is generally intractable, Prior-data Fitted Networks (PFNs) are proposed to approximate the PPD (Müller et al. 2022). Unlike traditional machine learning models that are trained directly on observed datasels, PFNs are pre-trained on simulated datasets that are generated according to a prior distribution. Specifically, it contains the pre-training and inference stages described as the following.
Pre-training on synthetic data. Massive synthetic datasets are generated for the pre-training stage, by first. sampling hypothesis (i.e., the generating mechanism) 6 ~ p(0), and then sampling dataset D p(D]o). For training, each dataset D can be split as Drest C D and Derain = D Drest. Thus, the PFN with parameters 0 can be optimized by making predictions on data points in Deest- For a Lest point (Xeest, Brest) E Diests the training loss is formulated as follows.
L
www.aaaaneeeaoo ((D)euDDDp(()))) log qo (ytest Xtest; D Derain))
(2)
The above loss can also be interpreted as minimizing the expected KL divergence between p(-1x, xx, D) and go(-|x, D) [Müller et al. 2022). In practice, a PFN model qo is typically implemented by a Transformer-based architecture (Vaswani et al.J 2017), which takes (Xtosts Dirain) as input, where Xtest E Drest and Derain contains an arbitrary number of instances. The output is the conditional class probabilities for Xtest As the whole training set Dersin is passed as input/context to the Transformer, it learns to predict class labels through sample-to-sample attention.
Inference real-world data. In the inference stage, a fresh real-world dataset D and some test instance Xtest are fed into the (frozen) pre-trained model, which computes the PPD pe(-|Xtest, Derain) in a single forward pass. Importantly, PFNs do not require gradient- based paramcter tuning on new datasets, where prediction is delivered in less than a second (Hollmann al. 2023).
In summary, PFNs are trained once, and can be used many times for zero-shot inference on new datasets with different characteristics. The main benefit is that no training tuning is required at the inference
stage. This type of learning ability is also termed as in-context learning (ICL) (Xie et al. 2021 which was
4

---

Published in Transactions on Machine Learning Research (09/2025)
shown to be effective for various tasks in languages (Brown et al. 2020 In fact, ICL with PFNs is recently shown to be a promising paradigm for supervised classification on tabular datasets (Hollmann et al. 2023
3 FoMo-OD: A Foundation Model for Zero-shot Tabular Outlier Detection
Inspired by PFNs (uäleer et al., 2022 Hollmann et al. 2023 Dooley et al. 2023 we propose FoMo-OD for zero-shot OD, which is pre-trained on large-scale synthetic OD datasots for zero-shot detection at inference time. FoMo-OD eliminates the need for model training on new dataset and for model selection (both algorithm and HPs), which is difficult without any labeled data. The new FoMo-OD paradigm (right) versus the typical OD setting (left) is illustrated in Figure 1
In the following we describe our OD data prior, training on prior-simulated datasets, inference on new datasets, and model architecture and improvements for scalability.
3.1 Designing Data Prior for Outlier Detection
Foundation models benefit from massive amounts of datasets available for pre-training, along with high- capacity model architectures, however, the quantity (and quality) of publicly available tabular OD datasets is mimuscule, compared to the massive size of open-domain text corpora. Even with large quantities of data, Ansari et al. (2024) show that using synthetic data in combination with real-world data improves the overall zero-shot performance for time-series foundation models. Hence, we design a new data prior from which we simulate numerous OD datasets for pre-training FoMo-OD.
Ideally, the data prior should reflect distributions as general and diverse as seen in the real world, however, "finding a prior supporting large enough subset of possible [data generating] functions isn't trivial" Nagler 2023). Surprisingly, our results show that a straightforward, achieve remarkable performance.
data prior is sufficient to
We simulate inliers by drawing from a Gaussian Mixture Model (GMM) with m-clusters in d-dimensions, with centers Mys E [-5,5]], j E [m,) k E [d] and diagona Σ with entries in 50.5). We create different GMMs with varying S M and d S D chosen uniformly random from [M] and [D], respectively. From each GMM, we draw set of S inliers, defined as instances within the 90th percentile of the GMM.
Inlier synthesis:
Following Han et al. 2022); we generate subspace outliers by first drawing a subset of dimensions K. at random, for K d, and then generate S points from the "inflated" GMM, which shares the same centers pyys 's with the original GMM but with the inflated (diagonal) covariances x Ez,ua' 's for & K. Outliers are defined as points sampled outside the 90th percentile of the original GMM, which are labeled based their Mahalanobis distances (see Property B.6 in the Appendix).
Outlier synthesis:
Specifically, we simulate datasets containing 2S = 10, 000 samples (half inlier, half outlier) from the two corresponding GMMs (original and inflated) with up to M = 5 clusters and up to D = 100 dimensions. Example 2-d synthetic datasets are illustrated in Appendix A
Remarks: Our model not trained on any real-world data but rather, on purely syuthetic data (although future work can combine existing benchmark OD datasets with synthesized data, as was done by [nsari et al. (2024) for time series). While we have intended to extend our preliminary attempt toward designing sophisticated data prior for OD, found (to our surprise) that even with basic, GMM-bbsed prior, FoMo-OD generalizes remarkably well to real-world OD datasets downstreant out outperforming numerous SOTA baselines. Therefore, we present FoMo-OD with this simple prior to showcase the prowess of PFNs for OD. We leave as future work the exploration of other priors (Hollmann et al., 2023) and other outlier types (contextual, the impact of different priors on performance, as well as prior
dependency, etc. [Steinbuss & Böhm 2021)) ), mixture composition to further improve performance.
"In carly experiments, we found no difference in test performance on synthetic datascts between using diagonal vs. nou-diagonal Σ, yct, it is casier to invert diagonal for data synthesis. Weo refer to Appendix G.2 and Figure for an exploration FoMo-OD performance and GMM gooduess on real-world OT) datasets.
5

---

Published in Transactions on Machine Learning Research (09/2025)
3.2 (Pre) Training and Inference
Model (Pre) Training (Once, Offline): FoMo-OD is a Prior-data Fitted Network (PFN, see Section 2.2 based on the Transformer architecture. In the synthetic prior-data fitting phase, it is trained on datasets drawn from our OD data prior for tabular data introduced in Section 3.1 Each dataset is simulated from a different GMM configuration based on randomly drawn parameters, and consists of varying number of training samples and dimensions to capture the diversity in real-world tabular datasets. Details are outlined in Algorithm T in Appendix C and described follows.
At each time, we first draw hypothesis (i.e. GMM configuration) uniformly at random, that is, Φ = {d E [D], m E [M]. } E [-5,5]", diai E [-5,5]]], and then generate a synthetic dataset D = (Din } containing synthetic inlicr and outlier samples from the drawn hypothesis and its variance-inflated variant, respectively.
We optimize ToMo-OD's parameters 0 to make predictions on Drest fest Dest conditioned on the inlier-only training data Dirain Dm based the cross-entropy loss (see Eq. (2. During training, Drest contains a bolanced nunber of inlier and outlier samples, where Dia Din \Dtrain and Dout C Dout contains an equal number of samples as Di To vary the training data size, we subsample Perain of randomly drawn size n E [nI, nvl, where nı, and ny denote the lower and upper bounds. In our implementation, we use nL = 500, and ny = 5, 000.
FoMo-OD trained on 200, 000 batches (200 epochs x 1, 000 steps/epoch) of B = generated datasots each batch. While this pre-training phase can be expensive, it is done only once, offline. Moreover, we introduce several scalability improvements to speed up pre-training, the training and implementation of FoMo-OD are given in Appendix C
discussed later in Section 3.3. Full details on
Zero-shot Inference (on Unseen/New Dataset): At inference, the pre-trained FoMo-OD can be employed on any unseen real-world dataset. Specifically, for a new unsupervised OD task with inlier-only training data Durain and mixed test data Drest: feeding (Derain, Xrest) as input to Fo (for each Xrest E Drest separately) yields the PPD 90 (y Dursin) in single forward pass. As such, FoMo-OD performs model "training" and prediction simultaneously at test time. In fact, as the training data is passed as context, FoMo-OD leverages in-context learning (ICL) (Xie et al. 2021 Garg et al. 2022 for inference.
The key contribution of FoMo-OD goes beyond eliminating the need for model training for a new dataset, it renders model selection an obsolete concern for OD. In other words, a practitioner with a new detection task no longer needs to choose an OD model to train or grapple with tuning any hyperparameters of the said model. Further, the speedy, easily parallelizable inference (for lese-ulan-a-second per lest sample) is the "icing on the cake". Figure | (right) illustrates (top) pre-train and (bottom) test phases of FoMo-OD, where the pre-trained FoMo-OD is reused during inference on new datasets directly, unlocking zero-shot OD.
Remarks:
3.3 Architecture and Scalability
Architecture and sample-to-sample attention: Like existing PFNs, FoMo-OD is based the Transformer (Vaswani et al. 2017). encoding each sample's feature vector a fixed size token through linear embedding layer (see Appendix C.2 and allowing token representations to attend to each other, hence enabling sample- to-sample attention. We also adopt the three customizations from TabPFN (Hollmann et al., (023),, which (1) computes self-attention among all the training samples and only cross-attention from test samples to the training samples, (2) enables varying feature dimensionality by zero-padding, and (3) randomly permutes input samples while omitting positional encodings to achieve model invariance in the dataset.
Given Dirain fxi....,,.lI X,), each self-attention layer outputs embeddings {z, 1 where the i-th token is mapped via linear transformations to key k,, query qi and value v,, where the i-th output is computed as
n
Z; softmax( ((q. ())))))) ) Vj
(3)
The sample-to-sample attention is intriguing from the perspective of OD: many classical OD algorithms Aggarwal 2013) are based nonparametries; in particular, they leverage the distances to the k nearest neighbors (kNNs) of a point to compute its outlierness, where l is critical hyperparameter. One can think

---

Published in Transactions on Machine Learning Research (09/2025)
of FoMo-OD as mimicking non- con-aaametrric models but by using parametric attention mechanisms. Interestingly, PFNs are much more robust and flexible than ANN based OD approaches, for (1) sample-to-sample relations are not pro-specified but rather Icarned through attention weights, and thus (2) they are not limited to just the nearest neighbors but rather can learn which training points are worth allending to, and (3) as attention is dataset-wide across all points, there is no need for specifying a cut-off HP value like to which most ANN based OD techniques sensitive to (Aggarwal & Sathe 2015 Campos et al.) 2016 Goldstein & Uchida 2016
Ding et al. 2022 We present analyses on sample-to-sampl attention in Appendix E
To seize the power of scale, we incorporate a scalable architecture and data synthesis into our design to benefit pro-training and inforence, as we describe next. The scale-up unlocks a larger context size for FoMo-OD, enabling pre-training and inference on larger datasets with fast speed.
Scaling up attention with "routers": The O(n²) quadratic sample complexity at pre-training presents an obstacle for achieving high performance at inference, as it limits pre-training to relatively small training datasets, and degenerates in-context learning that typically benefits from longer context (Xie et al. 2021
Toward a higl-performance model, we scale up FoMa-OD's attention via the "router mechanism" of Zhang & Yan (2023). As shown in Figure 2 the main idea is to learn a small number (R < n) of "routers", which gather information from all n samples and then distribute the information back to the n output embeddings, in effect, reducing complexity from (nn) to Rm)) = (n). This design allows FoMo-OD to scale linearly with respect both dimensionality d and dataset size n in pre-training as well as during inference.
a (+/x test D train
Z:
(R.M(R.M
lost
M(R-3)
(RZZ)
Z:
Zn
D Deni {Xx,, Yi), (x2, Y2),
Yn)}
test
Figure 2: FoMo-OD architecture employs the "router mechanism" for scalable attention.
Concretely, the representatives first aggregate information from all samples by serving the query in the multi-head self-attention (MSA):
M = MSA1 (R Z),
(4)
where R R Rxd depicts the learnable vector array of reprosentatives and M denotes the aggregated mossages. Then, the routers distribute the received information among samples by using the sample embeddings query and the aggregated messages as both key and value:
Ż = MSA (Z, M,
55)
Finally, we obtain Z = LayerNorm 2 + Z) after layer normalization. Note that the test samples only attend to the training samples' embeddings, computed in the described manner across layers, and are finally fed into the prediction head to estimate the PPD at the output layer.
Scaling up (pre)training data synthesis with linear transforms: Besides the scalability challenge associated with ieatttectur//ttennttiin another computational challenge in pre-training FoMo-OD arises from drawing samples from the data prior, which requires considerable time, especially in high dimensions
"This is because the inverse of the (d x d) covariance matrix plays crucial role in the process of drawing samples from GMMs, which has (()) time complexity. (It is also the reason why diagonal E, 's are favored in our data prior.) In addition, Mahalanobis distance for labeling nolierr/uullirss also requires the inverse.

---

Published in Transactions on Machine Learning Research (09/2025)
provided the large number of datasets we sample (specifically, we utilize a batch size of 8 datasets over 1,000 steps each for 200 epochs).
To give an idea, sampling a dataset with n = 10, 000 points in = 100 dimensions using 10 CPUs in parallel takes =0.4 seconds (see Appendix Figure 77. Across 200 training epochs with 1,000 steps each, it adds up more than 177 hours just to generate 1,6 million datasets on-the-fly. Of course, one can trade storage with compute-time by generating all these datascts apriori via massive parallelism. Neverthcless, synthetic data generalion demands considerable time (and/or storage).
To scale up data synthesis, FoMo OD employs two distinct strategies. First, we propose reuse at epoch level: that is, one can reuse the same 8K (8 x 1000) unique datasets at every epoch, or in general, the same &K x P datasets periodically at every P epochs. A larger P would lead to more diversity in terms of the overall pre-training data used.
Second, we propose reuse at dataset level via transformation: that is, having generated one unique dataset X E Raxd from GMM, propose a linear transform T(x) of the form Wx + b for randomly drawn parameters W E Rdxd and b E Rd (sce Appendix B.1 3 This simple yet efficient transformation creates a new dataset, akin to one being drawn from another GMM with centers T(p,) = WHy + b and covariance T(Σ,)= W WT, Vj E [m]. Note that we do not actually materialize these parameters but only transform the dataset. As we show in the following, such transformations preserve the Mahalanobis distances as well as the plrcentile thresholds for labeling points as /nlier/outlir. Details and proofs are given in Appendix B Lemma 3.1. Linear transform T with invertible W on g preserves Mahalenobis distances.
Lemma 3.2. Linear transform T with invertible W on Ih preserves the percentiles of the GMM.
The implication of these lemmas is that linear transformation of dataset from GMM retains the identity of the inliers and outliers, i.c. no relabeling is required. Morcover, notice that as a byproduct we obtain a transformed dataset as though it is drawn from a GMM with a non-diagonal covariance matrix which, besides the time savings, offers a slightly more complex data prior.
To reach 8K unique datascts for each epoch, we first generate 500 datasets from different GMMs (with varying configurations), then employ 15 different linear transformations each dataset by varying W and b. Drawing each (W, b) takes =0.02 seconds, while the matrix-matio product of X (n x d) and W (d d) takes negligible time (for d S 100). Thus, obtaining a transformed dataset offers 20x speed-up compared to generating one (0.02 vs. 0.4 seconds).
4 Experiments
4.1 Setup
We present. the experiment. setup briefly, including data synthesis, real-world datasets, baselines, metrics and HPs. For more details, we refer to Appendix D
Pre-training Dataset Synthesis: During pre-training, we generate unique GMM datasets by first drawing
a configuration, including dimensionality d E [D], number of components m E [M], centers { (each My E 5-5,5] and covariances {Ez} (diag(E) Ej) E [-5,5]]). We set M 5 and vary DE {20, 100} to study pre-training with relatively small and high dimensional datasets, respectively. We synthesize inliers
and outliers described in Section 3.1
Real-world Benchmark Datasets: While pre-training is purely on synthetic datasets, we evaluate FoMo-OD on 57 real-world datasets from ADBench (Han al. (2022) (see Table 20 in Appendix J.. Following Livernoche et al.] (2024), we use 5 train/test /tst splits of each dataset via different seeds and report mean performance and standard deviation. Note that the basclines require model re-training and inference for cach Prest split, while FoMo-OD uses the splits only for inference as Dirain is passed as context.
iIn practice, apply the linear transform the subspace of inflated features only, wherein inliers and outliers defined, which remains to be multi-variate GMM.
8

---

Published in Transactions on Machine Learning Research (09/2025)
We compare FoMo-OD against 26 baselines, from classical/shallow methods to modern/decy models. The baselines are imported from one of the latest papers that proposed the SOTA diffusion-based OD model, DTE (Liernochhe et al. 2024), and three variants; DTE-C, DTE-IG, DTE-NP. As such, the long list of baselines we compare to constitutes one of the most comprelenssve in the literature. We refer to the original paper for more details.
Baselines:
Model Implementation: We train our final model for 200,000 steps with batch size of 8 datascts. That is, FoMo-OD is trained on 1, 1,600,000 synthetically generaled datasels. This training takes about 25 hours on 1 GPU (Nvidia RTX A6000). Each dataset had a fixed size of 10,000 samples, with [Derainl E [ng = 500, ny = 5000], and the rest as Drest with balanced number of inlicrs and outlicrs. Other details of FoMo-OD, including the training algorithm, model architecture, data synthesis and reuse, and hardware are in Appendix C
Metrics and Hypothesis Testing: Detection performance is w.r.t. 3 widely-used metrics for OD: AUROC; area under ROC curve, AUPR; area under Ppoeision-Recalll curve, and F1 score; using threshold at the true number of outliers in the test data (varies by dataset) Livernoche et al. (2024).
To compare different methods on ADBench, we compute their rank on each dataset (lower is better), and present average rank across datasets. This is an alternative to the average metric (e.g. AUROC), which is not meaningful when tasks vary widely in terms of their difficulties.
In addition, we perform significance tests to compare two methods statistically, using the ono-sided paired Wilcoxon signed rank test (Demšar 2006) between FoMo-OD and a baseline based on the performances across all datasets, with the alternative hypothesis suggesting the Doneeinne-mius-FoMoooo performance gap is greater than zero. We consider results to be significant at a 0.05 following convention.
Hyperparameters (HPs): Importantly, Livernoche et al. 2024) picked for each baseline the best- performing set of HPs as recommended by the authors in their original paper. As for their own DTE, which behaves similarly to kNN, they use k = 5 and set the same for the NNN baseline (Ramaswamy et al. 2000 to be consistent. However, is well known that &NN is sensitive to the value of (Aggarwal & Sathe 2015
and are many other OD models to their respective HPs (Campos et al. 2016 Goldstein & Uchida
2016
Zhao et al. 2021 Ding et al. 2022).
Therefore, besides comparing FoMo-OD with the 26 basclines in Livernoche et al. 2024). respectively for AUROC, F1, and AUPR (Livernoche et al. 2024), we also compare to the top-4 best-performing baselines (in order: DTE-NP, &NN, ICL, and DTE-C) on their average performance across a list of different HP settings. Such an approach reflects their expected performance under HP values selected at raudom, in the absence of any other prior knowledge, as recommended by Goodstein & Uchida (2016) "to get a fair evaluation when comparing [OD] algorithms". We annotate the method name with avg for the version with performance averaged over varying HPs. The detailed list of HP values for each top baseline is given in Appeudix D.4
Overall, we compare FoMo-OD to 30 baselines; 26 from Livernoche al. 2024) and avg of the top-4.
4.2 Results
Detection performance: Table 1 presented the comparison of FoMo-OD w/ D= 100 to all baselines w.r.t. average rank across datasets as well as pairwise Wilcoxon signed rank tests based on AUROC, and we present full results all datasets and all metrics in Appendix I We find that among 30 baselines and 2 variants of FoMo-OD (w/ D = 100 and D = 20), FoMo-OD w/ D = 100 performs as well as the 2nd best model (kNN with default HP; k- = 5) on all datascts. While DTE-NP outperforms FoMo-OD with author-recommendeel k = 5, we find that DTE-NPayg is on par with FoMo-OD.
In our tests, p > a = 0.05 implies no statistical evidence for performance difference between two methods. FoMo-OD w/ D 100 performs statistically no different from all basclines on datasots with d I 100 (i.c., "at its own game" " when pre-training data dimensions align with real-world datasets), while it outperforms the majority of baselines (where p 1 ---a) a). These results continue to hold on datasets with d < 500.
6 rank the 26 baselines, we compute the 26x26 p-values of the pairwise Wilcoxon signed rank test (see Appendix Figure 24 and order them by their mean pavaue against other baselines.
9

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: p-values of the one-sided Wilcoxon signed rank test, comparing FoMo-OD (w/ D 20) to top 10 baselines with default HPs, and top 4 baselines with avg. performance over varying HPs (denoted w/ "rs) over All (57) datasets, those (24) w/ S 20 and (38) datasets w/ d < 50 dimensions. Although pretrained on datascts w/ small D = 20, FoMo-OD shows no statistically significant difference from the top 3rd baseline (ICL, w/ p 0.089) over All datasets, while it outperforms (w/ p 1 a) the top 5th (T.OF) and onward baselines over datasets w/ d I 20 (aligned w/ pretraining where D = =20) and on datasets w/ < 50 (generalizing beyond pretraining). We use underline and bold to indicate p < a and p > 1-66000 a. Rank is avg'cd over all 57 datascts, where methods are ranked on each dataset w.r.t. AUROC. (experiment setting: D 20, P 50, R 500, Irain/inference context size=5K, data transformation, a 0.05)
FoMo-OD DTENP ANN ICL DTE-C LOF CBLOF Feat. Bag. SLAD DDPM OCSVM DTE-N &NN' ICL me DY''-CWE
d S 20
0.572
0.789
0.968
0.616
0.993
0.989
1.000
0.978
0.906
0.992
0.813
0.924
0.999
1.000
d S 50
0.347
0.794
0.893
0.946
0.997
0.988
1.000
0.963
0.994
0.986
0.574
0.847
0.995
1.000
0.001
0.019
0.089
0.159
0.394
0.434
0.703
0.516
0.752
0.679
0.007
0.062
0.437
1.000
Rank(avg)
12.59
7.19
8.57
10.34
10.79
11.82
12.81
12.8
12.52
13.50
13.34
10.63
12.44
21.43
Table shows similar results for FoMo-OD w/ D 20, which is pre-trained on datasets with considerably fewer dimensions. Even in this limited setting, it performs par with the 3rd best bascline (ICL, with default HP) against 30 baselines, with increased p-value (0.437) when compared to ICL On dalasets with d < 20 which align with its pre-training data, it out perforrss the top 5th baseline and the majority of others. With FoMo-OD OD pre-trained purely on synthetic datasets from simple prior in small dimensions, these results showcase the prowess of PFNs for OD.
Figure shows the distribution of AUCROC across datasets for all models (See rank distribution Appendix Figure [7]. FoMo-OD achieves a competitively small average rank with notably higher AUCROC across datasets compared to the majority of the baselines. Appendix H presents another comparison between
detectors through performance profile plots (Dolan & Moré 2002
FoMo(D= 100)
FaMa(D-20 20)00
DTE.NP
ANN
ICL
DTE-C
LOF
CBLOF
Bag.
MCD
PCA
Flaw
HBOS
GANomaly
GOAD
DIF
COPOD
ECOD
DeepVVDD
LODA
DAGMM
DROCC
DTE-
KNN-aW
ICL
DTE-C-aVE
20
40
60
80
100
AUROC
Figure 3: (best in color) AUROC (higher is better) distribution across all 57 real-world datasets shown via boxplots for (from top to bottom) FoMo-OD in red, all 26 basclines ordered by mean p-value (shallow and deep baselines in green and blue), and top 4 baselines' avg variants. The vertical line depicts the mean, the box shows the 25 75%, bars range 5-95%, and circles show the datasets at the tails.
Running time: Table 3 presents the total training time and the average inference time per test sample for FoMo-OD compared with the top-3 basclines measured ADBench's 's largest dataset. Given new dataset, FoMo-OD bypasses model training (and HP tuning) and directly performs inference, within 7.7 ms per sample
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Train-time and Inference-time (in milliseconds) of FoMo-OD and top-3 baselines (w/ defoult HPs, excluding time for hyperparameter optimization) on our largest dataset donors (see Appendix Table 20 FoMo-OD skips any model training or fine-tuning and takes mere forward pass for inference out-of-the-box.
Method
FoMo-OD
DTE-NP
kNN
ICL
Train-time (total)
none/O-shot
1,170.29
174,157.38
130,412.86
Infer-time (per sample)
7.7
1.38
0.65
0.02
on average (see Appendix Figure G In comparison, all baselines need to train on each individual dataset before inference. Training time can be high for deep learning based models like ICL, further compounded by hyperparameter tuning that requires training multiple models. Even for non-parametric and/or shallow models like kNN and DTE-NP (which query & nearest neighbors), training involves various data pre-processing steps such constructing tree-like data structure" for fast (often approximate) kNN distance querying.
4.3 Ablation Analyses
Extensive ablations in Appendix F analyze (1) the effect of D in
F.1
(2) cost and performance by varying R
in
F .2 and F.3. (3) context size in F.4 (4) reuse periodicity P in F.5 (5) effect of data transformation T
on per formanceeee and speed-up F.6 and F.7 (6) data diversity and prolonged training in F.8 (7) quantile
transformation in F.9 and (8) model size in F.10
4.4
Generalization Analyses
Our results that synthetic pretraining at large, and GMM data prior in particular, enables FoMo-OD to reach remarkable performance on real world tasks call for deeper investigation on its generalization. To this end, Appendix G provides an extensive analysis on FoMo-OD's generalization to out-of-distribution (OOD) synthetic
datasets G.1 GMM statistical goodness-of-fit analyses of real-world datasets in ADBench in G.2 as well
as generalization to real-world OOD detection tasks in
5 Related Work
Outlier Detection (OD): Thanks to diverse applications in numerous fields, such as security, finance, manufacturing, to name a few, OD on tabular (or point-cloud) datasets has a vast literature with a long list of techniques. For earlier, shallow approaches preceding the advances in deep learning, we refer to the books by Aggarwal (2013) and Aggarwal& Sathe (2017). The modern, deep learning based techniques are surveyod in Chalapathy & Chawla (2019) Pang
(2021); Rulf et al. (2021). ). Most recent deep OD techniques take advantage of newly emerging paradigms, including self-supervised learning (Hojjati et al. 2022 Yoo et al.
2023 as well the most recently popularized diffusion- based models Yoon et al. 2023
Livernoche et al.
2024
Du et al.
2024 He ct al. 2024h.
Unsupervised Model Selection for OD: It is typical of models to exhibit various hyperparameters (HPs) that play a role in the bias-variance trade-off and hence the generalization performance, and OD models are no exception. Many carlier work on OD showed the sensitivity of classical (i.c. shallow) OD methods to the choice of their HP(s) (Aggarwal & Sathe 2015 Campos et. al., 2016 Goldstein & Uchida 2016). Similarly, sensitivity to HPs has also been shown for deep OD models more recently (Zhao et al.] 2021 Ding al.
2022), as well as for those relying self-supervised learning/ /data augmentation (Yoo et al. 2023
While critical, work on unsupervised outlier model selection (UOMS) is slim as compared to the vast literature on detection mothods. A handful of existing, mostly houristic strategics has been studied by Ma et al. 2023 reporting discouraging results; they have shown that existing heuristics are either not significantly different from random selection, or do not outperform iForest LLu et al. 2008) with its default HPs.
Both kNN (from PyOD Zhao et al. 2019 and DTE-NP use BallTree from sklearn (Pedregosa et al. 2011 for nearest neighhor search; however, KNN has extra computations after calling the BallTree, which might lead to the time differences for training and inference compared to DTE-NP. We encourage the readers to refer the official implementations for details.
11

---

Published in Transactions on Machine Learning Research (09/2025)
Recent UOMS approaches go beyond heuristic measures and design scalable hyperensembles (Ding et al., 2022;
2024 : or leverage meta-learning on historical real-world OD datasets (Zhao et al. 2021 2022: Zhao &
A 2024]. These approaches show the value of (meta)learning and transferring from historical tasks to a new task. Though grounded in the same idea of learning from a large set of (in our case, simulated) lasks, we differ in one key aspect: FoMo-OD is not a model selection technique, but rather, a foundation model that abolishes model training and selection altogether. As such, it unlocks zero(0)-shot inference on new task.
Prior-data Fitted Networks: Based on the seminal work by Müller et al. (022). Prior-data- fitted Networks (PFNs) establish a new paradigm for machine learning, where a PFN is pretrained on synthetic datasots generated from a data prior, and the pretrained PFN can then infer the posterior prodictive distribution (PPD) for test points in a new dataset in a single forward pass, through in-context learning (Xie Garg et al., 2022). - It. is shown that PFNs provably approximate Bayesian inference (Müller Follow-up TabPFN (Hollmann al. 2023) and its v2 [Hollmann et al. (2025) achieved SOTA
et al.
2021
eet al. 2022)
classification performance
small tabular datasets of size up to 1024. Other subsequent works designed
LCCPFN (Adriaenser et al. 2024 and ForecastPFN (Dooley et al. 2023 respectively wero-shot learning
curve extrapolation and zero-shot time-sories forecasting models, trained purely on synthetic data. PFN4BO
(Müller et a 2023) employed PFNs for Bayesian optimization, while Nagler (2023) studied the statistical founattion of PFNs. Others proposed scaling the context size to enable training on larger datasets toward
better generalization (Ma et al. 2024 Feuer et al. 2023
2024 Qu et al. 2025).
Our proposed FoMo-OD differs from these in being the first PFN for OD, using a novel inlier/ data prior, employing linear transform for fast data synthesis, and incorporating the "router" attention mechanism for linear-time scalability w.r.t. context size. See Appendix K for additional details.
Zero-Shot Outlier Detection: Poundation models pretrained on massive text and image corpora, such as large language and/or vision models (I((V)MMs) like OpenAT's GPT-series (Achiam et al. 2023). DALL-E. [Ramesh et al.) 2021) and Flamingo (Alayrac et al. 2022), CLIP (Radford et al. 2021) and LLaVA (Liu ct al.] 2024) to name a few, have demonstrated remarkable success on several zero-shot tasks in CV and NLP. Follow-up work extended these models for zero-shot out-of-distribution detection (Fsmaeilpour et al. 2022
zero-shot image OD (Liznerski et al. 2022 Jeong et al. 2023 Zhou et al. 2024 as well as ttallgue-basse
industrial image anomaly detection Guu et al.] 2024b.
Foundation models, however, do not exist for tabular data which is widespread across OD applications in the real world, such as detecting credit card fraud, network intrusion, medical anomalics, and any sensor measurement abnormalities, to name a few. The recent. ACR model by Li et al. (2023) on zero-shot OD does not rely on a pretrained foundation model, but rather is meta-trained on each specific domain using inlier-only datasets from the same domain. Concurrent to our work, [i et al. (2024) apply pretrained LLMs for prompt-based OD on tabular data which they serialize to text. Similar to our work, they also use simulated labeled OD datasets to fine-tune several existing LLMSs to improve their performance. Their work, however, is quite preliminary in several fronts; a key limitation is that they assume independent features and query the LLM ono-feature-at-a-time to reach an outlier score. Further, they finc-tune using only 5,000 data batches with up to 100 samples each, subsample 150 points and the first. 10 columns of each dataset for evaluation (due to GPU memory constraint),, and their testbed includes only two bascline methods. In contrast, FoMa-OD employs and pretrains PFNS a much larger scale with rigorous evaluation on a much larger testbed.
6 Conclusion
This work introduced FoMo-OD, the first foundation model for outlier detection (OD) tabular data. It capitalizes on the in-context learning of Transformer model pre-trained on a large number of synthetic datasets that can then perform zero-shot inference on a new dataset, without any hyper/ /pper//ameeeer tuning/trainie FoMo-OD breaks new ground by fully abolishing the notoriously-hard model sclection task for unsupervised OD (see Impact Statement). Further, FoMo-OD offers extremely fast inference thanks to a mere single forward pass. Against a long list of 26 SOTA basclines on 57 public real-world datasets, FoMo-OD performs on par with the 2nd best baseline, while outperforming the majority of the baselines. Future work could expand our data prior and explore similar directions for zero-shot OD beyond tabular data, For a detailed discussion on limitations and future directions, we refer to Appendix L
12

---

Published in Transactions on Machine Learning Research (09/2025)
Broader Impact Statement
FoMo-OD offers zcro-shot outlier detection (OD), abolishing not only parameter training but also model selection given a new dataset. This is a radical paradigm shift for the OD literalure, which historically focused on designing new models and recently unsupervised model selection. Obviating the need for either, we expect FoMo-OD to route attention of the community from new model design and selection to designing better data priors and gathering datasets for pre-training, along with better and more scalable architectures for PFN.
From the applied perspective, a zero-shot OD model like FoMo-OD is game changer for practitiones!! Given the plethora of OD algorithms to choose from, each with a list of hyperparameters to set, not having the tools for effective and efficient model selection leaves the practitioners with a "choice paralysis". With FoMo-OD, practitioners can not. only bypass such dilemmas on one dataset, but thanks to the "train once; use many times" " nature of pre-trained models, they can do so for any dataset, including those arriving over time, FoMo-OD is weigt and optimized for fast inference (without any additional training or tuning), making it especially attractive for real-time or resource-constranned applications. While attractive from a performance viewpoint, we remark that FoMo-OD currently does not factor into account metrics beyond detection performance, such as fairness, biases, or other potential blindspots. In sensitive real-world applications, social and ethical costs of incorrect detections should be taken into consideration.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. Advances in Neural Information Processing Systems, 36, 2024.
Charu C. Aggarwal. Outlier Analysis. Springer, 2013.
Charu C. Aggarwal and Saket Sathe. Theoretical foundations and algorithms for outlier ensembles. Acm sigkdd explorations newsletter, 17(1):24-47, 2015.
Charu C. Aggarwal and Saket Sathe. Outlier Ensembles An Introduction. Springer, 2017.
Samet Akcay, Amir Alapour-bbarghoeei, and Toby P Breckon. Ganomaly: Semi-supervised anomaly detection
via adversarial training. In ACCV, pp. 622-637. Springer, 2019.
Tean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXio preprint urx::22403. 03.02703.07815, 2024.
Lion Bergman and Yedid Hoshen. Classification- based anomaly detection for general data. In International Conference on Learning Represenlations, 2020.
Markus M. Breunig, Hans-Peter Kricgel, Raymond T. Ng. and Jörg Sander. Lof: Identifying density-basod local outliers. In International Conference on Management of Dala, 2000.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Newral Information Processing Systems, volume 33, pp. 1877-1901 2020.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Guilherme O Campos, Arthur Zimek, Jörg Sander, Ricardo JGB Campello, Barbora Micenková, Erich Schubert, Tra Assent, and Michael E Houle. On the evaluation of unsupervised outlier detection: measures, datascts, and an empirical study. Data mining and knowledge discovery, 30:891-927, 2016.
George Casella and Roger Berger. Statistical inference. CRC Press, 2024.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. arXiv preprint MKKi::991.03440 2019.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
Xucying Ding, Lingxiao Zhao, and Leman Akoglu. Hyperparameter scnsitivity in deep outlier detection: Analysis and scalable hyper-ensemble solution. Advances in Newral Information Processing Systems, 35: 9603-9611 2022.
Xueying Ding, Yue Zhao, and Leman Akoglu, Fast unsupervised deep outlier model selection with hypernet- works. ACM SIGKDD, 2024.
Elizabeth D Dolan and Jorge Moré. Benchmarking optimization software with performance profiles. Mathematical programming, 01:201-213, 2002.
Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha Venkat Naidu, and Colin White. Fore- cast Syuthetically-trainee trained zero-shot forecasting, In Shirty-seventh Conference on Newral Information Processing Systems, 2023.
Xuefeng Du, Yiyou Sun, Jerry Zhu, and Yixuan Li. Dream the impossible: Outlier imagination with diffusion models. Advonces in Neural Information Processing Systems, 36, 2024.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of- di---riiitiiittiion detection based the pre-trained model CLIP. In Proceedings of the AAAT conference on artificial intelligence, volume 36, pp. 6568-6576, 2022.
Benjamin Feuer, Niv Cohen, and Chinmay Hegde. Scaling tabPFN: Sketching and feature selection for tabular prior-data fitted networks. In NewrIPS 2023 Second Table Representation Learning Workshop, 2023.
Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable prior-data fitted networks, 2024.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Newral Information Processing Systems, 2022.
Markus Goldstein and Andreas Dengel. Histogram- outlier score (hbos): A fast unsupervised anomaly detection algorithm. KI-2012: poster and demo track, 1:59-63, 2012.
Markus Goldstein and Seiichi Uchida. comparative evaluation of unsupervised anomaly detection algorithms for multivariate data. PloS one, 11(4), 2016.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Simhadri, and Prateck Jain. Drocc: Deep robust one-class classification. In Proceedings of the 37th International Conference on Machine Learning, ICML/20.
JMLR.org, 2020.
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. AnomalyGPT: Detecting industrial anomalics using large vision-language models. In Proceedings of the AAAI Conference on Artificial Inlelligence, volume 38, pp.
2024.
A. Gut. An Intermediate Course in Probability, Springer Texts in Statistics. Springer New York, 2009. ISBN 9781441901620.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Advances in Neural Information Processing Systems, 35, 2022.
Haoyang He, Jiangning Zhang, Hongxu Chen, Xuhai Chen, Zhishan Li, Xu Chen, Yabiao Wang, Chengjie Wang, and Lei Xie. A diffusion-based framework for multi-class anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 8472 8480, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Hadi Hojjati, Thi Kieu Khanh Ho, and Narges Armanfard. Self-supervisod anomaly detoction: A survey and outlook. Xie preprinl urXi::2205.051 2022.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, 2023.
Noah Hollmann, Samuel Muller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Ti- bor Schirrmeister, and Frank Hutter. Accurate predictions on small data with tabular foundation model. Nature, 637(8045):33 326, 2025.
Catherine Huber-Carol, Narayanaswamy Balakrishnan, Mikhail Nikulin, and Mounir Mesbah. Goodness-of-fit tests and model validily. Springer Science Business Media, 2012.
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winelip: Zero-/fow-shor anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Poltern Recognition, pp. 19606 19606-19616, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883 088.61,,,.8.1...... 2020.
Diederik P Kingma, Auto-encoding variational bayes. arXiv preprint arXiv:1812.6114 2013.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt. Zero-shot anomaly detection via batch normalization. In Thirty Conference on Neural Information Processing Systems, 2023.
Aodong Li, Yunhan Zhao, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt.
Anomaly detection of tabular data using LLMs. arXiv preprint arXiv:2406. 16308, 2024.
Fei Tony Lin, Kai Ming Ting, and Zhi-Hua Zhon. Isolation forest. In 2008 Fighth IFEE International Conference on Data Mining, pp. 413 422, 2008
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Victor Livernoche, Vinect Jain, Yashar Hezaveh, and Siamak Ravanbakhsh. On diffusson modeling for anomaly detection. In ICLR, 2024.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joc Franks, Klaus-Robert Müller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. arXiv preprint Xrx::2200..1 11205.1144444 11474, 2022.
Junwei Ma, Valentin Thomas, Guangwei Yu, and Anthony L. Caterini. In-context data distillation with tabpfn. CoRR, 1b5/2400.066971, 2024.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Martin Q Ma, Yue Zhao, Xiaorong Zhang, and Leman Akoglu. The need for unsupervised outlier model selection: A review and evaluation internal evaluation strategies. ACM SIGKDD Explorations Newslelter, 25(1):19-36 35, 2023.
Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter. PFNs4BO: in-context learning for bayesian optimization. In International Conference on Machine Learning, 2023.
Samuel Müller, Noah Hollmann, Sebastian Pineda-Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. ICLR, 2022.
Thomas Nagler. Statistical foundations of prior-data fitted networks. In ICML, volume 202 of Proceedings of Machine Learning Research. PMLR, 2023.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2):1-38, 2021.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596-20607, 2021.
Judea Pearl. Causality. Cambridge University Press, 2009.
Fabian Podregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825 2830, 2011.
Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. Tabiel: A tabular foundation model for in-context learning on large data. arXiv preprint ar arXiv:2502.0556/ 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, 2021.
Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. In ACM SIGMOD international conference on management of data, 2000.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non bayesaa in-context learning for regression. Advances in Neural Information Processing Systems, 2024.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
Lukas Rulf, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification, In International Conference on Machine Learning, pp. 4393-4402. 2018.
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Rober Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, (09(5):556-795 795, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Tom Shenkar and Lior Wolf. Anomaly detection for tabular data with internal contrastive learning. In International Conference on Learning Represenlations, 2022.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523-19536, 2022.
Georg Steinbuss and Klemens Böhm. Benchmarking unsupervised outlier detection with realistic synthetic data. ACM Transactions on Knowledgee Discovery from Data (TKDD), 15(4):1-20, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL
https://arrav..org/ab//222. 13971
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-6008, 2017.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111 11.02080, 2021.
Hongzuo Xu, Yijie Wang, Juhui Wei, Songlei Jian, Yizhou Li, and Ning Liu. Fascinating supervisory signals and where to find them: Deep anomaly detection with scale learning. In International Conference Machine Learning, pp. 38655-38673. PMLR, 2023.
Jaemin Yoo, Tiancheng Zhao, and Leman Akoglu. Data augmentation is hyperparameter: Cherry-picked self-supervision for unsupervised anomaly detection is creating the illusion of success. Trans. Mach. Learn. Res., 2023, 2023.
Sangwoong Yoon, Young-Uk Jin, Yung-Kyun Noh, and Frank Park. Energy-basod models for anomaly detection: manifold diffusion recovery approach. Advances in Newral Information Processing Systems, 36:49445 494666, 2023.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer Scaling vision transformers. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognilion, 2022.
Jingyang Zhang, Jingkang Yang. Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, et al. Openood v1. 5: Enhanced benchmark for out-of- distribution detection. Xiviv preprinl ar arXiv22366.992 2023.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multi- variate time series forecasting. In JCLR, 2023.
Yue Zhao and Leman Akoglu. Toward unsupervised outlier model selection. In International Conference on Automated Machine Learning (AutoML), 2024.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96):1-7, 2019. URL http:://jmr..ot htttpp//jjaaroorraapppeee////010
Yue Zhao, Ryan Rossi, and Leman Akoglu. Automatic unsupervised outlier model selection. Advances in Neural Information Processing Systems, 34:4489-4500 2021.
Yue Zhao, Sean Zhang, and Leman Akoglu. Toward unsupervised outlier model selection. In 2022 IEFE International Conference on Data Mining (ICDM), pp. 773 782. IEEE, 2022.
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. AnomalyCLIP: Object-agnostic prompt learning for zero-shot anomaly detection. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpervwwwwwwwttteeeeeeeee wtwp://ooormm////eccC/C/22/
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dae ki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix
Table of Contents
We detail the contents in the appendix below.
Appendix A Illustration of Synthotic Sata in 2-d illustrates the inlier and outlier data synthesis for pre-training with a 2-dimensional example.
Appendix B Linear Transform for Scalable GMM Data Synthesis contains the proofs for efficient data synthesis in Section 3.3
Appendix Implementation Details includes the training and inference details of FoMo-OD.
Appendix D Detailed Experiment Setup introduces the details of pre-training and inference datasets, baselines, and their hyperparameters.
A Appendix E Qualitative Analysis on Sample-to-Sample Attention visualizes the altention of FoMo-OD.
Appendix Ablation Analyses studies different design choices of FoMo-OD.
Appendix G Generalization Analyses studies the generalization ability of FoMo-OD on out-of- distribution synthetic datasets, ADBench, and benchmarks.
Appendix H Performance Profile Plots presents comprehensive comparison of different methods via the cumulative distribution.
Appendix D Full Results presents the detailed metric results of FoMo-OD and the baselines, including AUROC, AUPRC, and F1.
Appendix J. each dataset in ADBench.
Benchmark OD Datasets shows the details (c.g., number of samples, fcatures) of
Appendix K Differences to Prior Work on PFNs for Tabular Data explains the difference and innovation of FoMo-OD from previous works.
Appendix L Discussion provides the summary of our work and discussions on the limitations and future directions of FoMo-OD.
Appendix M. Reproducibility Statement details the codebase for FoMo-OD.
18

---

Published in Transactions on Machine Learning Research (09/2025)
A Illustration of synthetic data in 2-d
We visualize our synthetic data in Figure with randomly created 2-d GMMs with the number of clusters (N = 1, 2,3.. We choose the 80th percentile as the criterion, such that inliers are samples drawn from the GMM and within the 80th percentile, and outliers are samples drawn from the inflated GMMs and outside of the 80th percentile.
4
4
X
2
2-
x
Y-axis
x
Y-axis
Y-axis
X
x
2
2-
Xx
-6
x
x
-6
55
-2
-2
X-axis
X-axis
X-axis
Figure 4: Illustration of synthetic data in 2D with 80th percentile as the criterion.
B Linear Transform for Scalable GMM Data Synthesis
B.1 Definitions
Definition B.1 (Gaussian Mixture Model). We denote an m-cluster d-dimension Gaussian Mixture Model as g ((w,, Mjs Mد>} which is the weighted sum of m Gaussian distributions:
p(x)
Wi g( E,),
(6)
where wy R++ the weight for the j-th Gaussian N( Ej) with Σ wj and g(-1,,, Σ,) is the density of the j-th component/cluster with mean/center M4 E Rd and covariance Σg IRdxd being positive semi-definite, such that xT Σxx for all E Rd.
Definition B.2 (Linear Transform). We denote linear transformation T in Rd as:
T(x) = Wx + bb,
(7)
where x E Rd, and W E dddd sexddbb E Rd are the parameters of T. Definition B.3 (Mahalanobis Distance). The Mahalanobis distance dist between a point x E Rd and a Gaussian distribution N(, E) is defined as:
dist v (x 1x p1)
(8)
Definition B.4 (x2-distribution). The Chi-squared distribution X2 with d degrees of freedom is the distribution of the sum of squares of d independent standard Normal random variables.
B.2 Properties
Property B.5 (Lemma 5.3.2 (Casella & Berger 2024 ). If ~ N(O, 1), then Z xi: X, Xa are independent and X₁ x2, then Ed X₁ x2.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Property B.6. The squared Mahalanobis distance dist ~ Xa: with x ~ Mn, E). Proof: If x ~ Np, Σ), then we have 2 = Σー > (x (4) (a,
Gut 2009
such that:
(x) z²zz
2
(9)
where 2; are independent standard Normal random variables. We have E 23 ~ x² from Property B.5 which completes the proof.
B.3 Lemmas
Lemma B.7. Linear transform T with invertible W on G preserves Maholanobis distances.
Proof:
We denote the transformed GMM as T(g") = {(w,, WHy + b, WE, w )ر then with x
Mi,, Ej), for the transformed point T(x) we have:
dist (T(x)) = (T(x) - b))( - (Wa, + b))
(10)
(W(x (WW")
(11)
(x
14,)
(12)
(V (ox-M,) (x m1) dist (x)
(13)
0
Lemma B.8. Linear transform T with inverlible W on 9h preserves the percenliles of the GMM.
Proof: Let x2(a) denote the a-th percentile of Xa= such that for x2:
Prob(X S xa(n)) 100
(14)
Based on Property B.6 we have Prob(dist (x) S x2(a)) a 100
Let x gd such that dist x2(a) for all N1 Ej), which indicates that is outside the a-th percentile of gd Since d is preserved under T (see Lemma B.7y, then we conclude that the linear transform with invertible W preserves the percentiles of the GMM.
C
Implementation details
C.1 Hardware
We base experiments NVIDIA RTX A6000 GPU with AMD EPYC 7742 64-Core Processors.
C.2
Training and inference
We train our models for 200 epochs with the Adam optimizer (Kingma & Ba 2017) and a learning_rate 0.001, and test with the model corresponding to the lowest training loss. The size of our D {20, 100) model is 4.87M and 4.89M parameters, respectively. We show the training process of PFNs and our model in
Algorithm I
Dealing with varying dimensions and dataset size To handle input with varying number of d features, we follow Müller al. (2022). Specifically for d < D, we rescale the input with & and pad the features size D with Os; and for d > D, we randomly sample D features out of d. In addition, FoMo-OD uses context. size of up to 5K at inference, where for each test sample Diest, randomly sample (5K-1) points Dusin from datasets with 5K.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1: Prior-fitting of a PFN (Mülee et al. 2022 and ours
Input A prior distribution over datasets p(D), from which samples can be drawn and number datasets
Q to draw for one epoch, the number of training epochs E, the periodicity P, the number of unique datasets g, linear transformation T.
Output : model qe that will approximate the PPD
Initialize the neural network q6; Initialize the epoch-level collection C. []: for i+ 1 to E do
if i S then
5
Initialize empty buffer B; = []: Initialize the dataset-level collection C2 []: for to Q do if j S.S q then
7
8
Step 1: sample Dy = Derain u {xx,, yz) Puaal p(D);
9
10
Cg + (D,)
11
end
12
else
13
jtj mod Dj - (()
14
15
end
Step 2: compute stochastic loss approximation &o = 1 (- (-log (o (xx, Derain))
16
17
Step 3: update parameters with stochastic gradient descent on Vola;
B, B, [D,]
19
end
20
Cg + Cg + [8.]
21
end
22
else
23
iti mod P
24
B. + C
25
for
j + to Q do
26
Dj (B)
27
Perform Step 2 and Step 3
28
end
29
end
30
end
Model architecture We use 4-layer Transformer with hidden dimension h dim 256, a linear embedding layer at the input (R + gh_din), and a 2-layer MLP layer the output (Rh_din + R22 for inlier vs. outlier binary classification. For cach Transformer layer, num head 4 for each attention module and R 500 for the router-based attention (Figure 2
Training loss In Figure we plot the training loss of our D = 100 model trained with 8K unique datasets/epoch (denoted as "SK") versus 0.5K unique + 7.5K transformed datasets/epoch (denoted as "0.5K+T"). together with the D 20 model trained with reuse periodicity P (denoted as "P=1", reusing the same 8K datasets across epochs) and P = 1 with transformation (denoted "P=1+T", transforming the 8K datasets across epochs). Notice that the loss with transformation is slightly higher than no transformation (i.e., D = 100, "0.5K+T" vs. "8K", and D = 20, "P=1+T" vs. "P=1") across all 200 epochs, which is reasonable since the transformed datasets have non-diagonal covariances that make the learning task harder and thus result in a higher training loss. The training losses of FoMo-OD with D = 100 are also higher than with = 20 since the subspace OD tasks are harder in higher dimensions.
Inference time Figure 10 (left) showed the inference time of FoMo-OD CPU, comparing typical attention versus the based attention (with R = 500 routers) under varying context sizes from 1K to 10K. The
21

---

Published in Transactions on Machine Learning Research (09/2025)
training loss
0.35
8K (D=100)
0.5K+T (D=100)
0.30
P=1 (D=20)
0.25
P=1+T (D=20)
loss
0.20
0.15
0.10
0.05
0.00
25
50
75
100 epoch
125
150
175
200
Figure 5: (best in color) Training loss of FoMo-OD (D 100) with 8K unique datasots/epoch (in blue) and using 0.5K unique + 7.5K transformed datasets /epoch (in orange), and FoMo-OD (D = 20) with P = 1 (in green) and P with transformation (in red) over 200 epochs.
time is measured on CPU to clearly showcase the scalability trends; quadratic without routers and linear with routers.
Inference s
Figure shows the inference time on GPU. Notice that the time much lower (in milliseconds), thanks to the Trans-
10
with (R=500)
without
8
former architecture taking advantage of GPU parallelism, while the compute time for attention without routers contin- ues to grow faster than that with routers.
6
4
In implementation, FoMo-OD (with R = 500 routers) uses inference context size 5K by default, which takes about 7.7 ms per test sample on average.
2
0 1k 2k
5k
10k
D Detailed Experiment Setup
Context size
Figure 6: Inference time of FoMo-OD on GPU with vs. w/out router-based attention under varying context size.
D.1 Pre-training Dataset Synthesis
During pretraining, we generate unique GMM datasets by first drawing a configuration, including dimension- ality d [D], number of components m E [M], centers 3m (cach H4 E [-5, 5)d) and covariances {E,} (diag(E) E (-5, We set M = and vary E {20, 100} to study pretraining with relatively small and high dimensional datasets, respectively. We synthosize inlicrs and outlicrs as described in Section 3.1
We then sample S 5, 000 points that are within the 90th percentile of the GMM. To synthesize outliers, we "inflate" subset of dimensions by randomly choosing |[] E [D] dimensions and multiplying the corresponding variances by x5 (following (Han al. 2022)). ), L.c. x Σj,t&' 's for k K, and then draw S = 5, 000 samples from the inflated GMM that are outside the 90th percentile of the original GMM.
To speed up data synthesis via linear transformations, we first draw 500 unique datasets using m E [5] and d E 21,2,... .... 100) (i.c. 5x 5x100) and transform each one 15x using varying parameters (W, b) as described in Section 3.3 This yields SK unique datasets (500 original and 7,500 transformed) to use al one training epoch (over 1, 1,000 steps with batch size B 8). We repeat this process at each epoch, drawing 500 new datasets and transforming them reach 8K datasets per epoch.
It is important ensure that the eigenvalues of W (L.e. variances) are not too small such that the dataset does not flatten in any direction. To this end, draw random orthonormal basis U E [-1, 1]dxd and diagonal A with eigenvalues Auk. E ([- 4.-0.1J (0.1, 1I), and obtain W UAU"" We also use b [-1,1]
22

---

Published in Transactions on Machine Learning Research (09/2025)
D.2 Real-world Benchmark Datasets
While pretraining is purely on syuthotic datasets, we evaluate FoMo-OD on 57 real-world datascts from the ADBench benchmark (Han et al. 2022) (see Table 20). They consist. popular labular outlier detection datasets, as well as 10 nexly-coustructed tabular datasets created from images and natural language tasks by using pretrained models to extract embeddings. We defer to the original paper for the details on these benchmark datasets.
We compare to DTE (Livernoche et al. 2024 and baselines therein as described next, thus, following their OD setting with inlier-only Dirain, we split each dataset five times into train using five different seeds and report the mean performance and its standard deviation. In particular, each random split designates 50% of the inliers as Dtrain, while Diest contains the rest of the inliers and all the outlier samples. Note that while the baseline methods require model re-training and inference for each Duain /Dtst split, FoMo-OD uses the splits only for inference Drrain merely passed as context.
Before passing the datasets as input to FoMo-OD, perform quantile transform such that the features follow a Normal distribution, to better align with the pretraining data from GMMs.
D.3 Baselines
We compare FoMo-OD against 26 baselines, from classical/shallow methods to modern deep models. Our baselines include all the basclines imported from one of the latest papers that proposed the SOTA diffusson- based model DTE [Livernoche et al. 2024), and its three variants; DTE-C, DTE-IG, and DTE-NP. Their
2022); both classical ones (ANN (Ramaswamy et al., 2008 HBOS (Goldstein & Dongel 2012), etc.) and deep models (DeepSVDD (Ruff et al. 2018 DAGMM Zong et al. 2018), DROCC (Goyal al. 2020), etc.).
baselines comprise all those in ADBench (Han et al.
2000), LOF (Breunig et al., 2000). iForest [Luu et al.
They also include more recent approaches based self-supervised learning (GOAD (Bergman & Hoshen 2023), etc.), besides the four additional generative baselines: normalizing planar flows [Rezende & Mohamed] 2015), DDPM (Ho et al. 2020), VAE (Kingma
2020), ICL (Shenkar & Wolf 2022), SLAD (Xu et al.
2013) and GANomaly (Akeay et al. 2019). We defer to the original paper for additional details. Overall, our
26 basclines consist of the most recent, SOTA approaches for OD that span a diverse family (nonparamotric, sell-supervised, generative, elc.).
D.4 Hyperparameters for Baselines
Table gives the list of HP values we used to study the HP sensitivity petitviityy/offfmmaccc variability of the (from top to bottom) top-4 baselines.
Table 4: Top-4 basclines (from top to bottom) and hyperparameter (HP) configurations.
Baseline
Hyperparameters
DTE-NP
k (5, 10, 20, 40, 50}
KNN
k E (5, 10, 20, 40,
ICL DTE-C
learning_rate E {10 100,10122000 2,10 1,10 10-0) -3 E 10, 20, 40, 50}
D.5 Ranking the 26 baselines
Figure 24 presents the visualization of the p-values of the pairwise Wilcoxon signed rank test w.r.t. AUROC among the baseline methods used by Livernoche et al. (2024). We rank these 26 basclines based their mean p-vuu (i.e., wow- average) against the other baselines.
23

---

Published in Transactions on Machine Learning Research (09/2025)
D.6
Comparison of top-4 baseline variants with varying HP configurations
Figure 25
27 28 give the p-values, respectively comparing the variants of the top-4 baselines (DTE-NP, KNN, ICL, DTE-C) C) among thenselves using different HP configurations, as well as the avg model with the
26
average performance across HPs. (Specifically for ICL, learning_res caating_rate (1r) {10-1, 10 10000- 10 9.10- 3] and for others, #nearest-ncighbors & E {5, 10, 20, 40, 50)). We find that for ICL,, 1r= 10-3 3 or 10-4 are preferable while those that are too smalll or too large perform poorly, For others, small & E (5,10) 10] tend to outperform larger & E {40,50 50}. Note that Livernoche et al. (2024) used k = 5 in their paper that proposed DTE (and variants) as well as the KNN baseline for fair comparison, while the DTE and ANN models across HP configurations perform subpar.
D.7 Sampling time of d-dimensional GMM
sampling time
Figure shows the sampling time of drawing 10,000 points from different GMMs with incrcasing dimensionality {10, 20, .0, 200}. We parallelize the sampling process over 10 CPUs, where each CPU draws 1000 samples.
0.4
We observe that the sampling time grows nonlinearly as the number of dimensions increases, which suggests that it may incur considerable computational overhead to directly draw from the data prior over hundreds of thousands of training steps, motivating the use of our proposed on-the- fly linear transformation T for scalability.
0.2
140
150
16p
280
190
200
Figure 7: Sampling time (in seconds) of 10,000
points from GMMs with varying number of di- mensions.
E Qualitative Analysis on Sample-to-Sampn Attention
We sample 50 inlicrs as context and 100 outliers from a 2-d GMM using the 80th percentile as the labeling threshold, and visualize the top inliers most attended by the 100 outliers based on the average (cross) attention weights over 4 heads from the last layer of FoMo-OD (D = 100), which accurately labeled all the 100 outliers. In Figure g the most frequently attended inliers are close to either the center of a Gaussian (e.g., 1st, 5th) or the criterion (e.g., 3rd, 4th), suggesting FoMo-OD tends to learn decision boundaries that reflect the prior data generation process.
4
2
0
-2
(context) top 5 attended
For each outlier, we compute the sum of L2 distances to its top-5 attended inliers (att), the sum of L2 distances to 5 randomly chosen inliers (rdm), and the sum L2 distances to top-5 inliers with highost likelihood undor the GMM (prob). We perform Wilcoxon signed rank lest between att and rdm (alternative: "less"), att and prob (alternative: "greater") over all the outliers, with p-value of 4.4 x 10-4 and 0.99, ro- spectively, suggesting the distances based on atlention weights
-4
80 th percentilc
-2.5
0.0
2.5
5.0
Figure 8: Top-5 attended inliers (all 50 inliers and only part of the outliers are shown for better visualization).
are significantly less than the random distances, and not significantly greater than the distances to inliers in high probability region.
We visualize the top-5 attended inliers for 3 outliers at different position the 2-d GMM in Figure For a specific outlier, there a similar trend of attending to the center Gaussian (as shown in Figure g). besides, inliers that reflect the criterion boundary or are close the outlier are actively attended (e.g., 3rd, 4th in the left, 1st in the middle, 2nd, 5th in the right), suggesting FoMo-OD is incorporating both boundary and nearest neighbor information dynamically for each outlier.
24

---

Published in Transactions on Machine Learning Research (09/2025)
4
top-5 attended outleers 80 percentie
2
2
2
0
0
-2
-2
-2
x
4
-
4 2
0
2
4
-
0
2
4
0
2
4
Figure 9: Top-5 attended inliers of 3 outliers at different positions of the GMM
F
Ablation Analyses
In this section, we perform varions ablations to study the effect of different design choices in FoMo-0D; namely, maximum pretraining data dimensionality D, the number of routers R on F.2 cost and F.3 performance, F.4 context size (both for training and inference), F.5 number of unique datasets used for pretraining (i.c.. reuse periodicity P), data transformation T during synthesis on F.6 performance and F.7 speed up, F.8
data diversity and prolonged training, F.9 quantile transforming the benchmark datasets preceding inference, and finally, F.10 how different model sizes affect performance.
Unless stated otherwise, most ablation results performed using FoMo-OD with D 20, as it is faster to pretrain under these many varying settings.
F.1 Effect of pretraining dimensionality D
How does FoMo-OD 's generalization performance change by increasing dimensionality of the pretraining data?
We start by comparing FoMo-OD pretrained on datasets with up to = 20 versus = 100 dimensions. Note that learning on higher dimensional datasets is harder, as evident from the relatively larger pretraining loss shown in Appendix Figure 5 While the statement is accurate in general, it is also partly because subspace outliers "hide" better in higher dimensions.
Comparing Table D [D = 100) with Table 2 (D= = 20) w.r.t. p-values over All datasets, we find that FoMo-OD at larger scale does better, where all p-values are larger for D = 100 than D = 20. We find that FoMo-OD with = 20 performs well on datasets with d < 20 (i.c., "on its own game"), however beyond its pretraining setting, e.g. on datasets with d < 50, 100 is superior to D 20 as shown in Appendix Table 13
F.2 Effect of routers cost
What is the running time and memory cost of FoMo-OD with & w/out router-based attention?
Figure left) shows the average inference time per test sample, comparing FoMo-OD using router-based attention mechanism with R = 500 routers (in green) versus FoMo-OD using typical attention without any routers (in blue). As inference context size increases, running time for traditional attention grows quadratically while router mechanism scales linearly
Similarly, memory cost with routers considerably lower when using routers, especially for larger context. sizes, shown in Figure
10 10 middle).
Note that the inference time is reported on CPUs to show scalability. On GPUs, w/ 5K context size, see Appendix Figure T
where typical attention takes advantage of parallelism (6.5ms), while baser- attention is slightly slower (7.7 ms w/ 500
routers) due its two sequential self-attentions; Eq.s and
25

---

Published in Transactions on Machine Learning Research (09/2025)
Inference time/sample (ms)
8000
DTE-4P (tup-]
with (R=500)
Memory Usage (MB)
with (R=500)
P-value (by AUCROC)
0.4
without
without
31
6000
DTE-C top
600
0.3
Lar stop 51
CBLOP (lop-6)
4000
400
0.2
200
2000
0.1
0.0
1k
2k
5k
10k
1k
2k
5k
10k
100
200
300
400
500
size
Context size
routers
Figure 10: FoMo-OD w/ router mechamism saves time and memory while more #routers perform better, offering a cost-performance trade-off: (left) inference-time (ins) per sample and (middle) memory cost (MB) with & w/out routers by varying context size; (right) performance (based on p-value against top baselines, higher is better) vs. number of routers. (setting: D 20, P
F.3 Effect routers on performance
What is the impact of the number R of routers (or representatives) on performance?
Router-based mechanism allows trade-off running time with expressiveness of the attention and hence performance. Figure 10 right)) shows the p-values of the Wilcoxon signed rank test as the number of routers R is increased from 100 to 200 and 500, comparing FoMo-OD to each of the top-6 baselines. We notice that FoMo-OD performance tends to increase monotonically with more routers.
F.4 Effect context size
What is the impact of contect size, both during model pretraining as well as during inference!
study how performance changes by context size, we train FoMo-OD with varying context size (1K,2K,5K) and employ each pretrained model for inference with varying context size in (KKK2,, Table 5 shows the results, where performance is depicted by the average rank of FoMo-OD (the lower, the better).
Table 5: Average rank (based on comparison to 30 baselines w.r.t. AUROC) of FoMo-OD across datasets under different context sizes for training and inference. Smaller ranks imply better performance. (sctting: D = 20, R = 500, P 1)
Infer:1K
Infer:2K
Infer:5K
Infer:10K
Train:1K
13.816
14.623
15.193
15.439
Train:2K
13.079
13.219
13.439
13.561
Train:5K
13.088
13.211
13.307
13.430
We find that training with larger context improves performance at any inference context size. On the other hand, perhaps counter-intuitively FoMo-OD with smaller inference context size does belter. We conjecture that is because the #rontees-to-conneext size ratio increases with larger context size at inference, limiting the expressive power of the "bottleneck" attention mechanism. The pairwise statistical tests among the x 4 = 12 models support these observations, as shown in Figure II Interestingly, when the training context size is large enough 5K, inference with 10K samples generalizes beyond training with no statistical evidence for performance difference (at 0.05) from other inference context sizes.
F.5 Effect of number of unique datasets
How do FoMo-OD performances compare when pretrained unique us. reused datasets, via varying periodicity P?
26

---

Published in Transactions on Machine Learning Research (09/2025)
1K
2K
5K
10K (inference context)
1K
0.00
0.00
0.00
1.00
0.96
0.77
0.66
1.00
1.00
0.96
0.93
2K
1.00
5K
1.00
1.00
1.00
0.8
10K
1.00
0.91
0.94
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
0.00
0.00
0.00
0.00
0.02
0.02
0.02
0.99
0.93
0.77
0.68
1.00
0.23
0.01
0.00
0.00
0.98
0.84
0.02
1.00
0.99
0.99
0.96
0.34
0.03
0.00
0.00
0.98
0.94
0.98
1.00
0.99
1.00
1.00
0.00
0.00
0.16
0.00
0.00
0.00
0.07
0.01
0.01
0.01
0.92
0.51
0.36
0.2
0.04
0.00
0.00
0.00
0.23
0.09
0.01
0.00
0.84
0.49
0.20
0.07
0.00
0.00
0.00
0.32
0.19
0.02
0.00
0.91
0.64
0.80
1K
2K
5K (train context)
Figure 11: p-values of the pairwise Wilcoxon signed rank test between models (larger implies col-method is better than (ow-mehhod) w/ different context sizes for training (IK/2K/5K, Lst/2nd/3en /3rd four grids, in black) and inference LLK/2K/5K/2 overall performance, while smaller inference context is preferable.
every 1st sst/2nd/ 3rd/ 4th grid, in red): Larger training context improves
Table 6: Ablation results on dataset reuse across epochs with varying PE(1, 50, 100} show stable p-values against the top-5 baselines, where there statistical evidence to suggest performance difference between FoMo-OD with D 20 and the top 3rd baseline at 0.05 w.r.t. pairwise Wilcoxon signed rank test comparisons, while it continues to significantly outperform the top 5th baseline (LOF) when d S 50. (setting: D=20, R=500, context sizc=5K, w/out transformation T)
P (#unique datasets: 8K)
P 50 (#unique datasets: 8x50 = 400K)
=
100 (#umique datasets: 8x 100=000KK)
top-5
DTE-NP INN ICL DTE-C LOF
DIE-NP KNN ICL
DTE-C
DTE-NP ENN
ICL
DTE-C
All
0.001
0.019 0.062 0.128 0.460
0.001
0.019
0.089
0.159
0.394
0.001
0.015
0.072
0.121
0.290
d
0.583
0.755
0.943
0.736
0.998
0.572
0.789
0.968
0.616
0.993
0.439
0.678
0.953
0.550
0.972
d
50
0.415
0.750 0.869
0.962
0.999
0.347
0.794
0.893
0.946
0.997
0.293
0.697 0.890
0.924
0.994
Next we study the effect of dataset reuse epoch level (w/out transformation) on performance as presented in Section 3.3 We vary reuse periodicity P in (1,50, 100), and accordingly, incrcase the number of unique datasets used for pretraining across epochs. As shown in Table 6 FoMo-OD (w/ D 20) performs similarly with varying dataset reuse. In fact, it is competitive even with P= remaining no different from the 3rd best baseline (ICL) across All (57) datasets, while significantly outperforming the top 5th (LOF) across (24) datasets with < 20 as well as (38) with d 50.
F.6 Effect of transformation T for synthesis
How do FoMo-OD performances compare when pretrained on datasets with vs. w/out linear transformation?
Sctting P 1, we next study the impact of lincar transformation Table presents the results, where we compare reuse of the same 8K unique datasets across epochs (w/out T), versus transforming these datasets with T at every epoch with different parameters (w/ T). FoMo-OD performance remains stable; no statistical evidence for performance difference from the top 3rd model on All datasets, while significantly outperforming the top 5th across those with d < 20 and 50. This suggests that T can be employed without sacrificing performance to save time during pretraining.
27

---

Published in Transactions on Machine Learning Research (09/2025)
Table 7: Ablation results on performance w/ & w/out linear transformation T show stable p-values against the top-5 baselines, with no statistical evidence for performance difference between FoMo-OD with D 20 and the top 3rd baseline at 0.05 w.r.t. pairwise Wilcoxon signed rank test comparisons. (setting: D = 20, R 500, context size=5K, P 1)
w w/out transformation T
w/ transformation
top-5
DTE-NP kNN ICL DTE-C LOF
DTE-NP
kNN ICL DTE-C
LOF
All
0.001
0.019 0.062 0.128
0.460
0.002
0.015 0.226
0.210
0.280
S 20
0.583
0.755
0.943
0.736
0.998
0.648
0.708
0.988
0.718
0.955
d S 50
0.415
0.750
0.869
0.962
0.999
0.264
0.382
0.971
0.900
0.963
Speed up by T F.7
1300
What is the time saving on data synthesis with linear transfor- mation?
1200
Time (sec)
1100
Figure 12 shows the distribution of pretraining running-time per epoch with and w/out data. transformation. Specifically, we compare (left) gen- erating 8K unique detasets/epoch on-the-fly and (right) first generating 500 unique datasets on-the-fi fly and then transforming each one 15 times using T with different parameters to reach 8K datasets at each epoch.
550
450
Notice that pretraining with T lakes about 450 sec./epoch on average, while without I' it requires 1200 sec./epoch to generate 8K unique datasets and gradient descent across 1000 steps. Different from other ablation results, which are based on the D 20 model, here report the running times for our D = 100 model. Overall, our final FoMo-OD took x25 hours for pre-training (450 sec. x200 epochs). Importantly, this is a one cost that amortizes across many downstream tasks with as low as 7.7 ms inference time per test sample (see Table and Appendix Figure
350
8K 0.5K+T
Figure 12: Runtime/epoch dist.n
over 100 epochs for FoMo-OD (D=100) with (left) P=100, ie. 8K unique datascts/cpoch vs. (right) 0.5K unique+7.55k trans- formed datasets/epoch.
F.8 Effect of data diversity and prolonged training
How does FoMo-OD) OD's 's performance change by increasing pretraining data diversity and number of training epochs?
Originally we have trained FoMo-OD w/ D= 100 using 0.5K unique + 7.5K transformed datasets over 200 epochs. As mentioned earlier, learning in higher dimensions tends to incur larger loss in general but also specifically here, subspace outliers are harder to detect in high dimensions.
Toward reducing the loss further, we resume the pretraining for another 100 epochs. Further, to simplify the tasks and thereby increase data diversity, we also decrease the inlier/outlier labeling percentile threshold from 90% to 80% during on-the-fly data generation in the last 100 epochs. In Figure 13 we present the training loss of FoMo-OD (D = 100) trained with 0.5K unique + 7.5K transformed datasets/ over 200 epochs (90th percentile as labeling threshold) and then 100 additional epochs (80th percentile as the threshold) to show how data diversity and amount affect model performance.
Figure 14 compares FoMo-OD's performance (w/ D 100) to top-5 baselines w.r.t. p-values of the paired Wilcoxon signed rank test on datasets with d < 100, after the first 200 epochs versus after 300 epochs. The increase in all the p-values showcases the benefit of additional training.
F.9 Effect applying quantile transform on benchmark datasets
What is the impact of quantile data transform preceding inference on performance?
28

---

Published in Transactions on Machine Learning Research (09/2025)
Training Loss over 200+100 Epochs
0.35
First 200 epochs
Last 100 epochs
0.30
Training Loss
0.25
0.20
loss:
loss: 0.0674
0.0679
0.15
0.10
0.05
90th percentile
80th percentile
0
50
100
150
200
250
300
Epochs
Figure 13: (best in color) Training loss of FoMo-OD (D = 100) with 0.5K unique + 7.5K transformed datasets/epoch for 200 epochs (in orange), followed with additional 100 epochs of training (in green). For the first 200 epochs train with 90th percentile as the inlier/outlier threshold, which reduce to 80th in the subsequent 100 epochs.
0.8
200 epochs
200+100 epochs
0.6
p-value
0.4
0.2
0.0
DTE-NP KNN ICL DTE-C LOF Vs top-5 baseline
Figure 14: p-values increase with additional 100 epochs pretraining, Le. FoMo-OD w/ D 1000 performs better against top-5 baselines on datasets w/ < 100.
We pretrain FoMo-OD on synthetic datasets from a simple data prior based on GMMs. The real-world benchmark datasets, on the other hand, may exhibit features with distributions different from Gaussians. To close the gap, we apply quantile transform (denoted QT) on the benchmark datasets prior feeding them FoMo-OD for inference, which transforms the features to exhibit a more Gaussian-like probability distribution.
Figure 15 compares the performance of three FoMo-OD w/ D = 100 variants with and w/out QT against the top-5 baselines w.r.t. the p-values of the paired Wilcoxon signed rank test. FoMo-OD tends to perform better as suggested by larger y-values when QT is applied.
F.10 Effect of Model Size
To understand how the performance of FoMo-OD scales with model sizes, we vary the number of transformer layers L 2, 3, where the default is L 4 (see in Model architecture). We present the p-values of
29

---

Published in Transactions on Machine Learning Research (09/2025)
0.6
200+100 & 0.5K+T w/out QT
200+100 & 0.5K+T W/ QT
0.5
200 & 0.5K+T w/out QT
200 & 0.5K+T w/ QT
200 & BK w/out QT
0.4
200 & 8K w/ QT
p-value
0.3
0.2
0.1
0.0
DTE-NP
kNN
ICL
DTE-C
LOF
vs top-5 baseline
Figure 15: y-values increase, i.e. FoMo-OD performance improves, against. top-5 baselines with quant transform (QT) preceding inference, for 3 different settings of FoMo-OD w/ D 100.
Table 8: v-values of the one-sided Wilcoxon signed rank test, comparing FoMo-OD (with D = 100) to top 10 baselines with default hyperparameters (HPs), and top 4avg baselines with avg. performance over varying HPs (denoted w/ avx) over all (57) datasets. FoMo-OD improves (i.e. higher p-values) as number of transformer layers L increases. At L = in-context learning ability appears to be quile limited. (selling: 100, R 500, train train/innennn /infereneeeeeeeee context size=5K, w/ quantile transform, a = 0.05)
#parameters
DTTE-NP
JCL
Post.Dag.
OCSVM
DIE
[CI,
1.343
1.56 10
4.30 x 10
1.25 10
1.31 x.1xx10 10
5.08 x 10
6.46 x 10
1.51 x 10
1.00 x
1.30 10
1.46 10
2.68 10
1.58 x 10
2.13 30
0.506
2.52M
0.006
0,157
0.412
0.557
0,703
0,759
0.806
0.384
1,000
3.700
0016
0.088
0.572
0.579
0.672
0871
0.888
0.662
0.021
0.911
0.085
0.385
0.662
1,000
different model sizes in Table g where a p-value > 0.05 means no statistical evidence to suggest performance difference between FoMo-OD and the compared baseline. We can observe that FoMo-OD not comparable to the top-10 basclines with one layer, and shows improved performance (i.c., p-value inercases and becomes larger than 0.05) as the number of layers increases, which suggests that scaling up the model size might help improve FoMo-OD's performance. On the other hand, we can see that the detection performance didn't 't. increase lot from L 3 to 4, which suggests there exist diminishing returns in scaling the model size, and to further improve performance, one has to consider other methods (c.g., increase prior complexity, more pre-training epochs) besides scaling up only the model size.
G Generalization Analyses
G.1 Generalization to Out-of- Distriiiitiiitionn Synthetic Datasets
We conduct analyses to understand FoMo's ability to generalize on out-of-distribution synthetic G.MM datasets. Besides the in-distribution setting for pro-training (i.e., μ 5], E E (0,5), S 5, d < 100), we consider the following out-of-listtrbuttion settings: (a) mean and covariance significantly out of range, with E [-50,-55 U [5, 50), Σ E [5, 50], denoted as "[p/) | E 50]" (b) number of clusters significantly out of range, denoted as "m E [5, 50]"; (c) number of dimonsions significantly out of range, denoted as "d E [100, 500]"; (d) binary outliers with values either 0 or in one dimension from the sub-dimensions, denoted as "binary"; (e) "all", which combines all the variants above. For each setting, we generate 1000 datasets with random seeds from 0 to 999, where on each dataset, we simulate 1000 test points with an outlier rate of 5% and evaluate FoMo-OD with context length of 5000. We present the results with averaged performance over 1000 datasets for each setting in Table g
30

---

Published in Transactions on Machine Learning Research (09/2025)
Table 9: Average metric score standard dev. over 1000 scods for different out-of-distribution (OOD)
synthetic GMMs. FoMo-OD remains robust against OOD lest datasets as in (a) (d), maintaining similar performance to in-distribution performance (top). Performance is affected more when datasets are OOD
w.r.t. multiple factors combined as in (c).
Datasct
AUROC
AUCPR
F1
in-distribution
98.55 2.73
91.17 13.07
86.74
15.43
(a) OOD w.r.t. [n]] Σ [5)50]
94.79 7.53
80.62 +: 21.19
76.32 + 19.85
(b) OOD w.r.t. m [5, 50
97.69 = 3.59
86.72 15.57
81.20 + 16.23
(c) OOD w.r.t. d E [100, 500]
96.22 = 9.01
86.37 + 23.27
83.23 + 22.08
(d) OOD w.r.t. binary variable
100.00 0.00
100.00 = 0.06
99.97 + 0.34
(c) OOD w.r.t. all combined
85.44 + 16.96
64.17 + 35.07
63.99 + 33.53
We can observe different extents of performance degradation when applying out-of-distribution variations. Compared to other single variations, FoMo-OD seems sulfer more from inflating the mean and covariances, as due to the significant deviation in the parameters of the GMMs, inliers generated under such setting are scemingly "outliers" w/o any reference points. Surprisingly, although FoMo-OD is only trained on continuous data, it can almost perfectly classify binary outliers hidden in one of the sub-dimensions, suggesting FoMo-OD could potentially generalize to discrete data at test time.
However, with all variations added, FoMo-OD becomes less capable compared to one single out-of-distribution variation, although there might exist some signals (e.g., binary labels) in favor of its deeision-makinn makinng process, for which training a powerful model with more comprehensive priors could possibly alleviate the issue.
G.2 Generalization to GuM-off-GMM-Dittbbuutto Real-World Datasets
the ratio of p-value less than 0.05 Is 0.53
the ratio of p-vallue less than 0.05 IS 0.56
25
@ cammaateesss
@WVRrc-A
25-
#W//Tcc-AD
10C=44aattre<-500
Xskhhanto
vinsture--00 500
#testure>500 500
*annthrrois
#comppnentts-
* *arinthyroid
#components--1
Xhenkmiqqe @ I5 Kisloddor
Amus
2
cottsunzee22 2
musk
#components--
#componentss
#components-3
FahhiD 200) rank by AUMOL
SVHN
vcomponentsie
FehooD 2001 mank by AUROC
#componentie4
#componentsigg
#componentsus
thyroid
20nees
3.5
1.5
tctp
#rttp
wavelorm
colcBep Wabuctlliss
vowela
vowals
lonoschere
oocc
+ lonasphere
Xspeech
10-
10
@
* Myypphagrde
Wietter
tymohography
wpbi
Xcaedia
caardoo
mmagic.gone mo
mmgic.gammee
popcttic fbult
Xi4eoottis
*gloss
* odwer
gloss
Y Xwdbc
e Bindssat donors
ppima
ydbcc verrteb 3
Misbdisat
Ypina 0wi
wiitt
Gdhors
vvrrtebrrl
0.0
0.2 Best pvollut selectod from GHWLL Sh/rth Quantile Trarsfomation) (a)
0.4
0.6
0.8
LG
0.0
0.2 Best value selected from GMHIL Siiwlo Quantill Trans"ormation)
0.4
DE
0.8
LO
[b)
Figure 16: Goodness of fit test results for 57 datasets in ADBench. The x-axis is the p-value of a dataset, where a small y-value indicates GMMs are not good fit for the dataset, and the y-axis is the rank of FoMo-OD (D=100) on that dataset (the smaller the better). We plot p-value = 0.05 (vertical) and rank = 15 (horizontal) as references. The left figure (a) is with quantile transformation while figure (b) is without quantile transformation. We use different colors to represent datasets at different dimensions, and use different markers represent different numbers of clusters.
To understand how FoMo-OD generalizes from the pre-training GMM data priors to complex real-world datasets when performing zero-shot OD, we conduct the goodness of fit test Huber-Carol et al. (2012) for datasets in ADBench. We fit GMMs to each real-world dataset Dreat with up to 5 components (as with our pre-training datasets), then sample Dayn from the mest-paramtter-fiteer GMM and perform a two-sample
31

---

Published in Transactions on Machine Learning Research (09/2025)
on Dreal and Dayn: with the null hypothesis that they come from the same distribution. A smaller p-value (s 0.05) of such a test provides evidence toward rejecting the null, which suggests GMM is not a good fit for the datasct (i.c., the pre-training data distribution is different from the test data).
test 10
test) vs. FoMo-OD 's performance rank (the lower the better) among 30 baselines. We report the result both with quantile transformation in Figure 16(a) and without quantile transformation in [6)b). Since the two figures are highly similar, our next analysis will primarily locus on the ligure with quantile transformation to aligu with our model's implementation. We plot the vertical and horizontal lines as p-value = 0.05 and rank = 15. For p-value 2 0.05 and rank < 15, we observe that performance is good on datasets with relatively large p-value where we cannot reject the mull (i.e. GMM is a relatively good fit).. This is where arguably FoMo-OD recalls its data prior distribution and generalizes to datasets similar to those seen during pretraining. We also see, for p-value < 0.05 and rank 2 15, datasets with relatively poor performance where we can reject the null (i.e. GMM is not a good fit)). These can be attributed to falling short in generalization to OOD datasets.
We present the results in Figure 16 depicting the p-value (of the goodness-of
On the other hand, we observe many datasets concentrate on p-value < 0.05 and rank < 15, where p-value is small (GMM not a good fit)) yet the performance is competitive those are the datasets on which Fo is likely to have achieved out-of-distributiin generalization. It remains an open (theoretical) question to understand what (algorithm, if any) FoMo-OD might have learned that generalizes to distribiittinnnnnnnnn datascts. It is also an open (empirical) quest to explore whether more complex data prior, beyond GMMs, could further push the performance up and by how much.
G.3 Generalization to Out of Distribution (OOD) Detection Tasks
We further evaluate FoMo-OD on more complex datasets (e.g., ImageNettlevell. Specifically, we employ OpenOOD Zhang et al. (002)), an out-of-distribution (OOD) detection benchmark, where models are trained on labeled in-distribution datasets, with K known classes, and then evaluated on out-of-disrribution datasets, aiming to detect K + 1, K + 2,... novel classes. Although OOD detection is inherently different from OD, we can construct an OD dataset from OOD datasets, treating all K class samples as inliers and the K K+1,, 11,KK+2 2 OOD samples as outliers. For the in-distribution datascts, we choose ImageNet1K,, which contains 1000 categories of images, and ImageNe(200, a subset of ImageNet1KK 1K containing 200 categories. We further choose SSB-hard, NINCO, iNaturalist, Textures, and OpenImage-O O as the dustr--iiittibuttin datasets, which gives us a total number of x 5 = 10 datasots that are ImageNet-level complex.
Following Han el al. 2022). we create 10 new OD datasets from OpenOOD containing 10,000 samples with 5% outliers, and use the embedding from the last average pooling layer of ResNet188 He et al (2016) as the feature (512) for each sample. Comparing FoMo-OD with the top-4 (on our original testbod) basclines in the order of: DTE-NP, ANN, ICL, DTE-C, we follow Livernoche et al. (024) and report mean (standard dev.) over 5 runs (seed=0/1/2/3/ on each dataset. We present the results with in-distribution datasets being ImageNet200 and ImageNetIK in Table 10 and II
respectively.
Table 10: Average AUROC score + standard dev. over five seeds for in-distribution dataset being Ima-
geNet200. We use blue and green respectively to mark the top-1 and the top-2 method.
dataset
DTE-NP
kNN
ICL.
DTF-C
FoMo-OD
habd
58,03 = 0.00
58.14 + 0.00
60.52
: 0.25
60.74
+ 1.88
58.34 t 1.55
ninco
53.28 = 0.00
54.14 + 0.00
59.56
= 0.63
58.83
t 1.54
55.16 + 2.19
inaturalist
29.38 = 0.00
29.51 + 0.00
35.96
+ 1.10
41.77
+ 2.84
38.85
3.29
textures
59.28 + 0.00
59.91 + 0.00
66.40
+ 0.69
70.33
+ 3.18
59.89 + 2.07
openimageo
52.82 + 0.00
53.79 + 0.00
55.20
+ 0.69
59.09
+ 1.50
54.77 + 1.19
average
50.56
51.10
55.53
58.15
53.40
We further report the p-valuc of the Wilcoxon signed rank test between the basclines and FoMo-OD on the 10 datasets from OpenOOD, as well as on the expanded benchmark combining those 10 with our original
We e-test from http:///wwww...ooooooooooo/o//e/
32

---

Published in Transactions on Machine Learning Research (09/2025)
Table 11: Average AUROC score + standard dev. over five seeds for in-distribution dataset being Ima-
geNettK. We use blue and green respectively to mark the top-1 and the top-2 method.
dataset
DTE-NP
kNN
ICL
DTE-C
FoMo-OD
ssb-hard
55.63 + 0.00
55.94 + 0.00
58.79
+ 1.20
59.17
1.82
56.73 2.65
ninco
48.23 + 0.00
49.10 + 0.00
55.25
+ 0.87
57.60
+ 3.93
52.70 + 2.70
inaturalist
30.24 = 0.00
30.28 + 0.00
35.03 + 1.42
41.96
- 3.13
38.94 = 4.59
textures
54.38 = 0.00
55.43 + 0.00
61.30
+ 0.95
63.
+ 3.72
55.18 + 2.92
openimageo
54.31 0.00
54.91 + 0.00
54.02 + 0.43
58. 71
2.08
56.95
3.89
average
48.56
49.13
52.88
56.11
52.10
ADBench (10+-77) in Table 12 In terms of metric values, FoMo-OD performs 2nd or 3rd best across OOD datasets. p-values show that it significantly outperforms DTE-NP and kNN (p>0.955, such that the p-value < 0.05 for rejecting the null hypothesis and accepting the alternative hypothesis that the "baseline-minuss FoMo-OD" gap is smaller than zero) and is no different from ICL (2nd best after DTE-C). These results demonstrate that FoMo-OD generalizes beyond OD datasets and maintains strong zero-shot OD performance complex, ImageNet-level OOD benchmarks.
Table 12: p-value of the Wilcoxon signed rank test (alternative: "greater") between baselines and FoMo-OD on OpenOOD and combined benchmark on AUROC. A small p-value (S 0.05) means that there is statistical evidence for the alternative hypothesis such that basclines achieve higher metric performance than FoMo-OD.
method
DTE-NP
KNN
ICL
DTE-C
OpenOOD
0.9951
0.1875
0.0009
OpenOOD+ADBench
0.1271
0.3308
0.3153
0.1265
Interestingly, we observe that ICL and DTE-C outperform DTE-NP and ANN on the OpenOOD datasets, whereas on ADBench, DTE-NP and ANN are the top-2 methods outperforming ICL and DTE-C. We hypothesize this is because it is harder for non-parametric methods like DTE-NP and kNN to estimate meaningful decision boundaries in high dimensions (c.g., 512). In contrast, the performance of FoMo-OD is consistently competitive, where the p-values on the combined testbed (OpenOOD+ ADBench) show that FoMo-OD is as competitive as all the top baselines across 67 diverse datasets, while maintaining zero-shot detection ability.
H Performance Profile Plots
To enable comprehensive comparison of different methods, we plot rank (w.r.t. AUROC, lower is better) distribution in Figure 17 and adopt 7 performance profile plots as described in Dolan & Morê (2002). Those plots display the cumulative distribution of the metric which quantifies suboptimality relative to the best-performing method, By computing sorted values along with their cumulative probabilities, we then use the area under each CDF curve as global performance indicator, where larger area signifies superior performance.
Figure 18 Figure 19 and Figure 20 illustrate performance profile plots of FoMo-OD and other baselines across all datasets. The results show that FoMo-OD OD (D=100) ranks at top-5 (w.r.t. AUROC), top-3 (w.r.t. AUPR) and top-1 (w.r.t. F1), respectively, outperforming many baselines.
Morcover, the performance of FoMo-OD (D=100) even better (i.e., ranked within top-2) when tested on datasets with dimensions less than 100. As shown in Figure 21 Figure 22 and Figure 23 the area under the curve of FoMo-OD (D=100) ranks at top-1 (w.r.t. AUROC), top-2 (w.r.t. AUPR) and top-2 (w.r.t. F1), respectively, under this setting.
33

---

Published in Transactions on Machine Learning Research (09/2025)
FD
20))
DTE-NP
KNN
ICL
DTE-C
LOF
CBLOF OF
Feat
SLAD
DDPM
CCSVM
DIE-IG
Forest
MCD
VAB
PCA
HBOS
GANomaly
GOAD
DIF
COPOD
ECOD
DeDD
LODA
DAGMM
DROCC
DTE-NP-aw avr
KNN-avT
ICL-avz
DTE-C-aVT
10
15
20
25
30
Rank
Figure 17: (best color) Rank (w.r.t. AUROC, lower is better) distribution across 57 real-world datasets shown via boxplots for (from top bottom) FoMo-OD in red, all 26 baselines ordered by mean p-value (shallow and deep baselines in green and blue), and top 4 baselines' avg variants. The vertical line depicts the mean, the box shows the 25-75%, bars range 5-95%, and circles show the datasets at the tails.
DTB Aetherrii (Are: 2.54)
TcHo1D=201
OCSVM (Arsa: 2.50)
(Aren 2.53)
CDAD (Aros: 2.45)
JM Aaea 2,64)
LOn (Area: 2.811
iPonet (Krea: 2.571
DTE.C.aa (Araz 2.11
Desgdvoo (Arze: 2.45)
DTE-C (awss 2.63
CSa.F (oww 2,50
NC3 (hima: 2.56)
Gotbems' y (Kows 2.511
L0OA (aww 2-451
POMM Aeea: 801
DTEAG Strea 2.50)
PCA (Arae 2341
OF (Aeaa 2.401
DAGMM1 (Aroa: 2.A21
Klauve (Areas 2.83)
DOPM (Arce: 2.80)
Poon Stroa 2.34)
6000 (Aros 2.41)
DHOCK (hrcal 2.34)
Kolleuur (Amm:
(frm: 2.551
Kank (aucroc)
0.8
Cumulssivv suboptimal
0.0
0.0
2.0
2.5
Figure 18: FoMo-OD ranks in top-5 based on the performance profile plots of all detectors w.r.t. AUROC across all datasets. In the plot, x-axis represents the values performance ratios that compare each method's metric to the best performance achieved, while y-axis displays the cumulative fraction of test datasets for which a method's performance is within the 7 value. We use the area under each CDF curve as a global performance indicator, where larger area significs superior performance.
34

---

Published in Transactions on Machine Learning Research (09/2025)
DTE-MF (eroa 6,317
DIE 10 (Arca 6.25)
SLAD (erca: 6.11)
Forest (erca: 5.05)
5c0D (Araa 5887)
MNN (&rea, 6.29)
<ollavt (Arca 23)
RestureBagping (Area, 6.07)
GANonary (Arco: 5.30
L004 (krea: 5.821
FoMoID- Olea 2.20
(eeen 6.23)
WA (Are:
HanarFiow (rrea 5.92)
cOPOB (Area 5.780
DIE-Mpasr Maee: 1.281
(tnea: 6.221
PCA (tres 6.01)
MCD shows: 9.92)
(toma: 5.771
(tom: 8.27)
DDPN Hrew 6.19)
OTT-Caur frew 6.00)
GOAD (loww:
(tree:
FUMUID=200 201 (Arco: 6.26)
CBLO- (Aroa. 5.14)
DvopSVDD (krea. 5966)
HB05 (Area. 5.90)
DROCC (Aroa. 5.87)
(Area: 6.253
OCSVN (Area: 6.131
Kank CDFS (aucpr)
1.0
Cumulative Tat-suboptimal Datacet Fraction
0.4
2
Figure 19: FoMo-OD ranks al top-3 based on the performance profile plots of all detectors w.r.t. AUPR across all datasets. In the plot, x-axis represents the т values performance ratios that compare each method's metric to the best performance achieved, while y-axis displays the cumulative fraction of test datasets for which method's performance is within the value. We use the under each CDF curve as global performance indicator, where larger signifies superior performance.
0TE4G 5.30
(Ares.
DTE-N (Ame:
(eens: 5.20
DeepSuDB (Aens:
HB0S (Are: 4.901
coPOn 4.63
LOP (Area: 28)
SLAD (Araz: 5.15)
DTB. (Area 4.55)
MCD (Arca. 4.92)
DROCZ (Area: 4.52)
JNN (Arca. 5.25)
#suturdBapging (Arez: 5.120
Rank CDFS (aucfi)
Cumulative In-subaptimll Delaves Fractin
0.0
0.4
0.0
Figure 20: FoMo-OD ranks al top-1 based on the performance profile plots of all detectors w.r.t. F1 across all datasets. In the plot, x-axis represents the I values performarcceeeeee ratios that compare each method's metric to the best performance achieved, while y-axis displays the cumulative fraction of test datasets for which a method's performance within the 7 value. We use the area under each CDF curve as a global performance indicator, where larger area signifies superior performance.
35

---

Published in Transactions on Machine Learning Research (09/2025)
Ro (Aroa 2.65)
KiW 16-00 2.621
Freest (Araa: 2.56)
HB0S (440a: 2.54)
GAliimall (frce: 2.471
RoMocB=20) (Are: 2.651
DTE (hrea: 2.633
OCSVM (érea: 555
FCA JArea: 2,530
6G4D jerea: 2,431
OTE-AP (ures: 2.61)
DEPM (6/ra: 2.601
FestumBaaggg (tees: 2,571
DTE-C-asr (hrre: 2.513
L00A (knen: 2.430
DTE-NI--urr (ten: 2,60
CRROF (towe: A10
VKF (Kre: 2,541
DF JAme: 2,50)
Drepduto (Arm:: 2.43)
<N0 2.64:
ICL (Arew 2 62)
VCD JAre:: 2,50)
TCO0 (Koe:: 2.501
DAGIAN (Aome: 2.421
STE-C (dees:
LOF (Kree: 2.59)
Flanart on (Aema: 2.54)
COPCB (Are: 2.40)
DEOCC (Amm: 2.33)
KL--sT (Area, 2.921
S1AD skrea: 2.55)
Rank CDPS (aucroc) (d less than 100)
1.0
Cumulative tuu-suboptimal Dataset Fraction
0.8
0.6
0.4
0.0
0.0
1.0
1.5
2.0
2.5
Figure 21: FoMo-OD (D=100) ranks at top-1 based on the performance profile plots of all detectors w.r.t. AU- ROC in dalasels with dimensions less than d < 100. In the plot, x-axis represents the values performance ratios that compare each method's metric to the best performance achieved, while y-axis displays the cumulative fraction of test datasets for which mothod's performance is within the value. We use the area under each CDF curve a global performance indicator, where a larger area signifies superior performance.
Do)))-20) (oree: 5.121
Khik-arr 9.23)
SLAD (Aroa: 6.131
ECOD (Arce: 3.241
COPOO (Arez: 5.82
D0000-100010 Arrea 5.1)
KL (A 5.22)
WAB (Arce: 6.02)
H80S (Ares; 5.20)
catiomaly (Arce: 5.81)
OTE farca 6.301
DDM (dee.: 6 213
PCA (Araa 3.2
DesDSVDD (écaa 2.1
LODA (Rrea, 5.771
AW (Area 281
DTE- (6rea: 6.181
FeotureBogging (éroa: 5.591
Hanarr (Ares: 5
bacler (Arca: 5,741
STEail--w (eees 6.25)
LOF (eea:: 5,100
OTE-C-wvT 5,97)
DCAD (hrea: 87)
Dir (Are:: 5,731
ic--ss (Area 5.291
COLOP (Area: 5:141
IPorstt Aere:: 3.25)
MCD (Arez: 5.85)
DROCE (Prea:5.55)
0TE4C (eeas 5.25)
CC3VM (Arsa: 6.13)
Rank CDFS (aucpr) (d less than 100)
1.0
Cumulssive Bu-suboptimal Delaset Fractio
0.6
0.4
0.0
Figure 22: FoMo-OD (D=100) ranks at top-2 based on the performance profile plots of all delectors w.r.l. AUPR datasets with dimensions less than d < 100. In the plot, x-axis represents the r values performance ratios that compare each method's metric the best performance achieved, while y-axis displays the cumulative fraction of test datasets for which method's performance is within the value. We the area under each CDF curve global performance indicator, where larger signifies superior performance.
36

---

Published in Transactions on Machine Learning Research (09/2025)
(Arca:5.07)
CDPS (aucTA) less than
Camulaaive Tau-suboppiiall
0.6
0.4
Tbu
Figure 23: FoMo-OD (D=100) ranks at top-2 based on the performance profile plots of all detectors w.r.t. F1 in datasets with dimensions less than d s 100. In the plot, x-axis represents the values- pertoos-eeroomaneee ratios that compare each method's metric to the best performance achieved, while y-axis displays the cumulative fraction of lest datasets for which a method's performance is within the 7 value. We use the area under each CDF curve as a global performance indicator, where larger area signifies superior performance.
Full Results
respectively show the AUROC, AUPR, and Fl scores of the top-4 baselines, DTEENP, KNN, ICL, and DTE C well their corresponding avg model with the average performance across HPs, as listed in Table
Tables
14.1&
14.2
15.1
15.2 and 16.1
16.2
Tables 17.1 17.2 18.1
18.2 and 19.1 19.2 respectively show the AUROC, AUPR, and F1 scores of all methods across all benchmark datasets. In all these tables, the last four rows show the avg_rank of methods across datasets, and p-values of the Wilcoxon signed rank test comparing FoMo-OD w/ D = 100 with other baselines. The preceding four rows are the same for FoMo-OD w/ D 20, when ranking 31 models (26 baselines avg variants of top-4 baselines + FoMo-OD w/ D 20).
37

---

Published in Transactions on Machine Learning Research (09/2025)
p-value of AUROC among different baseline methods
CBLOF
COPOD
ECOD
Feat.Bag.
HBOS
IForest
0.8
kNN
LODA
LOF
MCD
OCSVM
0.6
PCA
DAGMM
DeepSVDD
DROCC
GOAD
0.4
ICL
PlanarFlow
VAE
GANomaly
SLAD
0.2
DIF
DDPM
DTE-NP
DTE-IG
DTE-C
CBLOF COPOD
ECOD
Feat. Feat.Bag Bag.
HBOS iForest
KNN
LODA
LOF
MCD
OCSVM
PCA DAGMM DeepSVDD
DROCC
GOAD
ICL PlanarFlow
VAE GANomaly
SLAD
DIF DDPM DTE-NP DTE-IG DTE-C
Figure 24: Pairwise v-vuuu among baseline methods based the Wilcoxon signed rank test w.r.t. AUROC
performances across datasets.
DTE-NP-S
AUROC
AUPR
DTE-NP-10
3.000
0.000
0.000
0.000
8.502
0.000
1.000
0.000
0.000
0.000
0.000
1.000
0.000
0.000
0.000
0.000
3,000
0.000
0.00
0.000
0.000
DTE NP-20
0.6
DTE-NP-44
DTE-NP-50
2.000
3.000
2.000
0.000
1.000
1.000
1.000
1.000
0.000
0.000
2.000
3.000
0.000
0 0.0000
3.000
DTE DTE-NP-aVe
5.2
1.000
1.000
0.332
6.000
0.000
1.000
0.000
0.046
0.00
0.000
1.000
1.000
0.015
0.000
0 0.000
DTE-NP-5
DTE-NP-10
DTE-NP-20
DTE-NP-40
DTE-NP-50
DTE-NP-avr
DTE-NP-S
DTE-NP-10 10
DTE-NP-20
DTE-NP-40
DTE-NP-S0
DTE-NP-avT
DTE-NP-5
DTE-NP-20 DTE-NP-10
DTE-NP-AN 40
DTE-NP-SH 50
DTE-NP-avT
Figure 25: values w.r.t. AUROC/AUPR/HI among different HP configurations of DTE-NP (i.e., k E
{5, 10, 20, 40, 50)), along with the avg model with the average performance across HPs.
38

---

Published in Transactions on Machine Learning Research (09/2025)
AUROC
AUPR
KNN-5
0.000
0.000
1.000
0.903
0.009
9.000
0.000
0.009
0.000
6.000
0.000
0.000
0.000
0.000
0.000
KNN-10
1.020
0.000
0.006
-0
KNN-2
1.000
0.595
2.6
-40
KNN
04
KNN-50
1.000
1.000
0.000
020
0000
1.000
1.500
1.000
0.000
1.000
2.000
0.976
KNN avr
1.000
0.599
0.269
0.000
0.003
1.000
1.000
0.003
0.009
0.000
1.000
2.294
0.046
0.000
0.000
KNN-5
KNN-10
KNN-20
KNN-40
KNN-50
KNN-avr
KNN-5
KNN-10
KNN-20
KNN-40
KNN-50
KNN-av
KNN-5
KNN-10
KNN-20
KNN-40
KNN-50
KNN-avr
Figure 26: p-vauues w.r.t. AUROC/AUPR, among different HP configurations of KNN (i.e., k E
(5, 10, 20, 40, 50)), along with the avg model with the average performance across HPs.
ICL-1e-01
AUROC
1.000
3.992
5.938
1.500
ICL-1e-02
0.5
ICL-1e-03
0.000
0.000
0.988
0.093
0.000
0.000
0.000
0.809
0.062
0.009
0.030
0. 0.000
0.359
0.001
0.000
ICL-1e-04
0.000
0.032
0.000
0.000
0.000
0.000
0.101
0.000
0.000
0.000
07070
0.000
0.000
ICL-1e-05
ICL-avr
0000
9.222
1000
1
000
6.675
0.000
6.43
1,000
000
0.322
0.000
0
228
1.00
1.000
0.187
ICI-1e-01
CCL-1e-02
ICL-1e-03
1CL-1e-04
ICL1e-05
ICL-avr
ICL-1e-01
ICL-1e-02
ICL-1e-03
CCL1e-04
ICL-1e-05
ICL-wvr
ICL-le-01
ILL-le022
ICL-1e-03
ICL-l-044
ICL-ee-05
ICL-avr
Figure 27:
valuus W.I.t. CRROOC/AUP//A among different HP configurations of ICL (i.e.,
learning_ rate E (10-1, 1,10 10-2 10-3 10-4 10-5)), along with the avg model with the average performance
across HPs.
AUROC
AUPR
DTE-C-5
DTE-C-10
000
0.000
0.000
0:927
0.000
0.000
0.000
0.000
0.002
0.000
0.002
0.000
DTE-C-20
0.000
0.000
0.990
0.000
0.000
0.830
1,000
0.000
0.000
DTE-C-40
1.000
0.4
DTE-C-50
1.000
1.000
1.003
1.000
3.000
1.000
1.000
3.000
1.000
1.000
1.000
1.000
2.000
1.000
1.000
DTE-C-aur
DTE-C-5
DTE-C-10
DTE-C-20
DTE-C-40
DTE-C-50
avvr
DTE-C-5
DTE-C-10
DTE-C-20
D-E-C-40
DTE-C-50
DTV-C-avr
DTE-C-5
DTE-C-10
D-2-C-20
DTE-C-40
DTE-C-50
DTV.
Figure 28: p-values w.r.t. AUROC/AUPR/2 F1 among different HP configurations of DTE-C (i.c., E
(5, 10, 20, 40, 50)), along with the avg model with the average performance across HPs.
39

---

Published in Transactions on Machine Learning Research (09/2025)
Table 13: Comparison of methods across datasets. (top row) Rank w.r.t. AUROC performance avgled over 57 datascts is presented for FoMo-OD (with D 100), top-10 baselines with default HPs, and top-4 basclines with performance avg. 'd over varying HPs (denoted w/ avg); followed by p-values of the pairwise Wilcoxon signed rank test, comparing FoMo-OD to each baseline (from top to bottom) over All (57) datasets, those (24) w/ d S 20, (38) w/ d 50, (42) w/ d 100 and (46) datasets w/ d < 500 dimensions. FoMo-OD performs as well as (i.c., statistically no different from) the 2nd best model (ANN, w/ 0.106) across All datasets, while is comparable (p > 0.05) or better than (p > 0.95) all baselines over datasets w/ d 100 (aligned w/ pretraining where
100) and 500 (generalizing beyond pretraining).
FD DTE NP
ANN ICL
DTE C LOF CBLOF Feat.Bag. SLAD DDPM OCSVM DTE ANN" ICI. DTE C.ve
Rank(avg)
11.886
7.553
9.018
10.851
11.36
12.316
13.342
13.386
12.982
14.061
13.851
9.079
11.105
12.991
22.263
All
0.016
0.106
0.462
0.454
0.585
0.750
0.823
0.759
0.901
0.895
0.112
0.315
0.670
1.000
d 20
0.128
0.665
0.987
0.727
0.911
0.940
0.987
0.868
0.758
0.968
0.781
0.568
0.990
1.000
d s 50
0.731
0.923
0.992
0.973
0.989
0.987
0.999
0.918
0.985
0.986
0.918
0.967
0.989
1.000
d 100
0.415
0.700
0.949
0.953
0.970
0.971
0.996
0.876
0.980
0.978
0.752
0.560
0.958
1.000
g 200
0.315
0.605
0.923
0.919
0.944
0.977
0.990
0.904
0.970
0.983
0.663
0.789
0.937
1.000
500
0.220
0.894
0.910
0.979
0.607
0.756
1.000
40

---

Published in Transactions on Machine Learning Research (09/200)) (09/2025) )
Table 14.1: Average AUROC + standard dev. over five seeds for the semi-s semi-superre superviseeedd setting of D'EE-N, NP, ANN with varying hyperparameter (HP)
values; & E (5, 10, 20, 40, 50). Also reported is the awg model. We use bold and underline respectively to mark the best and the worst, performance of each model to showcase the variability of performance across different HP settings.
5) 01000000000
00 20
92-30 00
10 02/0000-17
75-0000000
05-0800000
78-7866022
98-7489000
80.22660000
67.0061 10
87.011775
33.2000000
1..0000000
05,010000
1000050000
9000200:00
850725000
95U755000
03 03.001230
0800000000
25. 30.000000
75.020000
98.115022
35 225020000
35 75.7250.00
56
43060000
07-001000
07
S2.000000
07 2REDOO
75.115046
56-4600-2
65 555000000
08 1000
1009
28.30000 (055
04.212011 2017
OK. 91 HUN
17.280011 IS
200021114
0UR-HUN
3030000
METT-00 1000
50801000
20U 1009
AL22-000
20
(800-10))
2000-003
TT. 00
0% 10.024
EING (000
NESE 1003
SUSS--00
(00)
1005 100)
27
2000
52.
05
KNNN-00
51.59 00:07:0:00
00 00
00. 01
06 19-5
90.27
08000000
50.3
0501.00
1000000000
87600.000
89.00000
76 I5
75-22000
96-7--010
99.99.000
30 30.0ruu
56 2025000
410661.22
78.365000
77.473000
9000
55.0-110
4131-600
0653 -0I
8151 .I0
1600-1022
5453-000
(0)
92.00
KNN-200
01 0016000000
60 00100000
02 00-3000000
06
01 1015000000
TA 1U0
923000000
05 1250000
DE DALT665000
1000
89.00
ME 20
2IM
220220000
SE 20
76786000
05 00
08
00215000
07102200
OF 02
DE 0028-5000
052065000
WE 00L0
93.25
MI.
77
KNN--00
51 00-30000000
00 02-0000000
00
02 20
02 08-00000000
12
22.50555112
21
20
ST.7N
10000150000
02:0050000
50.00-00
8200000000
35 15H000
9LMELE
35.67000
47.405 17.08LE 2T
85.3960000 00
25.00100000
35 25.320022
OF PROON0
20
UT ST
55 2011 AT
AL 61.001000
O7 301000
204155045
06. 10H0000
U5.
05
85
5100100
00-54-10000
92.80 HT L5000
99--1 90-77.50000
00:001122
022000000
00
90.73-2000
69.24-000
SM 1M.19-12 124
ST. 17-38-150
80.3150000
08 08.0100.0
20
61002000
BNUT AVO
01 31.005000
51.00550447
ST. 08-0044
ST 0000000
KNN--
90.01
99.51
90
DT DTR-MPaam
00 07.0000
100 04
10 50
38- 79-99-
02. H1U0
AS S1HRU0
72. 1110 1S
50000000
2ESTAL 45
9010000
50 00.0611 AS
20 535A000
30 24
B1 SESEOO
(6. 0050000000
51 3540
38
DTTE DTE-ND-- 30
51. 00-00-110
00:00
50 1000000000
00
36009 15
36 36 00.25
71 81880177777
39 0012255..0
43
510000000
2000
2010000000
60 0010000000
20 2000050000
52 12.000000
34 26201-00
30000000
1L 45
40.000000
76 50:0000
201050000
50.0000
13.00 10/0
SL115300
75
252055000
32 35
23.00
00.
15
00
36
28
51 00400 00
10
02 30 01
01 2000 io
06 01000
05
21.0840.11
20
1541
2800
1000000
2861
50.2-0000
ALSOHLI
4867810 01
75 75.1150.00
26 76.706000
95 02.220005
27
28.800
22
WAS800 20
GL 2U
OT 02
OF. 12
0. 1500
30 W1
05 05525400
O1
05
96
95
05
01 20500:00
00 33000:00
00
00.07 020 66
78-1 80000
213:20034
72-3860-00
7066000
000000000
65
10:00-0000
80.0055000
MI 00
20
820-50000
30 30
4175041
00 00152000
5133-000
07 25//00
8155
M051000
207450.40
80
59
ITE-N1--
51 01/2810000
0000800002
10. 2000000
10 25
2801200-5
98-718099
11,88009
33388136
777220144
05
27
50381000
00
050T1000
4000000
000020000
70
80/00 00
ORLB 00321001
5152H000
402000
71304101
20211005
80011000
01-02000
07 3630000
DERAPS
22.000000
00.70 00.76.10000
00.91 1001190000
0N.-M00 02
55. L08000
05.770 70750000
94,2 22:500000
00
00.00155:00
00 1L
43.004000
W. 10
41.122000
07. W7.88OL SS
79 35
0
09 2F
09 32
26
06 00
0R 00
60 200200 OF
08.811000
0T. 17.8
02.1 125500
50.8 3T
60.00 00-0055677
67 8755000
KT.
aontorodd
bachooor
brvoutte
Internee
vondd
MNNe--C
CIFAR In
SWIN
MTV IV
20turo
vilt
vile
41

---

Published in Transactions on Machine Learning Research (09/2025) )
Table 14.2: Average AUROC 1 standard dev. over five seods for the somi-superviee sctting of ICL and DTE-C basclines with varying hyperparameto (HP) values; For ICL, the learning rate E f0.1, 0.02, 0.001, 0.0001, le - 05), for DTE-C, k E (5, 10, 20, 40, 50). Also reported is the aeg model. We use bold and underline respectively to mark the best and the worst performance of each model to showcase the variability of performance across different.
07-6-667
5017-35
2621-106
54 200
4027-3112
0620-077
5000-58
20.000000
7T:00000
72000:04
50000000
07
$1000000
75.01-0155
0600-0000
06377-226
2.25
4.71
00.00 0005
05-0500119
300--UM
4515-055
52-2--5 5.22
5011-5002
722040042
77 414000
01.92200
41.74-000
712260015
7414-0105
0135-012
57 27 00
73 EZ
26 L00
65.0
DTR----000
200010000
000069000
0000390000
2000000000
000060000
200030000
2000100000
300000000
2000500000
0000
2000 1000
20000 1000
0000
0000000:0
GO 00.0000.000
200050000
DI 00360000
000500000
201100
000610000
DTE-CC46
2001-0000
100-000
0022-046
14
65
2010
200 -00
2000 000
M000 ME.0O1
2001 2001000
T1.20 71.26-250
0031 23,00
10.9 .022
168E
1117-135
DR ORI
07.00:100
5822-18.00
5L00=1 22
(001-293)
30020000
200-000
100-000
0000000
0000000
30 11
-02
2000-02
2000
300
2000
2000
00
07
300
2000
DTTE---0
02.04:000
36061131
60 62
1000-000
17
01
123
1872-122
43002430
0824-1.6
ALM27.00
81002172
29
03.003344
10
90750010
15
0 01.010.0
05102122
97-70
57.
08
30
77.710002
0L.062411
00111201
20
50.00
1000000000
UF 423001
W-51212H
K1.2000
0500011500
01372270
ST.025000
00701033
70.071154
00001003
10:0081.77
50
90
DTB--29
084001901
00.541955
70.02
SESTALRS
6180155
00
00.761900
9301
07.074233
000159:1
02,001155
50 13151019
06722211
634220300
50 25
66101110
08.75991
100
03352011
54.0000.00
100000011000
27
76
15
200EEL11
30
30
30
10021116
3001000
37
2011
72
50.10
MASTORT
05
969T100
25
MASTAUTTS
13
13
02 OS
22 H
02
03
O5
59871220
ALLEMIRR
10000010O
07
100055000
TEAINOR
6700LL12
10002000
084770000
HA. M630H0.0
8236-220
DBOTHAI
0449: 51
0651:120
20
00
27
6 31000
00-20
57.00000
ICL40000
4185-10
04973077
060000
10
0000011000
70362114
70138130
000000005
WATERO
1036
1000
4516-13H
75
70232282
92
31082008
72220
31281035
30321077
375221112
00.00
90.50 AIC 90,90
100
90
99
95
05
07-18-001
13-441101
01:11-126
10000000000
65.002200
1000000000
WOTTII
WUN 1800
7LW-3I
053
07:00:111
IT
242050022
1134-611
01
11
01
09 00
12.22
KAIN
MUDS
78
01
31
200
03. 06.868311
100 000
3002
21 18- 3.
89 21. 300
NT 57.00.977
01. 3:40
22 1830000000
322 90
TV. 20..012220 2.00
15
91. ML700 140
03 40.00-400
23. 01
25. 10.0011800 188
DE 2
02 74
90
22 70
OL 2.
55 12.24400
01 40
M. 21----0001
09 386001
22 21
OT DRALSST
20 HO
100,00
21.2
02.
05.
MT
anstendd
brooutte
pesilight
will
yoat
MISSTTC
CHESSEI
SNON
MERR AD
HP settings.
42

---

Published in Transactions on Machine Learning Research (09/200) (09/2025) )
(HPP values; & E (5, 10, 20, 40, 50). Also reported is the ang model. We use bold and underline respectively to mark the best and the worst, performance of each model to showcase the variability of performance across different HP settings.
Table 15.1: Average AUPR standard dev. over five soeds for the semi- semi-supers superriseed setting DTE-NP, ANN baselines with varying hyperparameter
0.
15.0000000
07 00.000000
090000022
443133900
240220444
780021118
08.30801000
31.915973
79485645
345000000
10.9000000
100005000
13.5000000
07.10650000
82.5450000
07:24400
97381000
92311010
90211526
2,75Ht0
05, (6.1227.00
20.022.1
25.025000
24,220000
61175508
103002000
(7-2040000
15-251020
41.010000
28.201000
19-0E00M
15. R5.R00
25.4850.22
124-40/66
15 400000000
06
05
1128 00
2231
MR T6 1616
NT. 10--N--00
44
58.01000
10U0E0000
20022000
07:0--000
25.00000
50.11.52
S211100)
81
50-000000
1220-202
20071000
D1 52:9222
30000000
25
BL-5-000
4000
TE 1003
1528 100)
ST.0
H0000000
23.30
02L
KNN--0
6.0365000
00
00
08.27 00:27-000
21
09013 17
2000
73
03472300
00000000
66.02-10
1000000000
1766
6641:00
70
80:19-000
97. 47.42
97.2
85.99
2.700004
02/00221
AL00-UM
1500
21373000
21030000
9,0650000
48.0 21
8.77-01
14.77 CIII
43.0 .CII
1927200
15.02000
72
12:15000
14.7-0000
13
96
XNN-20
6.0000
11 10050000
67 0800000
000 0000
10-45-100
10L 0060550000
01925500
41106
1000
211055000
07
OF 00
OT 10
82411000
27-0000
ON 12-22-00
8100500
106722
27 4150000
2107-000
52102550
1011500
MN 36
42 23225
3T ST
15171400
56 50000000
15115000
1200544
153965000
51.
6.034000
11-70.5000
6T 30
37.201135
25-7322244
15-88LL15 L1
44.0055000
15000150000
21.7020000
ST-100000
07
07
95 95-000-30
82202000000
2.2-000
05.899.71
81000000
2. 21.852233
25 25.7220.00
25 25.5240000
0510733
26
10
85 IS
41 AL.2A2
A7 154000
154510000
45 R31000
15:220000
12 12
15 1559400000
KNN-100
53. 63.7365
5,0200000
10:0000000
08-021000
10000
86. 16-70.3 22
22-024000 12.22000000
01
72.72-500
073000000
00.005000
1000
2 UUN
26.07=1.0
M0.150000
27.0022000
302300000
SG. 80.1811 A4
200150000
16.345000
E.. 18,70-0008
10 18-005000
5500
KNN--
18.3
100.000
90.31
08
21
DT DTR-NP-am Sarr
00020000
31 15
06 11000000
01 B8105)
40- S1R100
72000000
SE 2000000
28 38
20 SE20000
26 TRLLDOO
50 ST
31 12
SC. T1E1S
30 AL
ST ACHUOR
1O CSHOO
10 ORAAII
ST. 30850M
IN-GHILS
16. 5054000000
20 5715
50 SGA
DTT- 30
00
24.57 L0
08500.000 0357
10 50445
00.01 F0000
30
020024444
50000000
2000000
HUMISSOO
0.00
210010000
200055000
07.00 0M
35.0100000
2.700000
24
2020224
40
22.000000
21000000
ML1250100
2120-2000
IT M0-00
733015048
22.095566
1550000000
79-33-
TEM
STNN
33
35
22
NY
0-0634000
1.00 00
03 30.0 00
31 K0J00
OF: 02
350240335
316750441
44.8862 10
8102211
01.98600
TM
00030000
7000000
1X00E00
22:0730000
O7 30
91 615210
22
05
2643800
2. 25
27 375520000
246720000
95.4455 00
65 8879655 20
10 00
D1 N..2HL2 2I
AT 36
150010
35
193111000
24
12TTHIN
15,994000
86
15
NYTE--P-30
00
11.00.0000
222260000
278499
1L3455000
10:00-000
202550000
02
U7 02
W.S1 W.3.191
23
3150000
WIRERAI
1L3010000
WOREL
51 01
5754400
15 8010/0
55551000
35571000
16905000
28-31
MT
25
ITTENE--
0.9530000
010000000
06. 75
1718011 25
08100055
77-290099
57. 18-180000
11.8200-00
012222972
55.38
02
30000000
0000055000
30
(2A23000
07313000
7V002211
00
80-220000
2.0-10
17.000000
00
20.12000
10205122
12 360000
01 11041 &
14.301100
10. 00
65.74
5,02
11668.010 00000000
07-481.000
00.901.000 90
08-51 08-57.900 10:36
444229
07
37
00110000
22.2010000
W7-27-3.4
100 00
62. 03-785500
KG. 00
WE 00
BRLT 00
0K. 25
3.12 -U
48.77H5 30
01
00
00
08 08
32
0
09 21
2O 10.00::4 AS
17211000
59.321000
19.775000
18-445000
52 13.081-11
18-4581-1 12
17.005000
O7
20
PlahennkNnSs KNISS
aontorodd
bachooor
brvoutte
vordd
ggandima
CIFAR II
SWIN
MTV IVD
20turo
vilt
vile
43

---

Published in Transactions on Machine Learning Research (09/200) (09/2025) )
Table 15.2: Average AUPR- + standard dev. over five seeds for the semi- semi-supervee supervised setting of ICL and DTE-C basclines with varying hyperparee papperprrmmtte (HP) values; For ICI, the learning rate E {0.1, (0.1,0.02, 0.02, 0. 0.001, 0.0001, le - 05), for DTFF-C, & E (5, 10, 20, 40, 50). Also reported is the avg model. We use bold and underline respectively to mark the best and the worst performance of each model to showcase the variability of performance across different
DTE OTEE-Carr
55 29
5% 0RLIDS
57 1133m
01021029
HURPHUN
280111100
62 062000
2077L1 OZ
71.5562236
4SSRHHUM
73
3.093015
STAELLI LIS
21
CO CE
II ST
31 MEL
ST SUBUII
1820000.77
MTEE-C.
5.91-900
9,55500000
13,51-0000
1,555000
00 22-222-1111
50.000000
12.58.00
4.318000
LORHUIT
3
17.000000
BALION
MLOSILI
ST. (T
3
2USS-192 23
ESLIURI
2LIE LII
5228118
LLIG LO
LL 12
10 0000
9.00000
4.22200
9,4601200
50-00
30
DTE-C-00
5.744006
5,52-m000
$1.251100 18
15310000
08.001:000
00
20.23-0000
37.0 07.0000000
2612 1002
4.3 13000000
M0GE10
INSET
1110000
55000000
20 200
20000504
3,300000
30
HY 04
JAME CSS
OF MOTTMM
50 1.00
1E 10
5000000000
20.33
2000
DTFE-CO C-2
5,228001
11.00-0008
58.37-0088
63.04:111
141
91.0011
10
0021 90
120001
12-401966
27
05.1723000
100000000
A0
MUSELAT
05 10
03
083000
$412-000
2552011
RAMILL
2100011 22
TE 17
ROLL 11
18L10 32
56.31 2006
19-00100 IN
15
AL418122
63.00
85
70.20
05
XTFB-C-0
6,28-101
10:0081111
51.2540031
0. 21
85 8012200
05.77 05.7211 41
29- 90-00000
01-200023
11STR033
17:380132
1X0002000
0800 T5
ME-THOA
02.00531
52. 32.14
81 31302031
3.02-000
W. 50L21
3. AL
77
61
12-6011
47-30
15
10
84,00
15-7000-40
1778-CC
6,7640228
11:01:5488
82 00:00.00
03.50 33.9055:00
5:00
49 01
15.1011000
0588210.3
BELAOIO
0004L1
100.0000000
15
SEGULES
SG
GE-DETT 20
WILL ON
BUTIESSI
10
19002000
05
1001-500
79
08151 08
08100 31
0110 LT
W8LL3 00
47100 34
35.971944 47
1000
3671-7.00
KGLLS
35
30STELS
20
AL
MKOEESSO
SURMELT
SLIRES 40
BESSELT 7I
DE SREES TO
ST
10 12
9L72200 IN
12:005005
SAMLL
52
57
53
88
06
05 05.9442220
20
95 75
13
01.01:00
41 02
20.70A1 20
2LAH22I
02:20-2177
ILAT
06.20
55.00110000
22 RESTS
20 N
81 29
81 09 08
48
61 08E3 70
2130 D
ESGIENALS
23
65
53.
05
1C2-0.0000
6,00H4044
100150005
24
18.0030002
02
03
00011 29
130001 13
20 10.00
47:00-1000
13
BUSEESS
68
0000000000
ST 10
00.02035
50.01120
120
50.3 361
02
12
50 28
20
1931
18:0600
10
01
01
50.34
00.22
ICE 0.000
08:00::0000
12
88 09.991111
55
50
12
1909111 1-73
969--221 2212
103E99
18660000
200200
1000011000
METTHON
600-0000
00220.30
OA OF
MUNEIM DE
19
65407
K5 37:3000
29
10
18.2200
2.01
00.09
05
67
00
50
KCL-0.0
10:08-0000
30.02 5:32
08.1 1-
01
1100
00
10
65.7U111
36
11
02.50
133011 29
1113-223 2238
10:2- 108
0L0-1A1
0008-181
07.02-311
1012-00
MIMLIA
(0000000
5121-25
30
(0000100)
36.00.000
16.0 END
(6.) 1831
1231
2131-200
M00
750019X7
2210-120
3600 5.0
31.7 NIC
2181-100
SUMESA
2201-200
08
06-2-116 1151
10 19
9.91 9.11000
1005
19
17:000041
18.00
RCE-0.1
10
6% 15
S5.01 2- 65
58 00--000 S1
AT 00.00.0 41
68 0N
9.30
62 01 L510
00 17
10:00:10 123
10- 1945
50 28-8
57-8-0
51 LUS5::2
05
17-22-2-11
100 00
MLAE 20
100.000 000000000000
1EMLLSS 3-3 3-36
MURI 2000
15.20-148 143
AL 1L.2-2003 20
6000-1150 150
3. 28
25.25 4.5
$208E 12
1100=
60 27 77
08 28.21:344 6)
39
saulorsid ai
brcantte
compata
cambo
Depastos
Intersetaal
papobooole
ponslight
Hestile
spansaac
spocih
MNSTTC
MVVDDAD
skai
skis
will
HP settings.
44

---

Published in Transactions on Machine Learning Research (09/200) (09/2025) )
(HPP values; & E (5, 10, 20, 40, 50). Also reported is the aug model. We use bold and underline respectively to mark the best and the worst, performance of each model to showcase the variability of performance across different HP settings.
Table 16.1: Average F1 score + standard dev. over five seeds for the semi- supeiviseeeieee setting of DTE-NP, ANN baselines with varying hyperparametor
KNNN wa
0:013000
1010200:00
0R 1800000000
33 22
00-07800-11
05.00000
05.0000000
24
5213000
100810000
1200100
744000000
4.000000
08.000000
10005500
65,0050000
06.240000
93114400
98.11000
93801015
06301330
75001000
05.007.15
71356e000
13,002236
21. 30:000000
27 27.001000
78.7065.1
1,25000
80.022AL
01002222
4.711000
17-301200
41221000
57001000
22.0100M
154001000
05,280075
12/241.0
15, 1cH0000
02
6.3740000
31.000009
3ES0 10
20-02201222
51.21-10
32 (U0
51.90 man
10U0E0000
2,0000.00
36272000
72.00-210
SOW--00 100
0000000 12A
FLIDHU))
30010003
23001000
1000-000
100)
1000
(00)
22-073000
100
1000000
5000211
20
KNN--00
6.6760000
11 20
000
55.22 0000
30 2022 79
51
07
0500
52
66.7
100000000
2.7550000
6474-000
7167-510
6700-000
9395-010
9500
9
0639
2312-100
20000
20
10101223
0.755000
15.00
AUST .M
01.00100
BAS-S
1655000
KNN-20
5
100010000
06 C310000
20
28 71.
01.55 5505
04-800000
21-075555
02 2E.RELL
08
45.00
1000
00750000
06
11.062210
92A00
00
013010000
00001100
330H00
12
12
300000000
27001000
072000000
0016 26
4034-1000
17401000
ST
22285000
15305000
12308 48
18105500
U5
27
3
10.20100000
03 00.00000
09.00 00005
40
04
05.04055 23
20
10000150000
12:0050000
01 50
25 2L/2H000
2600650000
27 27:0010000
78.7365:1
1.56300
20475622
15.0510000
17A0-020
15.2110000
35 AR1000
22.002000
15.000000
05
1568H182
18:000000
KNN--0
28-0666
22
02.
T.
0T.
05.
42
5:9030000
11. 11--00:310
61. 0000000
09-0000-14
15308-0-.5
222-09-5000
ONEL
300
71 11.055000
21-00.2.00
71
MALEHOO
M0L020000
2827-000
28.02-.17
27.002000
50.02-11T
230-00
5591510
50.17-270
SUTSHOO
TRUN 300
46,30 AVO
OT 100
16.11 120
20. 195500
KNN--
52 022503
96T1
DTR-N Sarr
5
10 0010000000
00 3000000000
02 00--01133
06 221013
47 JOHNUG
ON COGR
4T 76666
20 038660
56 53
STH000
FLACHING
SO 28
1.75000
36 11
22 SRITIO
16/00140
10 0050000
DO
31
DTT- 30
0-9730002
11-00-0000 00:00
25. 90 C0
06. 1 220
51 00-ST-111 FUU
30
20,00000
2L
2000
3.5000000
28 0150000
99
21.010
20500000
2201-244
ALM AOOL
NY LO
12M0 45
21000
57
96
96
25
0:9034000
10 2010
05 SOLI
00
05 30
10
08
15
3303000
1000000
7.32000
00
II 1L
24 10.00000
50 00
28 00
07 22
81 BL00S1O
075-000
1077--000
35 80000
22 20.000000
7
12:069147
15
02
99
IN
5:00-000
00.2200000
00 12100
06
86
7700440000
456835
21
0.20-000
10:00-0000
120040000
50011100
0N 1519000
W. 111LO.
2006-130
270010000
51022108
100-00
IF 101000
5551000
30
1000000000
01
ITTENI--
5:013000
10 00
2000000
SE3AL1 87
0U321009
01309909
00-722000
11:828448
00372233
007220000
25. 12277
13
10102230
1.000..0
ALLEHOO
00
000005500
20-2000000
764000
25
M0202000
30
00301000
SORGETT1 IS
231200
0530100
10.5e8 10.52111
20. 00H0000
DERAPS
59030000
10 800000
00.5000000
06-151104
09-722 001722000
22.220334
10..1222-- 12
115000
100 00
00.0015:000
40 38
22 03
11
19100-00
17:11000
50-751000
19. 00.105500
70.8855000
18.05:111 41
200 50H50000
07
04
aontorodd
bachooor
brvoutte
Daad
vondd
vilt
vile
CIFAR
SWIN
MTV IVD
20turo
45

---

Published in Transactions on Machine Learning Research (09/200) (02)) )
Table 16.2: Average Fl score + standard dev. over five seeds for the semi semi-supereee supervised setting of ICL and DTE-C basclines with varying hyperparametes (HP) values; For ICI, the learning rate E {0.1, (0.1,0.02, 0.02, 0. 0,001, 0.0001, 1 le - 05), for DTFF-C, & E {5, 10, 20, 40, 50). Also reported is the avg model. We use bold and underline respectively to mark the best and the worst performance of each model to showcase the variability of performance across different HP settings.
1/TR- X/TE-C-am
3.96-104
2:0130722
3.
42. 12.7021155
92 78
3. 31.0020:44
31.0150000
25.2360-10
07-98-111
24J16277
34571148
0000100000
6.111100
32-0011
0.7812200
42-312724
27.3150036
20020033
25.011136
13 38
06 38
33.015557 76
1111 13
2638000111
13.5410-10
11.7860088
40.4480-44
STE-C-5
10:0000
0000000000
000000000
11534444
00000000
(208500)
0000000
0000000000
000000000
08.0220:0
0731200)
000000
11:01
20
BOD .UI
2210000000
0.0 (000
010081
06
ON.
0N
DTE-CC-0
6.5 1011
00000000
20.1100.00
(00
LOSHOD
10000000
6T.09
3120ER2111
87.0120000
WORLL SS
76.28
210
76
9)
00
0100 (00)
(100
30
000
0U0
STR-C-30
0.24 -22-1110
77 33.354
84 80
02 78
02 182
09 090T33022
01 068000
T30-100
8508226
100000000000
D21
BLACHUTT
00735
90 0326029
328-180
57 02
25 25 0D
19
375361
76
08
79.001110
37
50
DTE-C-10
11.9011:0
6103:2114
00001110 35
02 022461110 O7
0T 0E 63
20 29 52
17 177221 45
17:5-1220 70
0833550000
01
MADESS
MOMERRS
8102-210
1000010000
600L3.85
00
2 01
70111172 22
112221
10002UM
1O0TERTI
00 OF
1098
11:50:000
11:1--255
02 1.00
00:00077
09 29..994
TO 10-181274
17:00/1--2 22
75
11
MILOZAL
2000
100,000000
16.0LLLS
63077663
0611-200
2015-015
3.0000
MUDEASI
113--172
MUSETTI
78.005609
1051-200
94
44
00 22)
07
5000
10.8221222
10-90-215
DTE-
08
10huare
9.1--032
02. 26:329
86 7621 100
05 253025
02 23 23
01 83
35 2823 STT
RAELII
1X00013000
90 M..GLL
05 MALSS
DARLLES
3053002
BE 29
25 54
OF 75
23
05 FEED
12
20
OF
OF
0.
ICT--
07
05-97E511 11
08 40300 51
16 16175
2008113 1.5
0134-22121
827TH1
11
03
MOLSH
MUILEOT
2000-1000
TO IOTTTO
26
MUSSAA
13:000
5
00.1
32.0
ICT-400001
10508-1555 0US5
97 53
01 55
2:1
17
2210-1-72
1370 16.46
67246500
PRO0
2303000
30300000
10000000
200-200
100000000000
11 ON
9891-012
0159-1200
00 OF S0L0
19220
S100 331 M2
AST
S718-155
10ALE150
00
2287-087
300
11
86
57
35
19
3:M
ICE-0000
5,20-1.11 17
9.000034
17.11::000 100
088055 60
5000855022
26
M62 20
1731 109
22.00 32
23.00-447
350055-77
540055311
2966871
10:0-10
07.00 20
30516120
2000
DEGILASS
7430-110
DE MEN
21 31095100
0100
IS
1100
00
JRIGE)
14.750.22
58.05544
12000
51.00
96
58
38
XCT-0.01
1.3
9.76:58834
67.92:5399
11.055--0 57
410T5: 73
1707
35
200
21.0-210
11.3- 1777
9.53-190
92:2-2100
77:0-1001
1120-78
9007-300
1000H00
08.00-20
88
(0002100
0.20 O
6LIN 05
05.0 200
7250-2M
7264-1231
11.1-111 18
7200-5.11
59.007
10 28
29 190
2915-100
5.0
01.0 10.0
9051-5000
12:22-039
14:2-1110
95.07
0.
ICE-0.1
D4 1824
ST 588
00 04
12:50.00 30
40 000091111 28
36 14111110 22
15 02L022-2220
20
22 1.2
13 11.02
01
66-8-200 500
5002-5 70
100.000 000000000000
L0 3.00
DK LM
3500
0200-200 00
76 28
L.G 28
MAM 15
15
ON 30.30-5 1T
20.17 AM1 KM
47 SOEEUS
15.00-00 21
AJGET 7:30
07.00440 100
m1 11
ME30 91
9220000
sanforadd
compaiza
rando
Depastos
Interset
papo
pensiight
Mestile
speci
MVVIOAD
ska
skin
will
46

---

Published in Transactions on Machine Learning Research (09) (09/)025) )
Table 17.1: Average AUROC + standard dev. over five soeds for the somi-supervisee sctting ADBench. Rank of each model among 32 models (26 baselines + 4 ngg variants of top-4 baselines + 2 FoMo-OD variants w/ D = 100 and D = 20) per datasel is provided (in parentheses) (the lower, the mark the top-1 and the top-2 method, Last four rows show avg_ rank of methods across datasets, and values of the Wilcoxon signed rank test comparing FoMo-OD (D= = 100) with other baselines. The previous four rows are the same for FoMo-OD (D = 20), when ranking 31 models (26 baselines + 4 avg variants of top-4 baselines + FoMo- OD w/ D = 20).
ALLIWN 200 WIT
HIGH STINHI
BUSDMON
201
Than
1NTH
1000
12500
11:00
1.
I STHUR BI
NOW
AND
67%
SIORE CONAD DN 3 STESO 3
better). We use blue and green respectively
47

---

Published in Transactions
I on Machine Bearning Research (09) (06/2025) )
Table 17.2: Average AUROC- + standard dev. over five seeds for the semi-supervisey setting on ADBench. Rank of each model per dataset is provided (in parentheses) ) (the lower, the better). We use blue and green respectively to mark the top-1 and the top-2 method.
M MIZUCIIT
IT CE0TUU0
EL RES0S RESOTIN
CHNAAAN 3t a Hood
STEL STE LERD
ONE SE wesh
YOW EEOOOO
ET TELTRI
M ME220022
$4000000
54 MELHUR
$3:00
3.00
3000
1160
3.00
3000
DREAR LA ALAOD
543.08 MON0
15:44
14.22
0.065
0455
11554
0.58
BE STHOIOLL
65 01a00000
03.025
0.000
8.6
000
0.00
10. 10.7
5,05
0.5
0.115
0006
52777
503355
3100
1000
1/00
1100
75.23653 (000
30 303141 19300
12:00
0.
1.000
1.000
20.30
1.000
1.00
1O LOON
NUT LAMOR
AND 178
1,00
1100
1.30
in WESS
51 (0530000)
N: 163500011
IT TR00000
M.S--SSAA INSTETION
DULLIII 21 CIIIAA
0220-100 24 MINDAIN
53 11 MAL-LTI
31 207
3.8
3.00
1.00
1.00
76 2640010000
WISH
WITHR
IT
1.01
1.00
1.001
100
30
20
2
3,000
3.68
1.00
3100
1.00
$110
31.m
1.00
1.00
1.00
1.001
1.00
C7 2000
55 SEEELINO (1NI
ST LENIEE
51.007
1.00
1.00
1.80
1000
0120-110000
BESHIIN TEMM
D1 DELEOANNT SORD
THIS
171
So 14000
SEST
0.91
1.001
1.00
1.001
204 MOD
B2320 FOOD
WIRH (NCO
SSONN OZZI
0051
5137
35.000
1.00
3.00
27
1.00
1107
RESTAN HADH
SONI
192 1690)
6:007
3.00
20:344
3700
1000
BLOING HUSI
IISH
10 OROM
$129.0 4120
55-3950
32.703
60m
3
0164
1/00
17 1023 31
23.5 18 MODR O
NE (7H (00
22.50 PAASI
NOS I
CES01N COR
56 16-100000 ROSH
53 SENEELL
15-00M HOTT
87.02 07-2290003
17:355
0164
1/10
Dusk
Manil
25 mm
500
48

---

Published in Transactions on Machine Learning Research (09/2025) )
Table 18.1: Average AUPR d standard dev. over five seeds for the semi-su comi-supervised setting on ADBench. Rank of each model per dataset is provided (in parentheses) (the lower, the better). We use blue and green respectively to mark the top-1 and the top-2 method.
10-400
BUMEL e70
ILROIGM
TIMM 1100UN00
HILLON
35.00
3,48
7,06
2.06
PUISS RUT
60)
WITh
1.00
1.0
TNE WED
001550000 mm
676
5.90
1000
1000
3370
036
131
1.01
OF UOMINA
2011
13.00
409
SUNCSMAI
12
11:00
5:00
520
1:00
1000 m011
TET
15.
1.00
18:30
6126
ROETH
600
HE
HILL OF MAANDO
IN THAAO
602
631
ALLTH
130
0354
201--
10 6000
MAEE- H000
21
49

---

Published in Transactions on Machine Zearning Research (09) (09/2025) )
Table 18.2: Average AUPR + standard dev. over five seeds for the semi- semi-supervissee setting on ADBench. Rank of each model per dataset is provided (in parentheses) (the lower, the better). We use blue and green respectively to mark the top-1 and the top-2 method.
TLEE 1h to
di ta
STAL ts
2. Th on
mn m.)
2005
1390
1100
STORE LENIINI
THESH DEOS AND
SIEH
11.04
10000
6002
10000
1000
TESIRINN
CSTO AND
1000
6100
35:254
6:218
1.50
(631
RENIE RETH
SESLEN NITH
RED
DID
6.06
39557
6105
13:1
OL 30
912221 (022)
6L29 3000
AITIII DO
DO
CUN TAAI
BBOX
LO
2200
1.38
1ow
2008
199
1. 2050)
33511 1051
DISESRIION
NOL
28 2000
DAGHE
24.99
1:36
24553
188
3.36
sun (ovve)
LDDA
22.00
20.01
1.00
Dep
18520)
130000
14
63. 1120)
D0: 1400)
10.50
1140
1.00
2000
and
MUM
21.00
1.00
22:055
1.00
SEIMISTION TIRSS
RICE
2209)
1000
35333
1148
3100
39700 4030
0000
160K 12000
RUNERAUS
DIF
22.2
20.05
1:3
100
18E40 2003
MOCH
192H
YOU
2ICH
01 201H
113318 2014
014 5025
GOOD
3
1.00
39353
1166
3100
3U17 13520
HIIG
ON
1278:11990 1970
(5.0)
1:00
01
1100
1.00
167774 THA
CEA ROM
1000
15300
1000
55,30
1160
1100
10 -1LUM20
171120331
88111111111
10030
19.00
3.34
35:00
199
3.90
K.K 31
4.7 ()
2 1
S. 0000
POA
D1066
1:311
6900
13.37
11
1990
2.24 702777
8.0.R00 4)
MEE (1851
3018 41 (033)
2 1--003
S1 (115)
8-17 12.00
31
00:000
5:00
15.000
699
A0
catt
Cuik
Mered
080
50

---

Published in Transactions on Machine Learning Research 2025)
Table 19.1: Average F1 score = standard dov. over five seeds for the semi semi-supery sumi-supervisod setting on ADBench. Rank of each model per datasct is provided (in parentheses) (the lower, the beliter). We use blue and green respectively to mark the top- and the top-2 method.
1AM
15:00
1000
1/00
1000
M
WALL DID
15:00
BOL REHERINN BLOUE IO PUSUIDE
12:000
3.000 2,00 0110
30,00
21311
0312
ANEES SIRRI
SESET 6000
SE SEMURSI
SNGLES MU0
1.3
LIONE AND
13.30
1.40
17:01
I EIECO
11:60
2L THE
IS ONT
OITLE WALT
ALO TIE
TAY
X 0.33 Tul:
11:30
694
THE
IE
14
1172
TRATAA AND
THE
1228.4 DUEE
51

---

Published in Transactions on Machine searning Research (09) (025)
Table 19.2: Average F1 score + standard dev. over five seeds for the semi- semi-supery siiis-speeeioo setting on ADBench. Rank of cach model per dataset is provided (in herentheses) (the lower, the better). We use blue and green respectively to mark the top-1 and the top-2 method.
120 1000
2112 LAGE
G. 14020
26780009000 900))
CALTII TEDE
HISH LAAIL
2018 57830
setton
SUN 43442
625 100000000
G.
1.00
1100
1/00
DO DEG
6230
0.41
12.25
6332
RESSUN RESSURNNT 30TT
BUST
157945
13.053
6.22
0:03
13
0:07
TURTINNG
WEDE SIDE
It--1 14))
Aumb
11 1ES
IT 10
22 HE BURD
HUTEH 3T (920
BEE
100
5.00
1,00
38.255 4221
TE.EL S. MTTH
12 S/MI
IT STSITTSEE (E
1.00
34.44
1.00
1.80
13.2 (I0
SUIC T9
OUTTL DIED
INIM
(000)
601110011
$1.00
1000
1200
1200
517 100000
ST
409
16TH caati
1500
1,00
1,00
16:03
1,00
N1 (820
453551101
1/00
5.00
5.00
1700
11-488722
1.00
21475
1901
100
$1.00
1000
1100
1260
8.5 1000
4
13400
0040
3200
10:10
1100
2220
LEST SIL
1520
1.00
10.50
1.31
190
38.5509 1520
RUS
18.31
1000
55.54
1.00
1200
36
2002)
1700
1000
1200
50.11
1100
21
0164600111
Th 41100
4000-0
USIT (NNE6))
21S 63300
HRON SO0E
W1TH (NAO
PCA
1.00
1/66
12
14332 41166
27 (155
(ESSATIO
DS
500
2,00
3,00
1000
Raskon))
catta
455 56
AT
52

---

Published in Transactions on Machine Learning Research (09/2025)
J Benchmark OD Datasets
Table 20: Description of all datasets in ADBench Livernoche et al. (2024). Datasets in blue are image and text datasets that are vectorized through pretrained encoders. We refer to the original paper for details.
Dataset Name
# Samples
# Features
# Anomaly
% Anomaly
Category
ALOI
49534
27
1508
3.04
Image
annthyroid
7200
534
7.42
Healtheare
backdoor
96329
196
2029
2.44
Network
breastw
683
239
31.98
Healthcare
compaign
41188
4640
11.27
Fitunee
cardio
1831
21
176
9.61
Healtheare
Cardiotooography
11
21
400
22.04
Healtheare
celebe
202590
39
4547
2.21
Image
200285
500
18568
6.20
Socinlogyy
cover
2860488
10
2747
0.96
Botany
donors
619326
10
36710
5.93
Sociology
fault
19:1
27
31.67
Plysical
frund
2848877
29
4192
0.17
Fimance
glass
214
9
4.21
Forensic
Hepatitis
80
19
13
16:25
Healtheare
http
567498
2211
0.39
Web
Arls
1966
1555
368
18.72
Image
Tonosphere
351
32
126
35.90
Oryetognosy
landsaat
6435
1333
20.71
Astronantics
letter
1600
32
100
6.25
Image
Lyuphograply
148
18
6
4.06
Healthacare
magic gamms
19020
10
0B88
35.16
Plysical
mammography
11188
Healthacare
mnist
7003
100
700
9.21
Image
musk
3062
166
97
3.17
Chemistry
optdigits
5216
64
2.88
Image
P PlgeBlockks
5393
10
510
9.46
Docmmnet
pendigits
6870
16
156
2.27
Image
Pina
768
268
34.90
Healthacare
satellite
6435
2036
31.64
Astronantics
satimage-2
5803
36
1.22
Astronautics
shuttle
49097
3511
7.15
Astronantics
skin
215037
50859
20.75
Image
smip
95155
30
0.03
Web
SpamBase
4207
57
1679
39.91
Document
spooch
3686
400
1.65
Linguisties
Stamps
340
9.12
Document
thyroid
3772
2.47
Healtheare
vertebral
240
30
12.50
Biology
vowels
1456
12
3.43
Liuguisties
Waveform
3143
100
2.90
Physice
WBC
223
10
4.48
Healthccaee
WDBC
367
30
10
2.72
Healthcare
Wil:
4819
257
5.33
Botany
wine
129
10
7.75
Chemistry
WPBC
198
33
47
23.74
Healtheare
yrast
1484
507
34.16
Biology
CIFARIO
5263
512
263
5.00
Image
InshionMISST
6315
512
315
5.00
Image
MNIST-C
10000
512
500
5.00
Image
MV
5354
512
1258
23.50
Image
SVHN
3208
512
Image
Aguews
100000
768
500
5.00
NLP
Amazon
10000
768
500
5.00
NLP
Inddb
10000
765
500
5.00
NLP
Yelp
10000
765
500
NLP
20newsgroupe
11905
768
591
4.96
NLP
53

---

Published in Transactions on Machine Learning Research (09/2025)
K Differences to Prior Work on PFNs for Tabular Data
There cxist applications of PFNs (originally developed by Müller et al. 2022 that pro-date our proposed
FoMo-OD, namely, TabPFN (Hollmann et al., 2023 for supervised classification, LC-PFN (Adriaensen et al. 2024) for learning curve extrapolation, PFN4BO (Müller et al. 2023 for Bayesian optimization, and ForecastPFN (Dooley et al.) 2023) for time series forecasting.
Here we highlight the differences of our proposed FoMo-OD from these existing PFNs.
1. First PFN4OD: We employ prior-data fitted nelworks (PFNs) for outlier detection (OD) for the first time.
2. First large-scale pretrained OD model: FoMo-OD is the first model for zcro-shot OD that is pretrained at large scale on large collection of (symthetic) datasels, due to the minuscule nature of existing real-world OD benchmark datasets.
3. New data prior: Thanks to PFN's reliance on synthetically generated datasets, we establish a new data prior for OD, specifically for outlier synthesis.
4. Data transformation for scale: While drawing samples from a data prior may be relatively fast, pretraining a large foundation model requires many such draws for every step of each epoch. To speed up data synthesis on-the-fly, we are the first to leverage a linear transformation.
5. Router-based attention for scale: PFNs ingest the entire training dataset as context for in-context. learning at inference time. To accommodate larger datasets at both training (for better generalization) and inference (for large-scale real-world datasets), we leveraged a "bottleneck" architecture for scalable self-altention, and in turn, larger context size.
L Discussion
Summary: We introduced FoMo-OD, the first foundation model for outlier detection (OD) on tabular data. FoMo-OD a prior-data fitted network (PFN), pretrained on large number of synthetic datasets generated from a new data prior for OD, which can infer the posterior predictive distribution for test points in new dataset in zero-shot fashion where the training data is input as context, capitalizing in-contezt learnéng.
Zero-shot OD implics no additional OD model training or model selection, given new OD task. That is a revolution for OD (!), for which algorithm and hyperparameter selection are notoriously-hard without any labeled data, and also computationally taxing especially for today's modern deep OD models with numerous parameters and long list of hyperparameters. What more, FoMo-OD provides extremely fast inference thanks to a mere single forward pass, making it amenable for OD on data streams.
Building on the PFN paradigm (Mülee et al. 2022 FoMo-OD breaks new ground not only conceptually by abolishing the burden of model training and selection, but also empirically: Against 26 different (both classical and modern) baselines on 57 public benchmark datasets from diverse domains, FoMo-OD performs on par with the top 2nd basclinc, while significantly outperforming the majority of the basclines. Without the need to train any, let alone multiple models for HP tuning, FoMo-OD takes a mere 7.7 ms per test sample for inference only.
Limitations and Future Directions: FoMo-OD employs simple straightforward data prior based on GMMs. While it is remarkable lo see how far one can go with synthelic data from such simple prior, future work can design more comprehensive data priors, inclusive of discrete features as well as other possible outlier types. We have also pretrained FoMo-OD OD solely on synthetic datasets, while future work can augment both synthetic and roal-world datasets for pretraining.
Besides the lack of massive real-world datasets for tabular OD, a motivation for data prior to pretrain purely on synthetic datasets comes from neural scaling laws (Kaplan et al. 2020 Zhai et al. 2022). Interestingly, the scaling laws for large Transformer models have shown that their generalization error tends to drop as power law with the amount of training data (also, with number of parameters and amount of compute), but
54

---

Published in Transactions on Machine Learning Research (09/2025)
the power law exponent is very small-suggesting that acquiring more colossal real-world datasets would be a slow, if not expensive approach to advancing MT./AT. Others have proposed ways to subset-select smaller, non-redundant "foundation datasets" (Sorscher ct al. 2022 Paul ct al., 2021), and emphasized the importance of lask/ bask/dataset diversity in pretraining (Raventós et al. 2024p. Arguably, synthetic data from a complex and diverse data prior potential gateway obtaining non-redundant and diverse datasets for pretraining large foundation models like FoMo-OD. On the other hand, designing such data prior requires level of Comain/prior knowledge.
Another improvement could be scaling up to even larger context (i.e. dataset) size and dimensionality, While FoMo-OD generalizes beyond pretrained context sizes and dimensionality, it is limited to and performs particularly well on downstream datasets of similar nature as our experiments showed. A promising direction for size generalization is using PFNs as extremely fast ensemble components at inference; since " *PFNs are quick enough be used as ensemble members. The size constraints could therefore be overcome by boosting and bagging techniques" (Nagler 2023
Further, our work focused on unsupervised OD with clean/inlier-owe training data. Future work can study the unsupervised OD setting and pretraining with mixed/ "cottmm"nnttminatee"" data in the transductive setting, where the unlabeled lest data is the same as training data. In addition, we performed oflline evaluation of FoMo-OD on static datasets, while its fast inference lends itself to streaming OD, which future work can explore. Technically, both extensions (unsupervised OD and strcaming OD) are straightforward from the implementation perspective.
Our current work is limited to OD for tabular (or point-cloud) data. Our ideas can be extended to other data modalities, such as image, graph, and text outliers, to comprisc other domains with critical OD applications such as video surveillance, fraud detection and LLM hallucination detection. To that end, the design of novel inlier/outlics priors would be an open direction. A promising approach here could be the use of pretrained generative models to draw syuthesized of manually-designed data priors.
datasets for pretraining the PFN, in place
Finally, our quest here has been mainly experimental. Theoretically understanding why these models work well as they do and investigating their failure cases are important yet open questions. As empirical future work, one could systematically stress-test FoMo-OD using synthetically generated test datasets that contain known outlier types and inlier distributions distinct from those used during pretraining, while varying the degree of ont-of-distributiin characteristies in a controlled manner.
As the first foundation model for OD, FoMo-OD inspires many promising directions for future research that could lead fruition for additional practical applications.
M Reproducibility Statement
We expect that the disruptive nature of FoMo-OD will trigger future innovations in the OD litcrature, as well as a widespread adoption by practitioners thanks to its key desirable properties. To foster future research and accessibility in practice, we make all resources (our codebase used for prior data synthesis, data transformation, and pretraining as well as our pretrained model checkpoints) publicly available at httpe///atttuucccc///cccacttttte
Full implementation details are provided in Appendix C
55


Reviews:
[{"review_id": "GySVqComO3", "reviewer": "Reviewer_uvaG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Main Strengths:\n\n1. Although foundation models are widely spread nowadays, the field of unsupervised anomaly detection still lacks of pretrained models for zero shot inference; This paper fills the gap;\n\n2. ADBench is well recognized by the community as a thorough repository for the evaluation of unsupervised outlier detection models, as it is comprehensive of a large variety of OD models and benchmarks;\n\n3. The paper is clearly written. Especially, I appreciate the redundancy of stating multiple times that only synthetic datasets are used for training, and only benchmarks are used for testing. Being very specific for the experiments is helpful to understand how to reproduce the results.\n\n4. Related work is sufficiently included, and recent papers are used for comparison in the experiments and in the story of the paper.\n\n\n# Main Weaknesses\n\n1. Although the related work at the end of the main paper describes that unsupervised and semi-supervised (in the sense of one-class) are two different settings, the paper confuses them and creates ambiguity of which of the two is used. For example, the abstract uses unsupervised, while the proposed method is semi-supervised as it requires normal training samples to be used for inference. Also, it is unclear how the compared models are trained (unsupervised or semi-supervised). This aspect is important and has to be cleared.\n\n2. The difficulty of outlier/anomaly detection, as opposed to traditional binary classification tasks, is that the outlier class might not follow a specific distribution (or, equivalently but more technical, the training and test outlier class might have different distributions). Because of this, anomalies may be categorized in different types and might behave differently according to the specific scenario (see e.g. the Chandola survey on outlier detection). This makes it hard for OD models to be better than all the others over all the benchmarks, as one model is usually great at detecting a specific type of outlier - the one that follows the intuition behind the OD model. For this reason, the foundation model might work because the generated training datasets cover \"all types of anomalies\" present in the benchmarks, which is definitely not exhaustive for real-world settings. \n\n3. The paper introduces the foundation model for tabular data, but the evaluation is inconsistent: ADBench does not only have tabular data, but also images and NLP datasets where the authors extract features using resnet-18. These do not conform to the definition of tabular, which (please let me know if you think of it differently) includes interpretable features and not embeddings. Otherwise, any type of data with the appropriate feature extractor is tabular.\n\n4. The evaluation metrics used are the AUC, AUPR and F1 score as stated in the experimental section. However, the visuals in the main paper only report the rankings for the AUC (and the p-values), which makes me wonder why the other metrics are even included. Also, rankings and actual AUC values provide different information: the former how often baselines win against each other, the latter by how much they win. I feel that both pieces of information should go into the main paper, to give the correct overview to the readers.\n\n5. The selling point of the experiments is, as far as I understand, that FoMo-0D is \"not that bad\", rather than being the best or being sufficiently complementary to the others. For example, table 2 claims that it \"shows no statistically significant difference from the top 3rd baseline\", which means it is statistically worse than the first two. This way to phrase the comparison tends to put too much emphasis on the proposed method's good properties, while ignoring completely the main limitations.\n\n6. Although the proposed method looks interesting and has some value, methods like kNN and iForest have very little training time, are often interpretable, and are not stat sig different from the proposed one. Thus, the question is: why should one adopt FoMo-0D for real-world tasks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the challenge of learning a foundation anomaly/outlier detector for tabular data. Because unsupervised outlier detectors strongly depend on their hyperparameter configurations, learning a single model that is able to detect outliers on a variety of different scenarios is hard. This paper solves this problem by using PFN to approximate the posterior bayesian distribution, which is trained on synthetic data (no real-world), and only requires the normal training samples and a test example to compute predictions at test time. Following the benchmark ADBench, the paper compares the proposed approach to 27 OD models on 57 benchmark datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "The main critical point concerns the text:\n\n1. The confusing unsupervised/semi-supervised settings;\n\n2. The phrasing of the experimental results;\n\nIn addition, I would expect to have more detailed and clearer evaluation in the experiments, by removing the image/NLP datasets and including the AUC values.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BspI3KRZK0", "reviewer": "Reviewer_w67h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. Model trained entirely with synthesized data that are easy to sample.\n2. Forward pass only in test time, no retraining.\n3. Scalable to large models with the help of \"router mechanism\" in previous work\n4. Demonstrated good performance in real world datasets, with extensive experiments carried out.\n5. Code publicly available.\n\nCons:\n1. Computational cost. Detecting each outlier $x$ with $q_\\theta(y\\mid x,D)$ that uses the entire dataset $D$ as input could be really expensive, especially for high dimensional data.\n2. The reasons that synthesized training data can be effective is not explored.\n3. The model size (No. of trainable parameters) should be somewhat compared. Since the proposed model introduces different mechanisms, a comparison of model size should be provided to help analyze the model performance.\n4. [minor] Explanation of Fig 3. is a bit unclear, if it is showing rand distribution, some methods will have non-zero density at rank 1 right? Why does the figure showing non?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript propose a novel Zero-Shot Outlier Detection model. The model utilizes the Prior-data Fitted Networks (FPN) to model the posterior distribution $p(y\\mid x,D)$ with Neural Network $q_\\theta(y\\mid x, D)$ for binary classification to separate inliers and outliers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think this manuscript propose an interesting OD approach, it would be better if the author could:\n1. Provide model size comparison table in the revision.\n2. Provide a rough analysis of why synthesized training data can be effective, as this journal expect the accepted papers to be convincing.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wZRfT930Lv", "reviewer": "Reviewer_oXoC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Claimed Zero-Shot Capability. \n- The model demonstrates rapid inference times (~7.7 ms per sample), making it suitable for real-time applications.\n\nWeaknesses:\n\nFoMo-0D is tailored exclusively for tabular datasets. While this domain is vast, many existing methods already achieve high performance due to the relatively low complexity of such data. For instance, traditional models like Isolation Forests [2] and Local Outlier Factor [3] have been effective in various tabular OD tasks. The paper does not sufficiently justify the need for a foundation model in this context, especially when simpler models perform adequately.\n\nThe term foundation model typically refers to models trained on broad data distributions that can generalize across multiple tasks and domains, often learning semantically rich embeddings, as demonstrated by GPT for text [5] and CLIP for vision-language tasks [4]. FoMo-0D, trained solely on synthetic GMM-based data, lacks evidence of such broad generalization or semantic understanding. The paper should clarify how FoMo-0D meets the criteria of a foundation model beyond its zero-shot capability. While it is fair to call the model general-purpose within the scope of tabular OD, calling it a foundation model implies stronger and more widely applicable representation learning, which is not clearly demonstrated.\n\nMore broadly, foundation models are typically capable of supporting a wide range of downstream tasks. In contrast, FoMo-0D is limited to outlier detection, albeit across multiple datasets. If the authors intend to argue for its status as a foundation model, they should either explore applications beyond OD or make this limitation explicit and discuss its implications.\n\nThe model’s zero-shot capability is an important claim, but the explanation for how it is achieved and validated remains underdeveloped. In other fields, zero-shot OD frequently involves the use of semantic priors or auxiliary information to generalize to unseen anomaly types [6, 7]. FoMo-0D’s synthetic-data-driven generalization approach is novel, but the paper does not convincingly show that the synthetic training sufficiently covers the variability and complexity of real-world anomalies. The authors could strengthen their position by discussing generalization challenges more openly, as touched upon briefly in Appendix G, and by presenting FoMo-0D’s performance on datasets with anomaly structures that differ significantly from the GMM-generated training tasks.\n\nFurthermore, it remains unclear whether averaging results across 57 datasets is appropriate or informative. Different OD applications, such as fraud detection, manufacturing defect identification, or medical diagnostics, differ substantially in their tolerance for false positives and false negatives. Each domain has distinct requirements for sensitivity and specificity, and the model’s ability to accommodate such domain-specific trade-offs is not addressed. Without tuning or adaptation mechanisms, it’s uncertain whether FoMo-0D’s one-size-fits-all strategy is viable for high-stakes use cases in finance, security, or healthcare.\n\nThe writing style of the manuscript tends toward verbosity, and the frequent use of em dashes (—) makes the paper harder to follow and could be an indicator for extensive use of writing support by an LLM like ChatGPT. A more concise and direct presentation would significantly improve clarity and accessibility.\n\nReferences:\n\n1.\tShen, Y., Wen, H., & Akoglu, L. (2024). Zero-shot Outlier Detection via Prior-data Fitted Networks: Model Selection Bygone!. arXiv preprint arXiv:2409.05672.\n\n2.\tLiu, F. T., Ting, K. M., & Zhou, Z. H. (2008). Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining (pp. 413-422). IEEE.\n\n3.\tBreunig, M. M., Kriegel, H. P., Ng, R. T., & Sander, J. (2000). LOF: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data (pp. 93-104).\n\n4.\tRadford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (pp. 8748-8763).\n\n5.\tBrown, T. B., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, 33, 1877-1901.\n\n6.\tHendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136.\n\n7.\tEsmaeilpour, S., Liu, B., Robertson, E., & Shu, L. (2021). Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP. arXiv preprint arXiv:2109.02748. AAAI'22", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a model designed to perform outlier detection on tabular datasets without the need for dataset-specific training or hyperparameter tuning. The model, FoMo-0D, is based on the PFN framework, which enables it to compute a posterior predictive distribution conditioned on any new training dataset. To achieve generalization across tasks, the model is trained on millions of synthetically generated outlier detection problems, each constructed using GMMs to mimic diverse data characteristics.\n\nThe authors position FoMo-0D as a zero-shot outlier detection system, arguing that it can be directly applied to new datasets without model selection or retraining. They refer to it as a foundation model”for OD, citing its ability to generalize across a wide variety of tabular OD tasks. The model is lightweight (around 2 million parameters) and optimized for fast inference, making it practical for real-time or resource-constrained applications.\n\nEvaluation is performed on 57 real-world datasets from the ODDS benchmark suite, where FoMo-0D is compared against 26 existing OD baselines, including classical methods such as Isolation Forest and newer neural approaches. The results suggest that FoMo-0D is competitive with, or outperforms, these methods on most tasks, despite having never seen real-world data during training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should clarify and justify their use of the term foundation model. As commonly understood, foundation models support multiple downstream tasks and learn semantically rich, transferable representations. FoMo-0D, while general across tabular OD tasks, is limited to a single task trained on synthetic data. Either stronger evidence of broad generalization should be provided, or the terminology should be revised.\n\nThe explanation of zero-shot capabilities requires expansion. In related domains, zero-shot detection relies on semantic priors or auxiliary information. Here, the synthetic GMM training process is not clearly shown to generalize to real-world anomalies. The discussion in Appendix G should be brought forward and elaborated in the main paper.\n\nThe evaluation strategy across 57 datasets may obscure domain-specific needs. OD applications differ widely in their required sensitivity and specificity. The paper should discuss how FoMo-0D performs under these different constraints and whether it can be calibrated for critical use cases.\n\nAdditionally, the writing style could be more concise. The heavy use of em dashes and verbose phrasing hinders readability and gives the impression of (too much) LLM-assisted writing. A clearer presentation would improve the paper’s accessibility.\n\nThe limitations of relying entirely on synthetic data should be discussed more openly. Not all real-world anomaly types may be covered by GMM-based training, and this may constrain generalization.", "broader_impact_concerns": "The paper does not currently include a Broader Impact Statement\nWhile FoMo-0D offers promising generalization without dataset-specific tuning, this strength may also pose risks in high-stakes domains such as healthcare, finance, or security, where overly confident deployment without proper validation could lead to harmful false positives or negatives.\nLack of domain-specific calibration or uncertainty quantification may create a false sense of reliability. In real-world settings, outlier detection often supports decision-making in sensitive applications, and there are social and ethical costs of incorrect detections.\nThe model is trained entirely on synthetic data, biases or blind spots inherent in the synthetic task distribution may propagate into downstream use.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)