Forte : Finding Outliers with Representation Typicality Estimation
Authors: Debargha Ganguly
Warren Richard Morningstar
Andrew Seohwan Yu
Vipin Chaudhary
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Models
Out-of-Distribution Detection (OOD)
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Generative models can now produce photorealistic synthetic data which is virtually indistinguishable from the real data used to train it. This is a significant evolution over previous models which could produce reasonable facsimiles of the training data, but ones which could be visually distinguished from the training data by human evaluation. Recent work on OOD detection has raised doubts that generative model likelihoods are optimal OOD detectors due to issues involving likelihood misestimation, entropy in the generative process, and typicality. We speculate that generative OOD detectors also failed because their models focused on the pixels rather than the semantic content of the data, leading to failures in near-OOD cases where the pixels may be similar but the information content is significantly different. We hypothesize that estimating typical sets using self-supervised learners leads to better OOD detectors. We introduce a novel approach that leverages representation learning, and informative summary statistics based on manifold estimation, to address all of the aforementioned issues. Our method outperforms other unsupervised approaches and achieves state-of-the art performance on well-established challenging benchmarks, and new synthetic data detection tasks.

Published as a conference paper at ICLR 2025
FORTE: : FINDING OUTLIERS WITH REPRESENTATION TYPICALITY ESTIMATION
Debargha Ganguly Warren Morningstar', Andrew Yu', Vipin Chaudhary' CCase Western Reserve University, Cleveland, OH, USA
?Google Research, Mountain View, CA, USA (debargha, asy51, vcaae)
@case. edu, wmorningegoogle.com
ABSTRACT
Generative models can now produce photorealistic synthetic data which is virtually indistinguishable from the real data used train it. This is a significant evolution over previous models which could produce reasonable facsimiles of the training data, but ones which could be visually distinguished from the training data by human evaluation. Recent work on OOD detection has raised doubts that generative model likelihoods optimal OOD detectors due issues involving likelihood misestimation. entropy in the generative process, and typicality. We speculate that generative OOD detectors also failed because their models focused on the pixels rather than the semantic content of the data, leading to failures in near-OOD cases where the pixels may be similar but the information content is significantly different. We hypothesize that estimating typical sets using self-supervised learners leads to better OOD detectors. We introduce a novel approach that leverages representation learning. and informative summary statisties based manifold estimation, to address all of the aforementioned issues. Our method outperforms other unsupervised approaches and achieves state-of-the art performance well- established challenging benchmarks, and new synthetic data detection tasks. Our code is available at gighhb..com/DDbaagaa/fffo..
INTRODUCTION
In the past decade, deep learning has made significant strides, primarily due to the availability of large-scale annotated datasets (Deng al. (2009) used in supervised learning and the emergence of
self-supervised learning utilizing vast web-scale crawled open data (Schuhmann et al. 2022 Gao
et al. 2020 Sharma et al.
et al. 20216). The transition from large-scale annotated datasets to self-supervised learning was driven by the expensive and labor-intensive nature of creating
these datasets and yet the concerns surrounding data usage rights persist (He al. 2022b Carlini
et al. 2021 Huang et al. 2022). Recent advancements have led to the development of generative models that excel in generating highly realistic and detailed synthetic images (Stein al. 2024).
In this paper, we investigate identifying out-of- -itor--distibbttonn (OOD) data and generated synthetic data created using large-scale pre-trained generative models, commonly referred "foundation
models" (Bommasani et al. 202T).
Broadly speaking, data encountered during deployment that was not sampled from the distribution used to generate training data is considered OOD. OOD data represents challenge to safe deployment of predictive models because they can make confident incorrect predictions, leading to actions which could have negative outcomes. Foundation models complicate the traditional definition of OOD slightly. These models are trained on extensive and diverse datasets, making the data generating process, and thus possible OOD inputs difficult to specify cleanly. Despite its diversity, the data generating process only samples a small portion of the input space, leaving many potential subspaces open for OOD contamination. This contamination can erode the calibration of predictive models trained using the foundation model as a base, representing a significant hazard for safe model deployment.
"This research was supported in part by NSF Awards 2117439 and 21 11166

---

Published as a conference paper at ICLR 2025
Predictive model failures due to OOD inputs can be understood through the lens of typicality. The concept of typicality arises from information theory and codifies the difference between likelihood from generative process, and the probability of generating sample from that process with particular likelihood. In low dimensions these two are generally equivalent, but this is often not the case in high dimensions (Nalisnick et al.] 2019 Choi et al.] (0188). The asymptotic equipartition property asserts that most of the probability mass of distribution is contained in the region of high typicality, referred to as the "typical set" of a distribution (Cover 1999). Thus, OOD inputs confound predictive models because the model has no incentive to make calibrated predictions on this data, since any errors in the predictive model for OOD inputs do not contribute to the risk to the hypothesized model, which is being minimized via training. At the same time, the atypicality of OOD data presents a potential way to identify them (Nalisnick et al. 2020: Morningstar et al. 2021 and thus avoid making risky predictions.
Measuring typicality directly challenging because one cannot typically query the probability of an observed datum under the (unknown) data generating process. This has led many prior works to propose using generative models to approximate the process and use it to measure the probability, and therefore test the typicality of the input. However, Zhang et al. (2021) and Caterini & Ganiza (2022) have pointed out challenges in this approach that arise due to likelihood misestimation, which can cause OOD inputs to have likelihoods under the generative model that consistent with ID) data.
In this paper, we hypothesize that many of the shortcomings with typicality- approaches could be addressed using statistics which tune to the semantic content of the data. We propose to leverage self-supervised representations, which extract semantic information while discarding many potential confounding features (e.g. textures, backgrounds). Our specific contributions are:
Forte, a novel framework combining diverse representation learning techniques (CLIP, ViT-MSN, and DINOv2) for typicality estimation, uses both parametric (GMM) and non- parametric (KDE, OCSVM) density- based methods to detect atypical samples, i.e. out-of- distribution (OOD) and synthetic images generated by foundation models. Forte requires no class labels, exposure to OOD data during training, or restrictions on the architecture of generative models.
2. A set of per-point summary statistics (precision, recall, density, and coverage) that effectively capture the "probability distribution of the representations" using reference and unseen test samples in the feature space, enabling more nuanced anomaly detection.
3. Extensive experiments demonstrating Forte's 's superior performance compared to state-of-the- art supervised and unsupervised baselines on various OOD detection tasks, and synthetic image detection, including photorealistic images generated by advanced techniques like Stable Diffusion.
4. Insights into the limitations of relying solely on statistical tests and distribution- metrics for assessing the similarity between real and synthetic data, highlighting the effectiveness of our proposed per-point summary statistics and anomaly detection framework.
2 RELATED WORKS
In discriminative machine learning, the assumption that inference data mirrors the training data distribution foundational, yet often flawed. The occurrence of out-of-distributioo (OOD) inputs can lead to misleadingly confident but incorrect predictions by models, posing significant reliability and safety concerns. Large neural networks are vulnerable adversarial perturbations (Szegedy et al. 2013) and poor calibration (Guo et al. 2017). necessitating OOD detection. OOD detection methods are either supervised, using labels and examples to calibrate models or train them to identify
OOD data (Liang et al. [2018] Hendrycks et al. 2019b] Meinke & Hein 2020 Dhamija et al. 2018) or unsupervised, employing generative models to approximate the training data density q q(X) and
determine prediction reliability via q X), assuming OOD inputs have lower probability (Bishop) 1994).
Unsupervised OOD detection methods typically generative models to measure the likelihood of the data. (Bishop 1994) assumed low likelihood would be observed on OOD data, but this may
fail in high dimensionality (Choi et al. 2018 Nalisnick et al. 2019 Hendrycks et al.) 2019b Serrà
2

---

Published as a conference paper at ICLR 2025
Phacisio
Ueisity
Reternce urple
Eatiee tample
sulernnce cincle
Layane
hoaett Pal
centty
-2-
ralerscc cungle
Featume
Testure
Figure I: Visualization of Precision, Recall, Density, and Coverage metrics for reference and test samples in a 2D feature space, with nearest neighbour k=1. Precision is depicted by blue circles around reference samples, recall by red circles around test samples, density by solid teal circles around reference samples, and coverage by green circles around test samples.
et al. 2019). Fixes include using WAIC (Choi ct al. 2018), likelihood ratios (Ren et al. 2019), or typicality tests (Nalisnick al. 2019). but these have limitations associated with high dimensional likelihoods. |Moriinsstar et al. (2021) proposed the Density of States Estimator (DoSE). Inspired by principles from statistical physies, leverages multiple summary statistics from generative models to distinguish between in-distribution and OOD data.
Supervised OOD detection leverages labeled in-distribution and/or known OOD data. Techniques
include using VIB rate (Alemi et al. 2018), calibrating predictions with ODIN (Liang et al. 2018 or
its generalization (Hsu et al.) 2020). ensembling classifiers [Lakshminarayanann et al. (0117). training with OOD examples to encourage uniform confidence (Hendrycks et al. 2019b), adversarial attacks to lower confidence near training data (Stutz et al. 2019), visual concept networks to model human understandable features in graphs (Ganguly et al. (2025) and fitting Gaussian mixtures to latent representations (Meinke & Hein 2020). While effective, these methods require labels or specific outliers.
Generative model assessment can be done over four categories as shown by Stein et al. (024):
ranking, fidelity, diversity, and memorization. Ranking metrics include FD (Heusel et al. 2017)
which measures the Wasserstein-2 distance between real and generated distributions; FD
(Coon
Forsyth 2020), which removes bias due to finite sample size; sFID (Nash et al. 2021 which uses spatial representation; KD (Binkowski et al. 2018), proper distance between distributions; IS (Salimans al., 2016), which evaluates label entropy and distribution; and FLS (Jiralerspong et al.3 a density estimation method. Fidelity and diversity are measured using precision, recall (Sajjadi et al., 2018 Kynkäänniemi et al., 2019b), density, coverage (Naeem et al. 2020a),, rarity score (Han et al. (2022), and Vendi score (Friedman & Dieng 2022). which quantify the quality and variety of generated samples. Memorization is quantified using AuthPet (Alaa et al. (022),,
Cr score (Meehan et al., 2020), FLS-POG (Jiralerspong al. 2023), and memorization ratio with calibrated 1/ distance (Somepalli et al. 2022), which detect overfitting and copying of training data. These metrics can be computed in various supervised (including Inception- (Szegedy al., 2016)) and self-supervised representation spaces, including self-supervised model families like contrastive (SimCLRv2 ( et al. (2020)) self-distillation (DINOv2 Oquab et al. (2023a)), canonical correlation analysis (SwAV Caron et al. (2020)) masked image modelling (MAE He et al. (2022a). data2vec Baevski al. (2022). DreamSim Fu et al. (2023), and language-iman (CLIP Radford al. (2021a).
sometimes using the OpenCI.IP implementation IIharco et (202) trained on DataComp- 1B Gadre
et al. 2023)).
FORTE: A FRAMEWORK FOR OOD DETECTION USING PER-POINT METRICS
In this section, we introduce Forte, novel framework that combines diverse representation learning techniques with per-point summary statistics and parammemtiii density estimation models to detect out-of-distribution (OOD) and synthetic data.
3

---

Published as a conference paper at ICLR 2025
3.1 PROBLEM SETUP
We start with a dataset X = fx [/ that comes from a true but unknown distribution p. where each 2j E I' is sampled independently and identically from p. In the context of out-of-distribution detection, consider that the unseen data { y might come from mix of the true distribution p and an unknown confounding distribution á (which can be from an OOD benchmark or synthetic data generated using a model such as Stable Diffusion). The mixed distribution is denoted by X (x) (1 ())p()) where an unknown mixing parameter. Since both a and p are unknown, we cannot directly obtain samples from P or make any assumptions about and p. The goal of OOD detection is to develop decision rule to determine when the input data X is atypical.
We build on Density of States Estimation (DoSE: Morningstar al.) 2021), which is the SoTA unsupervised method that addresses the OOD detection problem. The first step in DoSE is to create a summary statistic Tn that can help evaluate new, unseen data. Some examples of T, are the negative log-likelit aigllikelihood of X, the L2 distance between X and the sample mean, or the maximum likelihood estimate of joint distribution g( [X, 0n). To determine when the input data X is atypical, DoSE models the distribution of T values in the training data and uses the density to score the typicality of inputs based on the measured values of their statisties. While DoSE gives valuable mechanism by which one can determine the typicality of a particular input, it does not identify which statistics are informative. DoSE further limited its focus likelihood-based generative models, which Caterini & Loaiza-Ganem (002)) showed may be suboptimal for OOD detection.
3.2 CREATING GENERALIZED NOVEL SUMMARY STATISTICS
We propose the following per-point metrics, novelly adopted to OOD detection from previous work in manifold estimation inside representation spaces by Naeem et al. (2020b) and Kynkkännnee et al. (2019a) for capturing different facets of the generated samples. Let 1(-) be the indicator function, ((
m Bxx NNDDa<:))) )) where (ax, r) is Euclidean ball centered at with radius
r, and NND (x,)
) is the distance between x. and its k-th nearest neighbor in {x j i=1 excluding
itself.
1. Precision per point is a binary statistic that indicates whether each test point within the nearest neighbor distance of any reference point:
precision (i) pp 1 (a)) (((x))m
(1)
A high overall precision indicates that the test samples are closely aligned, and similar to the reference data distribution.
2. Recall per point computed for each test point by counting the number of reference points within its nearest neighbor distance and dividing by the total number of reference points:
m
recall ()) PP
m Σ 1
(af E (ND))NN
A high recall implies that the test distribution collectively covers a significant portion of the reference data distribution, i.e. that the test distribution contains diverse samples that represent the different regions of the reference data distribution.
3. Density per point is computed for each test point by counting the number of reference points for which it is within that points nearest neighbor distance and dividing by the product of and the total number of reference points:
0) density pp km Σ n (af NND,
A test point with high density suggests that it is located in region of high probability density in the reference data distribution, i.e. it captures the underlying density of the reference data. Density per point essentially measures the expected likelihood of test points against the reference manifold, providing a more informative measure than the binary precision per point.
4

---

Published as a conference paper at ICLR 2025
4. Coverage per point is computed for each test point by checking if its distance to the nearest reference point is less than its own nearest neighbor distance:
coveragepp ① = (min x[)) < NNDs(a)))
High coverage indicates that the test samples are well-distributed and cover the support of the reference data distribution. Coverage per point effectively improves upon the original recall metric by building manifolds around reference points, making it more robust to outliers in the test data and computationally efficient.
These per-point metrics serve as summary statistics that capture local geometric properties the data manifold in the feature space. They enable us to model the distribution of in-distribution (ID) data and identify out-of-distribution (OOD) samples effectively. We extract representations from reference and test images using CLIP Radford et al. 202 which learns map images and text into shared latent space, self-supervised models Gui et al. (2023) like ViT-MSN Assran et al. (2022). which predicts similarity between masked views of the same image, and DiNo v2 Oquua al. (2023a), which distills knowledge from teacher to a student network. Combining representations from these diverse models captures distinct aspects of the data via different manifolds and improves robustness of our anomaly detection approach, as we observe later in Table 3
3.3 DEVELOP DECISION RULE
We use the summary statistics to develop an non-parametric density estimator as an anomaly detection model. First, we generate feature vectors for the reference data using self-supervised learning methods. We split the data into three parts: one-third for held-out testing, one-third as the reference distribution, and one-third statistics for the second and third to understand what these statistics look like when the test data matches the real data (i.e. p Q). We train One-Class SVM (Schölkopf et al. 2001). Gaussian Kernel Density Estimation (Parzen 1962), and Gaussian Mixture Model (Reynolds et al., 2009 on the reference summary statistics. These models learn a decision boundary enclosing the typical set of the reference data distribution. We then test the models' ability to distinguish between a mix of held-out test and reference features by comparing their atypicality i.e. summary statistics to the reference distribution. By evaluating the models' performance on the test set, we assess their effectiveness in detecting OOD samples and distinguishing real from synthetic data distributions.
test distribution that is drawn from the reference distribution. We calculate summary
To assess the performance of the anomaly detection models, using an unseen part of the training real distribution and the generated data distribution, we use Area Under the Receiver Operating Characteristic Curve (AUROC), which measures the ability of the model to discriminate between normal and anomalous points across different decision thresholds, and False Positive Rate at 95% True Positive Rate (FPR @95) which indicates the proportion of normal points incorrectly identified as anomalies when the model correctly identifies 95% of the true anomalies.
Under certain theoretical assumptions, we can justify the effectiveness of our per- per-point metrics in distinguishing between in-distribution (ID) and oui-t--dist (OOD) data. Specifically, if the reference data (x"] } and the test data {x]] jiz are drawn from Gaussian distributions with the same covariance but different means (with a significant mean difference), the expected values of these metrics differ markedly between ID and OOD samples. For ID data, the expected per-point precision and coverage approximate 1 k the expected per-point recall is roughly k/ t/m, and the expected per-point density is close to 1. In contrast, for OOD data, these expected values are near zero due to the large mean difference, which causes the test samples to fall outside the typical regions of the reference distribution. This substantial disparity provides strong theoretical foundation for using these metrics as effective summary statistics for OOD detection. By computing these metrics for test samples and comparing them to the expected ID values, we can effectively identify anomalous data points. A detailed proof and further explanations are provided in the Appendix
4 EXPERIMENTS
Unsupervised Baselines We train an ensemble of deep generative models on in nn-distribution data, validate heldout set ensure memorization, and compute DoSE scores on in-distribution
5

---

Published as a conference paper at ICLR 2025
and OOD test sets. We measure OOD identification performance and compare against several unsupervised baselines: single-sided threshold (Bishop) 1994). single-sample typicality test (TT) (Nalismick et al.] 2019), Watanabe-Akaike Information Criterion (WAIC) (Choi et al. 2018), ) and likelihood ratio method (L.LR) (Ren et al. 2019). Given the best demonstrated unsupervised OOD detection results were in (Morningstar al.) (02T), stick to using Glow models (Kingma & Dhariwal 2018). where we use summary statisties: log-likelihood, log-probability of the latent variable, and -og-determnnant of the Jacobian between input and transformed spaces.
We also compare our approach against state-of-the-ar methods in out-of- distribution (OOD) detection literature, selecting the best reported results on established benchmarks, regardless of the model architecture or techniques employed. Specifically, we consider results from NNGuide (Park et al. 2023), Virtual Logit Matching (Wang et al., 2022), and OpenOOD v1.5 (Zhang et al., 2024a). For OpenOOD v1.5, we select the top-performing entries from the leaderboard, which utilize a Vision Transformer (ViT-B) trained with cross-entropy loss, along with Maximum Logit Score (MLS) (Hendrycks et al., 2019a) and Relative Mahalanobis Distance Score (RMDS) (Ren) al. 202T) for OOD detection. To explore the potential of foundation models in OOD detection, OpenOOD also employs linear probing of Dinov2 (Oquab et al. 2023b) in conjunction with MLS,
Supervised Baselines:
however performance is demonstrated to be poor.
4.1 BASELINE PERFORMANCE FOR SYNTHETIC DATA DISTRIBUTIONS
Distribution Divergence: We compute pairwise distances and divergences between real and gen- erated feature distributions. First we start with KL Divergence which measures information loss
when approximating with Q Kullbacck& & Leibler 1951): p: DRa. Q) = Ezex P(xr) log Q())
P(a)
and JS Divergence which the symmetric KL divergence variant, comparing distributions with disjoint support Lin (1991): Dis(P || Q) = 2 (Px || M) + Dia(Q) M)) .M (P
The Wasserstein Distance on the other hand gives the minimum cost to transform one distribution into the other Rubner et al. 2000): W(P, Q) inf B(x, y) ~ >|1x vII. T(P, Q): joint distri- butions with marginals P.Q. while the Bhattacharyya Distance gives the similarity between P and Q,
O (identical) to oo (separated) Bhattacharyya 1943): Dn(P, Q) In ( x E X ((x))
CLIP-based Zero Shot Strong OOD Baseline: To establish baseline performance for detecting anomalous generated images directly from representations, we split the in-distribution reference features X" extracted from any of the feature extraction model (e.g. CLIP) into training X and testing Xist sets and train all the same non-parametric density estimation models with the same hyperparameter tuning, including OCCSVM, KDE and GMM. We evaluate each model on a test set consisting of representations from held-out reference images X and anomalous images X9. This provides strong initial assessment of how well these models can distinguish between reference and anomalous data based on, for e.g. the CLIP features.
To rigorously compare the real and generated feature distributions, we perform several statistical tests. The first is the Two-sample Kolmogorov-Sminn Smiinvvvvmiovvvviio (KS) Test, which is a non-parametric test that compares the empirical Cumulative Distribution Functions (CDFs) of two samples. The KS statistic measures the maximum distance between the CDFs, with a corresponding p-value for the null hypothesis that the samples are drawn from the same Iistributior/Massey Jr (195I). The Whinney U Test gives a non-parametric test for whether two independent samples are drawn from the same distribution. It is based on the ranks of the observations in the combined sampleMann & Whitney (1947). The Z-test compares the standardized differences between the means of two samples, assuming they are normally distributed. The Z-score measures the distance between the sample means in units of standard error (Sprinthal] (201T). Finally we also calculate the Anomaly Detection with Local Outlier Factor (I.OF) and Isolation Forest (IF): These algorithms assign anomaly scores to each point based on their local density (LOF) Breunig et al. (2000) or case of isolation (IF) Liu et al. (2018) relative to the real data. Higher scores indicate a generated image is anomalous.
Statistical Tests:
Generative Evaluation Techniques: In line with suggestions from (Stein et al. 2024). we use the Fréchet distance (FD) and FD.c computed with the DINOv2 encoder, instead of the inception network to tell how different the generated images are from the reference set of images. We also use, the CLIP maximum mean discrepancy (CMMD), which is distance measure between probability

---

Published as a conference paper at ICLR 2025
Table I: Performance comparison between our method and established supervised SoTA methods various out-of- distibbbibttionn (OOD) detection tasks. Our method consistently achieves the best performance across a range of dataset pairings, particularly outperforming on challenging datasets like NINCO and SSB-Hard. It also excels at detecting covariate shifted ID datasets. Key configurations include ViT-B trained with Cross Entropy and the RMDS postprocessor, and DINOv2+MLS, which utilizes training with Linear Probe DINOv2 alongside the MLS postprocessor, For reliable measurements, Forte is run with 10 random seeds. More comparisons are provided in Fig. 16
VIE-B-CR-RMSS
DINOVZAMILS
Forte-GMM[(Urr)
In-Distr (ImogeNeet-lk)
AUROC
AUROC
AUROC
AUROC
FPR95
FPR95
iNotumaiat
99.57
96.10
19.47
98.41
00,03
0.64
Texture
95.82
21.58
95.34
20.31
89.38
37.22
91.82
33.95
96.04
00.10
5.61 1: 00.25
Far-OOD
Openlmage-O
92.32
29.57
95.58
16.88
96.73 + 00.11
11.77 + 00.57
NINCO
87.31
46.20
88, 35
41.00
98.34 :: 00.09
5.18 1: 00..1
Near-DOD
SSB-Hard
72.87
84.52
77.28
72.00
94,95 + 00.17
22.30 =:: 01.45
ImageNett-C
82.25
00.16
42.56 1: 02.23
ImageNect
Covoriate
ImageNes-V2
00.21
Table 2: Comparison of performance figures between our method and various un- supervised baselines for out-of-distribution (OOD) detection tasks, demonstrating superior performance of our method across all challenging dataset pairings. For reliable measurements, Forte run with 10 random seeds. This presented as a
visualization in Fig 15
Method
In-Dist (CIFAR-10)
OOD (CIFAR-100)
OOD (Celeb-A)
OOD (SVHN)
AUROC
$2.00
91.40
6.50
FPR95
AUROC
53.20
92,80
14,30
WAIC
EPR95
90.16
46.04
98.83
AUROC
54,80
$4.80
87.00
FPR95
92.06
74.00
61.28
AUROC
48.13
80 08
64.21
FPR95
93.44
64.79
76.36
97.30
DOSE
FPR95
91.95
12.82
13.16
Forte+SVM (Ours)
AUROC
97.29
00.55
100.00
00.00
99.84
00.05
FPR95
09.08
01.82
0,00
00.00
00.00
00.00
Forte+KDE (Ours)
AUROC
94.81
+ 01.22
99.75 1: 00.06
37
02.07
FPR95
18.20
03.82
0.06
00.07
07.53
10.77
Forte+GMM (Ours)
AUROC
97.63
: 00.15
100.00 1: 00.00
99.49 00.48
PPR95
distributions that presents benefits over the Fréchet distance. With an appropriate kernel, CMMD does not assume any specific distribution, unlike the Fréchet distance which assumes multivariate normal distributions.
5 RESULTS
Tables & 2 present the performance comparison between our proposed methods (Forte+ SVM, KDE. and Forte+GMM) and various state of the art supervised and unsupervised techniques for out-of-distributtin (OOD) detection techniques. Our methods consistently outperform techniques across all challenging dataset pairings, demonstrating their effectiveness detecting OOD samples without relying labeled data.
Table 3 presents an ablation study investigating the impact of incorporating multiple representation learning techniques into the Forte framework, when detecting arbitrary superclasses from the Im- ageNetlk hierarchy. In this challenging task example, the in-distribution data consists various vehicle classes from the ImageNet dataset, such Ambulance, Beach wagon, Cab, Convertible,
7

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study investigating the impact of incorporating multiple representation learning
techniques for detecting arbitrary superclasses from the ImageNet hierarchy, using FoMt++GMM.
In Distribution (Superclass ImageNet 2012)
Far OOD
OOD
AUROC
FPR95
AUROC
FPR95
Forte (CLIP)
99.42
00.04
88.84
61.04
Forte (MSN)
99.13
00.39
82.63
77,53
Forte (DINOv2)
99.79
00.03
86.97
79.25
(CLIP+MSN)
99.96
30.77
Forte (CLIP+DINOV2)
99.98
00:01
91.35
26.89
Forte (DINOv2+MSN)
99:97
00.00
88.71
31.65
Forte (all
100,00
00.01
91.17
28.91
Jeep, Limousine, Minivan, Model t, Racer, and Sportsear. The
data is divided into two categories: Near OOD, which includes other utility vehicle classes like fire engine, garbage truck. pickup, tow truck, trailer truck, minivan, moving van, and police van; and Far OOD, which consists of animal classes such as Egyptian cat, Persian cat, Siamese cat, Tabby cat, Tiger cat, Cougar, Lynx, Cheetah, Jaguar, Leopard, Lion, Snow leopard, and Tiger. Our method is successful in very challenging near-OOD situations, and becomes more so when using multiple representations. The combination of all three techniques (CLIP, MSN, and DINOv2) achieves the best overall performance for both Far OOD detection, and comes close second in Near OOD detection. These findings suggest that leveraging diverse SSL representation techniques within the Forte framework can capture complementary information and enhance the overall OOD detection capabilities.
5.1 PERFORMANCE ON SYNTHETIC DATA DETECTION
To generate synthetic data for assessing distributional robustness, we first employ the Stable Diffusion Img2lmg setting (Rombach et al. 2022), where diffusion- based model can generate new images conditioned on an input image and a text prompt. We use the Stable Diffusion 2.0 base model and generate images with varying strength parameters (0.3, 0.5, 0.7, 0.9, 1.0) to control the influence of the input image on the generated output, essentially allowing our real distribution to be prior of controllable strength for the generated distribution. We also use the Stable Diffusion 2.0 text-to-image model to generate new images conditioned on the captions generated by BLIP et al. 2022) for each real image from the reference distribution. This allows us to create images that are semantically similar to the real images but with novel compositions and variations. Finally, we also generate images directly from the class name associated with each real image (e.g.. "a photo of a (monarch butterfly), ), in natural setting"). This provides a baseline for generating images that capture the essential characteristics the class without relying on specific input images or captions. The image generation pipeline is implemented using the Hugging Face Transformers (Wolf et al. (2020) and Diffusers (von Platen et al., 2022) libraries, which provide high-level APIs for working with pre-trained models. Examples can be found in Figure 2
Figure 3 & Figure 4
Fréchet Distance (FD), F FD.., and CMMD scores provide insights into the distribution shift between real and synthetic images, with generated images moving further away from the real distribution as diffusion strength increases. However, these distribution-level statistics do not provide information about individual images within the distribution, necessitating an OOD detection strategy. Tables 4 & 7 compare the performance of Forte+GMM against a strong CLIP-based baseline on the Golden Retriever and Volleyball classes from ImageNet. For the well-represented Golden Retriever class, Forte+GMM consistently outperforms the bascline across all image generation settings, achieving near-perfect AUROC scores and low FPR95 values for Img21mg with strength parameters 0.9 and 1.0, Caption- and Class-based image generation. Lower performance at lower strengths is due to images being too similar to the real distribution. The Volleyball class, part of Hard ImageNet (Moayeri et al. 2022), focuses on classes with strong spurious cues. Volleyballs rarely occur alone in ImageNet images, and generating images without appropriate priors results in mode collapse (see Figure 4). Forte+GMM surpasses the CLIP-based baseline in all image generation scenarios, with notable improvements for Img2lmg with higher strength parameters, Caption- based, and Class-based image generation (AUROC scores 97%, FPR95 values 6%).
8

---

Published as a conference paper at ICLR 2025
Table 4:
Performance comparison of FoMte+GMM against CLiP-based baseline, and Gen Eval methods for detecting synthetic images of Golden Retrievers generated using various techniques.
FD
CMMD Score
Baseline
CLIP
Forte+GMM
AUROC
FPR95
AUROC
FPR95
Tmg2Tmn S-0.3
453.63
418.22
0.52
61.19
86.27
68.28 + 02.14
68.07 + 05.56
1mg2Tmg S=0.5
648.97
624.01
0.64
59.49
86.27
82.93 1: 02.50
46,80 1: 10.68
Img22mng S=0.7
762.17
735.42
0.64
61.23
86.36
94.19 1: 01.85
24.90 1: 12.99
Imn2ling S=0.9
845.90
819.06
0.66
59.05
88.87
97.59
01.23
14.41 + 13.55
Img2Img S=1,0
891.39
870.17
0.73
60.15
89.23
98.11 :: 00.75
06.09 +: 05.72
Caption-based
575.18
546.42
0.90
80.71
71.54
96.77 = 01.14
18.90 = 14.38
Class-based
1,065.90
1,048.18
1.07
75.73
82.23
98.26 + 01.12
10.22 + 13.04
Tables [ & 7 compare various statistical measures for assessing the similarity between distributions of real and generated images across different image generation methods. The results shown for the Golden Retriever and Volleyhall classes. For both classes, the statistical tests fail to provide clear and consistent indications of distributional differences between real and generated images. Z-scores are close to zero, suggesting no significant difference in means, while K-S test and Mann-Whitner U test p-values inconsistent across generation methods. LOF and IF scores show limited variation and struggle to identify anomalies in the generated images. JSD, KLD, Wasserstein distance, and Mahalanobis distance do not exhibit clear trends generation strength increases, making it difficult to draw meaningful conclusions about distributional similarity. To add to this problem, JSD and KLD are not defined for any distribution of representations apart from MSN embeddings. Bhattacharya distances are not defined any representation, indicating overlap. These results highlight the limitations of relying solely on statistical tests comparing representations of new points to the original distribution, as they often fail to capture subtle distributional differences between real and generated images, particularly when generated images are highly realistic. The inconsistencies and lack of clear trends underscore the need for more sophisticated approaches, such the proposed Forte+GMMM framework, which leverages diverse feature extraction techniques and anomaly detection algorithms effectively detect synthetic images.
5.2 PERFORMANCE ON MEDICAL IMAGE DATASETS
Magnetic resonance imaging (MRT) datasets and models present unique challenge in high stakes scenario, making robust duitti--dstribuutinn (OOD) detection paramount. These datasets, typically acquired under specific study protocols, suffer from batch effects that hinder model generalization even when changes in protocols are minute (Horgg) 2023). The problem is exacerbated by dataset homogeneity within studies and limited dataset sizes in clinical applications, making it impractical to train separate models for each batch. Subtle distribution shifts between datasets of the same subject matter, though not immediately apparent to human observers, can significantly impact model performance. While data augmentation and harmonization methods have been explored (Hu et al.) 2023), existing metries for detecting distribution drift and assessing harmonization effectiveness are limited. Our work proposes deploying Forte as a more robust metric to address this gap by effectively differentiating between datasets that may appear similar but have crucial distributional differences. Using public datasets like FastMRI Zbontar et al. (2018); Knoll et al. (2020) and the Osteoarthritis Initiative (OAI) Nevitt et al. (200)), we simulate realistic scenarios where models trained on one dataset (treated as in-distribution) are confronted with another (considered OOD). Table 5 demonstrates the effectiveness of Forte in differentiating the datasets. In other tests over closed-soure but real-world hospital data, observe similar perfect performance, suggesting that Forte can have zero-shot applications in a variety of other sensitive domains.
DISCUSSION
While just using precision and density as summary statistics would lead Forte to be considered single- sample tests, recall and coverage make our framework two-sample test, they take into account

---

Published as a conference paper at ICLR 2025
Table 5: Out-of-distribution (OOD) detection using Forte, applied to medical image datasets. Strong performance by Forte suggests the presence of batch effects and a need for data harmonization. Refer to Appendix [ for more details on the FastMRI and OAI datasets. For reliable estimation, performance is measured over 10 random seeds.
In-Dist: OOD:
FastMRI NoES
FastMRI FS
FastMRI FS OAI MPR
Method
Metric
OAI TSE
OAI T1
AUROC
100.00 = 0.00
100.00 = 0.00
100,00 = 0.00
Fo (Ours)
AUROC
97.97 + 5.63
95.98 + 7.84
95.99 + 7.86
Forte+KDE (Ours)
FPR95
9.49 + 26.76
19.75 39.10
19.77 39.31
Forte=GMM (Ours)
AUROC
99.95 1 0.13
99.91 1: 0.16
99.91
1 0.17
FPR95
0.00 + 0.00
0.00 = 0.00
0.00 0.00
the relationship between each synthetic image and the entire set of real images. This also allows us to better understand the behavior of the generated images in relation to the real data distribution. This is usually not problem in production, and benchmarks, since when you check whether sample is out of distribution, you usually sample from the mixture probability distribution of in and out of distribution, which is what we really use here. Prior work has also operated in this paradigm, i.c. two sample OOD test such as the typicality test (Nalisnick et al. (2019).
When selecting non-parametric density estimators to model typicality, it is important to consider the manifold properties. We observe that GMM excels with clustered data, especially with bigger number of gaussians when operating with data from multiple possible classes, while KDE struggles with sharp density variations. OCSVM is robust to outliers and performs well in high-dimensional spaces, making it suitable for cohesive normal data.
DoSE (Morningstar et al. 2021) pioneered chaining multiple summary statistics for typicality measurement, using ID sample distributions to construct typicality estimators rather than direct statistic values. While groundbreaking. DoSE's reliance generative model likelihoods proved problematic, as subsequent work (Caterini & Doaiza-Ganena 2022, Zhang et al., (02I) showed these can be unreliable for OOD detection. Our approach addresses these limitations through four key improvements: (1) utilizing self sule-super representations to capture semantic features, (2) incorporating manifold estimation account for local topology, (3) unifying typicality scoring and downstream prediction models to minimize deployment overhead, and (4) eliminating additional model training requirements. These advances yield substantial empirical gains. While building upon DoSE's fundamental statistical machinery, our modifications dramatically enhance practical performance.
7 CONCLUSION
Our work underscores the importance of developing robust methods for detecting out-of-distribution (OOD) data and synthetic images generated by foundation models. The proposed Forte framework combines diverse representation learning techniques, non-parametric density estimators, and novel per per-point summary statistics, demonstrating far superior performance compared to state-of-the-art baselines across various OOD detection tasks and synthetic image detection tasks. The experimental results not only showcase the effectiveness of Forte but also reveal the limitations of relying solely on statistical tests and distribution-level metrics for assessing the similarity between real and synthetic data. We hope that as generative models continue to advance, strong test frameworks like Forte will play crucial role in maintaining the reliability of ML systems by detecting deviating data, unsafe distributions, distribution shifts, and anomalous samples, ultimately contributing to the development of more robust and trustworthy Al applications in the era of foundation models.
REFERENCES
Mohamed Akrout, Bálint Gyepesi, Péter Holló, Adrienn Poór, Blága Kineso, Stephen Solis, Katrina Cirone, Jeremy Kawahara, Dekker Slade. Latif Abid, et al. skin disease classification: Impact across original medical datasets to fully synthetic images. In
data augmentation for
10

---

Published as a conference paper at ICLR 2025
International Conference Medical Image Computing and Computer-Assistel Intervention, pp- 99-109. Springer, 2023.
Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? Sample-Level metrics for evaluating and auditing generative models. In International Conference Machine Learning, pp. 290-306. PMLR, 2022.
Alexander A Alemi, lan Fischer, and Joshua V Dillon. Uncertainty in the variational information bottleneck. arXiv preprint arXiv: /80700096, 2018.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pp. 456-473. Springer, 2022.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning,
1312- PMLR, 2022.
Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust classification via generated datasets. arXiv preprint arXiv:2302.025503 2023.
Anil Bhattacharyya. On measure of divergence between two statistical populations defined by their probability distribution. Bulletin of the Calcutta Mathematical Society, 35:99-110, 1943.
Christopher M Bishop. Novelty detection and neural network validation. IEE Proceedings- Vision, Image and Signal processing, 141(4):217-222, 1994.
Mikolaj Binkowski, Dougal Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL https
//openreview.rr net.: f 1o
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2 108.078...00 100.022888.77 2021.
Markus Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93-104, 2000.
Max F Burg, Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, and Chris Russell. A data augmentation perspective on diffusion models and retrieval. arXiv preprint arXiv:2304. NoXi:2230410023,, 2023.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), pp- 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9012-99222 2020.
Anthony L Caterini and Gabriel Loaiza- Ganeaa. Entropic issues in likelihood- -ased ood detection. In (Still) Can't 't Believe It's 's Not Better! Workshop at NeurIPS 2021, pp. 21-26. PMLR, 2022.
Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2135-12144, 2022.
11

---

Published as a conference paper at ICLR 2025
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervien learners. In Advances in Neural Information Processing Systems, volume 33. pp. 22243-22255, 2020.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXtv:18100001322..0000 2018. 2018.
Min Jin Chong and David Forsyth. Effectively unbiased FID and inception score and where to find them. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6070-6079, 2020.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In CVPR, 2009.
Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. In Advances in Neural Information Processing Systems, pp. 9157-9168, 2018.
Dan Friedman and Adji Bousso Dieng. The Vendi Score: A diversity evaluation metric for machine learning. 100.02400, 2022.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learningg new dimensions of human visual similarity using synthetic data. arXiv:2306.099 09344, 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Sarah Daras, Giannis adn Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Veneu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song. Hannanch Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: In search of the next generation of multimodal datasets. arXiv:2304. 1144.41000 /4108, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, arXiv preprint 2rXiv:2288.01615 2022.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Designing an encoder for fast personalization of text-to-image models, arXiv e-prints, pp. arXiv- 2302, 2023.
Debargha Ganguly, Debayan Gupta, and Vipin Chaudhary. Visual concept networks: A graph-based approach to detecting anomalous data in deep neural networks. In Christian Wallraven, Cheng-Lin Lin Liu, and Arun Ross (eds.), Pattern Recognition and Artificial Intelligence, pp. 219 232, Singapore, 2025. Springer Nature Singapore. ISBN 978-981-97-8702.9
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint (0X.x:220110000 2020.
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images. arXiv preprint arXiv:2302. 13861, 2023.
Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self- supervised learning: Algorithms, applications, and future trends. arXiv preprint 2:X1022301.05712 2023.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1321-1330. JMLR. org, 2017.
12

---

Published as a conference paper at ICLR 2025
Jiyeon Han, Hwanil Choi, Yunjey Choi, Junho Kim, Jung- Woo Ha, and Jaesik Choi. Rarity score: A new metric to evaluate the uncommonness of synthesized images. In The Eleventh International Conference Learning Representations, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022a.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr. Song Bai, and XIAO- JUAN QI. Is synthetic data from generative models ready for image recognition? The Eleventh International Conference Learning Representations, 2022b.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling aut-of-distribution detection for real-world settings. arXiv preprint arXiv: 132, 2019a.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019b. URL https
//openrevien nww. forum? htt.mmmmiddxxxxxxccccc
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Hannah Horng. Batch Effect Detection and Harmonization Methods for Quantitative Features Extracted from Medical Images. PhD thesis, University of Pennsylvania, 2023.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-of-distribution data. arXiv preprint arXiv:2002.11297, 2020.
Fengling Hu, Andrew Chen, Hannah Horng. Vishnu Bashyam, Christos Davatzikos, Aaron Alexander-Bloct Mingyao Li, Haochang Shou, Theodore D Satterthwaite, Meichen Yu, al. Image harmonization: review of statistical and deep learning methods for removing batch effects and evaluation metries for effective harmonization. Neurolmage, 274:120125, 2023.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint 2xXiv:2205./2628 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori,
Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL /ttp:////oo....055252225220
rerodoo554377
Marco Jiralerspong, Avishek Joey Bose, and Gauthier Gidel. Feature likelihood score: Evaluating generalization of generative models using samples. arXiv:2302.06 044400 2023.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33:1 33:121M 1214-12114, 2020.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text- based real image editing with diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018
Florian Knoll, Jure Zbontar, Anuroop Sriram, Matthew Muckley, Mary Bruno, Aaron Defazio, Marc Parente, Krzysztof J Geras, Joe Katsnelson, Hersh Chandarana. et al. fastmri: A publicly available raw k-space and dicom dataset of knee images for accelerated mr image reconstruction using machine learning. Radiology: Artificial Intelligence, 2(1) 190007, 2020.
13

---

Published as a conference paper at ICLR 2025
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-8)) 86, 1951.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019a.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, volume 32, 2019b.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances neural information processing systems, pp. 6402-6413, 2017.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference machine learning, pp.
PMLR, 2022.
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018 URL https:://pperreviiet / forummid--H1vkiI.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
Zhen Liu, Xin Liu, Jin Ma, Hui Gao, al. An optimized computational framework for isolation forest. Mathematical problems in engineering, 2018, 2018.
Lorenzo Luzi, Ali Siahkoohi, Paul M Mayer, Josue Casco-Rodriguez. and Richard Baraniuk. Boomerang: Local sampling on image manifolds using diffusion models. arXiv preprint arXiv:2210.121000 2022.
Henry B Mann and Donald R Whitney. On test of whether of two random variables is stochastically larger than the other. The annals of mathematical statistics, pp. 50-60, 1947.
Frank J Massey Jr. The kolmogorov-smirnov test for goodness of fit. Journal of the American statistical Association, 46(253):68-77) 1951.
Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A non-parametric test to detect data- copying in generative models. In International Conference Artificial Intelligence and Statistics, 2020.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. In International Conference on Learning Representations, 2020. URL https
7/openreview net forum? Btd.://rrmmi---yyyyy----
Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh in Neural Information Processing Systems, volume 35, pp. 10068-10077. Curran Associates, Inc., 2022. URL Dtt.hhttpss//ppcccadddnn.eeeurrp merripss
and Boo__Bencharrss.. pdf
Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alex Alemi, and Joshua Dillon. Density of states estimation for out of distribution detection. In International Conference on Artificial Intelligence and Statistics, pp. 3232-3240. PMLR, 2021.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jacjun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7176-7185. PMLR, 2020a.
14

---

Published as a conference paper at ICLR 2025
Muhammad Ferjad Naeem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7186-7185. PMLR, 2020b.
Eric Nalismick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using typicality. arXiv preprint wwrXiv:900.000 2019.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan, Detecting out- ofdistribution inputs to deep generative models using typicality. arXiv preprint 1rXiv:1906.020 02994, 2020.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv:2103.0384/ 2021.
Michael Nevitt, David Felson, and Gayle Lester. The osteoarthritis initiative. Protocol for the cohort study, 1:2, 2006.
Maxime Oquab, Timothee Darcet, Theo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:3304.07193, 2023a
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.071993 2023b.
Kai Packhäuser, Lukas Folle, Florian Thamm, and Andreas Maier. Generation of anonymous chest radiographs using latent diffusion models for training thoracic abnormality classification systems. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pp. 1-5. IEEE, 2023.
Jaewoo Park, Yoon Gyo Jung, and Andrew Beng Jin Teoh. Nearest neighbor guidance for out-of- distribution detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1686-1695, 2023.
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathemat- ical statistics, 1365)61005-1076, 1962.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 8748-8766 PMLR, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021b.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for caat-o--distrbuttinn detection. In Advances in Neural Information Processing Systems, pp. 14680-14691, 2019.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan, simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2/06.00022, 2021.
Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Aniket Roy, Anshul Shah, Ketul Shah, Anirban Roy, and Rama Chellappa. Diffalign: Few-shot learning using diffusion based synthesis and alignment. arXiv preprint arXiv:22 22500444,, 3, 2022.
15

---

Published as a conference paper at ICLR 2025
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance metric for image retrieval. International journal of computer vision, 40:99-121, 2000.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucie, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, volume 31, 2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in Neural Information Processing Systems, volume 29, 2016.
Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning (s) from synthetic imagenet clone. 2022.
Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of high-dimensional distribution. Neural computation, 13(7):1443-1471, 2001.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systemx, 35:25278-252944 2022.
Joan Serrà, David Álvarez, Vicenę Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-based generative models. arXiv preprint arXiv:1909, 11480, 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1/553/v1/P18.. 1238.. URL hntp:tttpss.. /ttc//acaanatt antholooo Logy oxq/P18-1238
Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Diversity is definitely needed: Improving model-agnostic zero-shot classification via stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 769-778, 2023.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? Investigating data replication in diffusion models. arXiv:2212.03866) 2022.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-1244 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances Neural Information Processing Systems, 34:1415-1428. 2021.
Richard Sprinthall. Basic statistical analysis. (No Title), 2011.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metries and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
David Stutz, Matthias Hein, and Bernt Schiele. Confidence-calibbaaed adversarial training: Generaliz- ing to unseen attacks. CoRR, abs/1910.06259. 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2013.
16

---

Published as a conference paper at ICLR 2025
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2016.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmenta- tion with diffusion models. arXiv preprint urXiv:2302.07944 2023.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models, diffusers 2022.
com com/huggiingffaee
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4921-4930, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. Transformers: -fate-of-the-ar natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Shin't ya Yamaguchi and Takuma Fukuda. On the limitation of diffusion models for synthesizing training datasets. In NeurlPS 2023 Workshop on Synthetic Data Generation with Generative AI, 2023.
Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprint arXiv: 1./.v88///108833 088839, 2018.
Jingyang Zhang, Jingkang Yang, Pengyun Wang. Haoqi Wang. Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. OpenOOD v1.5: Enhanced benchmark for out-of-distribution detection. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2024a. URL https 7/openreview nwt. forum ttte://fooundddddaaaaa...ii
Mingtian Zhang, Andi Zhang, and Steven McDonagh. On the out-of-distributtin generalization of probabilistic image modelling. Advances in Neural Information Processing Systems, A4:3311-38223 2021.
Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. Advances in Neural Information Processing Systems, 36, 2024b.
Qihao Zhao, Yangyu Huang, Wei Hu, Fan Zhang, and Jun Liu. Mixpro: Data augmenta- tion with maskmix and progressive attention labeling for vision transformer. arXiv preprint ar04v:2304. /2043, 2023.
Dominik Zietlow, Michael Lohaus, Guha Balakrishnan, Matthäus Kleindessner, Francesco Locatello, Bernhard Schölkopf, and Chris Russell. Leveling down in computer vision: Pareto inefficiencies in fair deep classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10410-10421, 2022.
17

---

Published as a conference paper at ICLR 2025
A APPENDIX: SYNTHETIC DATA SAMPLES
(d)
(f)
AMYGNINDSS
Figure 2: Synthetic images generated for the class "ambulance" using various techniques. Row (a) shows real images, while rows (b) to (f) display Img2lmg generated images with strength parameters 0.3, 0.5, 0.7, 0.9, and 1.0, respectively. Row (g) presents images generated using caption descriptions of the real images in row (a), and row (h) shows images generated solely based the classname.
18

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(d)
(e)
(g)
(h)
Figure 3: Synthetic images generated for the class "golden retriever" using different techniques. Row (a) shows real images, while rows (b) to (f) display Img2Img generated images with strength parameters 0.3, 0.5, 0.7, 0.9, and 1.0, respectively. Row (g) presents images generated using caption descriptions of the real images in row (a), and row (h) shows images generated solely based on the classname.
19

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(d)
(e)
Vodiiah
Figure Synthetic images generated for the class "volleyball" using various techniques. Row (a) shows real images, while rows (b) and (c) display Img2lmg generated images with strength parameters 0.3, 0.5 respectively. Row (d) presents images generated using caption descriptions of the real images in row (a), and row (e) shows images generated solely based on the classname. Row (e) exhibits mode collapse properties.
20

---

Published as a conference paper at ICLR 2025
PCA of Image Features
Image Tyee
anazing s-0.2
engting 5-0.5
maziing s=0.7
ingzimg 5-033
ingling
caption
Coss
Companene
Component
Figure 5: PCA of Golden Retriever Images in CLIP space. We can clearly observe that the distribution of real images is most diverse, whereas the "classname" based generated images are least diverse. We also see progression of images tending towards the modes of the distribution with increasing strengths allotted to the diffusion process.
B APPENDIX: BASELINES STATISTICAL TESTS
Table 6: Comparison of statistical measures for assessing distribution similarity across different image generation methods. The table presents results for various statistical tests and metrics, including Z-score, Kolmogorov-Smminu (K-S) test, Mann-Whitney U test, Local Outlier Factor (LOF),, Isolation Forest (IF), and Wasserstein distance. The image generation methods evaluated include Img2Img with varying strength parameters (S=0.3, 0.5, 0.7, 0.9, 1.0), caption-bass generation, and class-based generation for Golden Retrievers. KL Divergence and Jensen Shannon Divergence were calculated on MSN embeddings, since CLIP embeddings had no overlap.
7-Score
K-S Test
M-W U-Tel
LOF
IF
JSD (MSN)
KLD (MSN)
Wass
Mahulanobis
stat
p-val
stat
p-val
Img2img Sm1L3
-1.176-9
4.17633
3.30c44
1.27e+11
6.64-11
31.118
6.11%
2.56-1
3.46-3
5.52o-3
4.38c=1
Img2imng S=0.5
4.82e-9
4.72e-3
291e-5
2.76++10
2.15e:
27,825
5.97%
-1.81e3
2008e
6.19e3
4.78c=]
Img2ing S=1.7
3.80--9
4.20633
293e-4
1.28++11
2.65e2
6.32%
2.236-3
2.36e-3
16e-3
4.61e=1
Img2ling S=0.9
1.09-4
1.0e
S=1.0
793c-9)
4.58-3
5.496.5
1.27e+11
3.44o-1
34,0995
NaN
NoN
6386-3
4.03cs1
Caption- based
5.56e-9 9
1.02-22
7.366-23
1.29c+11
1.856-29
31,8398
6.38%
NaN
NoN
1.39e-2
4.08ce1
Class-hased
.2.98--1
1.176-2
5.196-30
1.29
3.26c-00
40.639
7.39%
NuN
NoN
1.600-2
3.43+41
21

---

Published as a conference paper at ICLR 2025
1.00
citer
(a) Normalized Mahalanobis Distances for gener- ated distributions vs. real distributions, for the class "golden retriever".
Generre MIC: Desribudion 5-6.3
milopar
pam
313
(b) PRDC Value distributions for golden retrievers, with generated images using Stable Diffusion 2 Base with strength = 0.3.
Figure 6: Comparison of Mahalanobis Distances and PRDC Value distributions for real and generated images of golden retrievers.
(b) UMAP of Volleyball Images. Mode collapse clearly observable for classname generated images.
(a) UMAP of Golden Retriever Images in CLIP space.
Figure 7: UMAP visualizations of Golden Retriever and Volleyball images in CLIP space, highlighting mode collapse in classname generated volleyball images.
22

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of statistical measures for assessing distribution similarity across different image generation methods using CLIP embeddings. The table presents results for various statistical tests and metrics, includ- ing Z-score, Kolmogorov-S Smimmoy (K-S) test, Mann- Mann-Whitne Whitney U test, Local Outlier Factor (LOF), Isolation Forest (IF), and Wasserstein distance. The image generation methods evaluated include Img2Img with varying strength parameters (S=0.3, 0.5, 0.7, 0.9, 1.0), caption-based generation, and class-based generation for the Volleyball class. KI. Divergence and Jensen Shannon Divergence were calculated on MSN embeddings, since CLIP embeddings had no overlap.
LOF
JSD
p-val
p-sal
Img2ling S=0.3
2.25p 9
4.14e:3
3.47e44
1.29++11
1.4e
27.89%
4.54%
NaN
NaN
4,576:3
4.90e+
Inngling S-0.5
-2.15e-9
3.02e-3
2666-3
1.30e411
1.27e-1
22.98%
4.00%
-9.28e-3
-7.38--5
5.75e-3
5.7041
Imgg2ing S-0.7
6.11e-9
7.99e-3
2.16e-14
1.30e+11
1.00e88
20.13%
3.67%
-900e-3
-5.44-55
3.80e-2
1.08e12
1mg2imng S-0.9
1.88e-8
1.13e-2
1.4p-288
1.31e-11
1.00-16
19.06%
2.59%
NoN
NAN
1.120-2
6.6641
Imglimg S=1.0
2.06c 9
1,22e-2
6.56e-33
1.3e-111
4.99-117
19.42%
2.26%
NaN
NaN
1.23e-2
6.69e11
Caption-beed
6.77c-9
6.91e-
7.12e-11
1.30e+11
3.46-100
30.89%
5.27%
N&N
NaN
7,7ee3
4.31e+1
2.82e-154
1.32e+11
C APPENDIX: THEORETICAL PROPERTIES OF PER-POINT PRDC
Theorem C.1. Under the following assumptions:
1. Feature Space: X = RRRD where D is the dimensionality.
2. In-Distribution (ID) Data:
Training Set: Xtrain {x1, ,.....,,,,.... NNam) with x2 ((mm,
Test Set: X {x(,- xNenn with x! N(PID.
3. Distr-btiittiittiionn (OOD) Data:
Test Set: Xoon {x" x"" Now with x" N(Hoon, I21). where A P HOOD >> O.
4. Distance Function: Euclidean distance d(x, y) = 11x--18 y112.
5. k-Nearest Neighbors: For a point x, NN+(x) denotes its k-nearest neighbors in X train
ID
Using previous definitions of Per-Point PRDC Metrics:
1. Precision per point (P(x')):
d(x',x) S ra(x))
P(x')
min
x
where rx(x) is the distance from x to its k-th nearest neighbor in X Ximain
2. Recall per point (R(x')):
R(x')
Σ
x rx(()))
= Neain
3. Density per point (D(x')):
D(x')
Σ
I[d(x', :))=))3)) & rx(x)).
N
xE Xamm
23

---

Published as a conference paper at ICLR 2025
4. Coverage per point (C(x')):
C(x') =I min d(x', ((xxx)) S nx(x2) Xx
where rx(xx)) is the distance from x' to its k-th nearest neighbor in Xutain
Then, the expected values and variances of these per-point PRDC metrics are:
For ID Test Samples (x' E ):
1. Expected Precision per point:
E[P(x') =1 e
Expected Recall per point:
E [[[(x)) 2J k
Nirain
3. Expected Density per point:
E[D(x') = 1.
4. Expected Coverage per point:
[[C(x')] 2 -
For OOD Samples (x" E XooD):
Expected Precision, Recall, Density, and Coverage per point are all approximately zero
Variances:
Precision and Coverage per point:
Var(P(x()] [[(((()) -1-
R(P(x')))
Recall per point:
Var[R(x')] E(R(x') (1 - E[R(x)))
Nmrin
Density per point
Var D(x') EiD(x') (1 EiD(x)))
k
Proof. Let us first clarify some assumptions and notation, specific to this proof for clarity.
Feature Space: X -
2. ID Data Distribution: N (HID r2)).
3. OOD Data Distribution: (HOOD, o²!), with the mean difference A = || WOODl
> 0.
4. Sample Sizes:
(a) Numin Number of ID training samples.
(b) Nust Number of ID test samples.
(c) NOOD: Number of OOD samples.
5. Distance Function: d(x,y) = ||x - y/|2-
6. N-Nearest Neighbor Distances:
(a) rx(x): Distance from x to its k-th nearest neighbor in Xtrain
(b) (x'): Distance from x' to its k-th nearest neighbor in Xttain
24

---

Published as a conference paper at ICLR 2025
Calculating Distribution of Distances
Distance Between ID Samples
Let x,y y (HDD I)) The difference - (O, 2o²1).
The squared distance (x,,)) y) is the sum D squared normal variables:
D (xx, Σ
( - yi)
1=1
where, each (z; V/) is distribution with D degrees of freedom.
(O0 Therefore,
d(x,y) 2n
xi iec. follows a chi-squared
Expected and Variance of the Squared Distance:
E (x. =202D.
Var Varld = (2 2D = 42D( = 80 D.
Distance Between ID and OOD Samples
Let x ~ N(HD, I² and x" ~ ((HooD o I).).
The difference x - x" ~ N6, 20 I), where ö = Pn POOD.
The squared distance follows a non-central chi-squared distribution:
d (xx,x)
xb (A),
20
where = ||8 -0
Therefore, the Expected Squared Distance of non-central chi-squared distribution is:
Eid (x,")x = 20 D + A) 20 + A
And the Variance of Squared Distance of a non-central chi-squared distribution is
Varld (x,x") = 42D + 42) = 80 D + 16o
Expected k--Nerest Neighbor Distance (x)
Distances from to other training samples are i.i.d. and follow the distribution of d(x, y) as previously defined.
Empirical CDF of Distances:
F(d) = P dd)
Expected k-th Nearest Neighbor Distance:
1. For large D (number of dimensions in the vector), the chi-squared distribution can be approximated by normal distribution:
ND,2())
2. This yields standard normal statistic
2
2
22D
25

---

Published as a conference paper at ICLR 2025
3. Since p-value of test statistic follows a standard uniform distribution under the null hypothesis:
d D
+
2
el(0 1)
V2D
f is a monotonically increasing function of variable X whose kth order statistic in sample of size n is X(k), then the kth order statistic of f(X) is f(X()) when X is taken from the sample. Note that & is monotonically increasing as is a CDF, and so
X²
D
f(X)
2
on the positive reals which are the domain of rx (x).
v2D
de
D
By definition d(k) rx(() for some x. Then if
2n
is standard
V2D
uniform, its kth order statistic in the sample is
(rx(x))² 2
D
U(x) +
V2D
6. The keth order statistic of standard uniform follows a Beta distribution with a k and 3 = n k 1:
(x))
D
20
Beta(k,, n K + 1)
V2D
7. Because the expected value of Beta distribution is at.-
(ra()))²
D
E
2n
22D
Expected PRDC Metrics for ID Samples Expected Precision per point (E()(x)))
Definition:
E[((x')) E min P ((d(x(x)<Txxx))
S
II
(1
P x) S (x))))
X
XE ID
Also:
d(x'xx) 2n²
(ra(xi)² D
(ru (x)) D
D
(d(x',x) S rx(x)))
V2D
V2D
2DD
Probability for Single Training Sample, using the above Beta distribution formula, is
(z) D
)
k Niran
E <x) E
(xx
+
2n
2D
Expected Precision per point:
Because we are dealing with in-distribution data and p-values are uniformly distributed under the null hypothesis,
Nuw
A
E B[ =1
~ 21-
-
Nurai
since,
n41
lim
(1-
k
lim
nt1
e
k
n +
1 n+
26

---

Published as a conference paper at ICLR 2025
Expected Recall per point (x()())))
Definition:
R(x') Nuain
[d((' x) S (xx)x)
XEXp
Understanding [xxx:): For each x', (xx))) is the distance to its k-th nearest neighbor among the Nusin training samples. The distances d; = d(x', x;) for 1.... Nuain are independent and identically distributed (i.i.d.) random variables because x' and x; independent samples from the same distribution.
Probability that a Training Sample is within rx(')): The variable [d(x', xx) S Tu (x']] is indicator that if x; among the nearest neighbors of x' the training set. The probability that particular training sample x, is within the k k-nearest neighbors of x' is:
k
P (d(x', x1) S rx(x)))
Nirain
This is since nearest neighbors are chosen among Nusin samples, each training sample has equal chance of & of being among the closest & samples to x'. Nom
[ x:) S (x()x)]]
Viain
N
1
k
B[R(x()
[T x:) [xx(x')]]
E
x Nirain X
N Nirain
Nerain
N Nirain
N
Vrain
i=1
k
E[R(x')]
N N
Expected Density per point (xx))
Definition:
D(x)
Σ
K
X
Expected Sum, for large N:
k 2 k.
E
Σ I[d(x'x) S rx(x))
Nuaio
Nirain +
XEX
Expected Density per point:
E(D(x')) A
Expected Coverage per point (E[C(x'))
Given both x' and the training samples x are drawn from the same distribution N ((AD, T²T). is the smallest among Nmmi distances drawn from
The minimum distance minx, the chi-squared distribution.
Distance to k-th Nearest Neighbor rxx')) rx(x') the k-th order statistic of the distances d(x', xx). We are interested in the probability:
min d(x', xx) S rx(xx)
27

---

Published as a conference paper at ICLR 2025
Since the minimum distance is always less than equal the k-thh smallest distance, the event is always true:
min ((x'd S (xx)
Therefore:
P min d(x din S rx(xx)) =1
This is because the closest training sample x' is by definition among the k-nearest neighbors used to compute (x'.. Therefore:
[[((x')) =
Expected PRDC Metrics for OOD Samples
Due the large distance A, the probability that an OOD sample is within (x(x) of any ID training sample is negligible.
Therefore:
1. [[((x)] 2 0.
2. [[[(x") 2 0.
[xD("") & 0.
B[C(x") 0.
Variances of PRDC Metrics
Variance of Precision and Coverage per point
1. Since these Bernoulli random variables:
Var[P(x')]
=e -2k
2. The variance of coverage per point for ID samples is zero. This reflects the certainty that ID test samples will always satisfy the coverage condition.
Variance of Recall per point
1. Recall per point is an average of Nurii Bernoulli variables with success probability
k
p
Nurai
2. Because the sum of Nusin Bernoulli variables with probability p is binomial distribution and are dividing that by Nirain the Variance becomes:
p(1- - p)
Var[R(x())]
=
N2
x Nurai x xp - p)
Nuran
urin
Variance of Density per point
Density per point sums Nuni Bernoulli variables and divides by kt
Variance:
1 E2 x Nuai x p1 - p) p(1 - p) Nurin
Var(D(x')]
1.2
Since p N. we get:
le
VarlD(x))
R To
N
28

---

Published as a conference paper at ICLR 2025
Didtis didi
Der tecal
CerrRUtse Oeiddt
Durisden Covenge
0.66
1453 600 1.044
340 485 CITA LAT
Figure 8: Distribution of PRDC metrics for the scenario where both the training (ID) and test (ID) data are drawn from the same Gaussian distribution with zero mean. Blue lines represent the ID data, and red lines represent the OOD data (which is actually the same ID in this case). The histograms overlap entirely, indicating identical distributions for metrics due to the same underlying data.
Delrditee Deadd
Didetizenn a Doenage
340 600 1904
Figure 9: Distribution of PRDC metrics for the scenario with moderate shift, where the OOD test data is generated from a Gaussian distribution with its mean shifted by 0.01 units compared to the ID data. Blue lines represent the ID data, and red lines represent the moderately shifted OOD data. The histograms show noticeable differences and partial overlap between ID and OOD distributions, reflecting the moderate shift in data.
For large Nuin this simplifies
1
Var[D(x))] ~ T
For ID samples, per-point Precision and Coverage have expected values close 1 which increases with k.
Per Per-point Recall is small when k Nurin with E[R(x')] Nusi
Per-point Density is approximately 1 for ID samples, regardless of k.
For OOD samples, all per-point PRDC metrics approximately zero due to the large mean difference A.
Variances depend on ls and Nuain but are generally small for large datasets.
Note Assumptions and Approximations
Independence Assumption:
While distances not strictly independent, the independence approximation simplifies calculations and reasonable for large datasets.
Large D Approximation: Approximations using the normal distribution for the chi chi-squared distribution valid when D is large.
Small & Relative Numin: The results most accurate when & Nuran-
29

---

Published as a conference paper at ICLR 2025
Der
DAATIS Derily
D51 Dutitaatin Caverage
0--
0.00
Figure 10: Distribution of PRDC metrics for the scenario with a large shift, where the OOD test data generated from Gaussian distribution with its mean shifted by units from the ID data. Blue lines represent the ID data, and red lines represent the significantly shifted OOD data. The histograms display minimal overlap, indicating that the PRDC metrics effectively distinguish between the ID and heavily shifted OOD data.
Factzon
Secalper
Number
6.00
Line plots showing the mean PRDC metrics Precision, Recall, Density, and Cover- age-per data point as a function of the number of samples (N) for varying values of (number of nearest neighbors). Each subplot corresponds representing k=3, k=5, and k=10. The semi-logarithmic x-axis emphasizes the impact of sample size on the stability and reliability of the PRDC metrics across different neighborhood sizes. Notice that in precision and coverage, the variance goes down considerably (theoretically, exponentially) on the basis of k, supported by our theorem.
Figure 11:
of the PRDC metrics, with different lines
30

---

Published as a conference paper at ICLR 2025
C.I ROBUSTNESS TO THE CURSE DIMENSIONALITY
The curse of dimensionality often manifests as the concentration of distances in high- dimensional spaces. By reducing four scalar metrics, PRDC sidesteps many of these issues. Let X, be uniformly distributed in the n-dimensional unit ball. As n + oo: maXpcicjsk [X,=X,] 5 1 for any fixed As the number of dimensions increases, the ratio of the maximum minimum distances between any pair of k points approaches in probability. This means that the distances between points become almost the same, regardless of their specific locations in the unit ball. This contributes to why directly using high dimensional representations does not help.
Intuition:
mingg:<ga X,
PRDC metrics are not directly affected by this distance concentration, as they capture relative rather than absolute distances. These metrics provide more meaningful comparisons between datasets, especially in high-dimen high-dimensional spaces where traditional distance measures lose their discriminatory power. By emphasizing relative rather than absolute distances, PRDC metrics are less affected by the concentration of distances in high-dimensional spaces.
Experiment: We want to understand how increasing the dimensionality of data affects the ability of PRDC metrics and other measures to distinguish between inliers and outliers.
We the following methodology for data generation:
Inliers: For each dimension ranging from 2 to 200 (in steps of 5), we generate 1000
samples from D-dimensional standard normal distribution centered at the origin:
Xinlier ~ (0,
where In is the D D identity matrix.
We generate 100 samples from D-dimensional normal distribution centered at μ
[3 3J:
Xontlier (N(31,1))
where is a vector of ones.
We calculate the PRDC per point metrics, and then the following additional measures:
Mean Euclidean Distance:
1
Mean Distance
x 2
N
where N the number of inliers.
Mean Cosine Similarity:
2 N(N 1)
x
Mean Cosine Similarity
X 1| |2
PCA Variance Explained: Sum of variance explained by the first two principal components obtained from Principal Component Analysis (PCA) on the inliers.
We iterate over each dimension D and:
1. Generate inliers and outliers.
2. Compute PRDC metrics for inliers vs. inliers and inliers vs. outliers.
3. Compute additional measures.
4. Record the results for analysis.
Observations
Precision:
Inliers exhibit high precision across dimensions, indicating that inliers are close to each other. For outliers, precision decreases with increasing D, and then stabilizes, due to increased distances in high-dimensional spaces.
31

---

Published as a conference paper at ICLR 2025
Soul Diecmiaatty
Deeeinully
404
0000
6.03
00000
Figure 12: Precision, Recall, Density, and Coverage (PRDC) metrics plotted against the number of dimensions for Inlier (ID) and Outlier (OOD) datasets. In each subplot, the blue line represents the average PRDC metric for inliers compared against themselves, while the orange line depicts the average PRDC metric for outliers compared against inliers. This figure illustrates how increasing dimensionality impacts the effectiveness of PRDC metrics distinguishing between inliers and outliers across different degrees of distributional shift.
Eacidian
PCA No4 aapeeeeeeerereeeeeee..ee wo
(a) Average Euclidean Distance from the Origin
(b) Total variance explained by the first two Principal
and Average Cosine Similarity among Inliers plot-
Components (PC1 and PC2) of Inlier (ID) datasets plotted against number dimensions. The
against the number dimensions Inlier (ID)
datasets. The left y-axis (colored in green) shows the
line represents the cumulative variance captured by
mean Euclidean distance of inliers from the origin,
the first two principal components. As dimensionality increases, the proportion of variance explained by the first two components decreases, indicating that data variance becomes more dispersed across additional dimensions. This trend underscores the challenges of dimensionality reduction in high-dimensional spaces, sidestepped by Forte.
while the right y-axis (colored in orange) displays the
mean cosine similarity among inliers. This dual axia plot highlights how higher dimensionality leads to increased Euclidean distances and decreased cosine similarities, reflecting the effects of the curse of di- mensionality data geometry.
32

---

Published as a conference paper at ICLR 2025
Recall: Inliers show higher recall, reflecting better coverage among inliers. Outliers on the other hand show lower recall, as outliers do not cover the inlier distribution well.
Density: Inliers demonstrate high density, indicating dense clustering. For outliers, density decreases with D, showing sparse connections to inliers.
Coverage: Inliers exhibit high coverage, demonstrating that they effectively cover the inlier distribution. Outlier coverage decreases with D, indicating poor coverage of inliers by outliers.
Mean Euclidean Distance: This metric increases with D for both inliers and outliers, due to the phenomenon where distances between points in high-dimensional spaces tend to increase.
Mean Cosine Similarity: This measure decreases with D, approaching zero for both inliers and outliers, as random vectors in high dimensions become nearly orthogonal.
PCA Variance Explained: This metric decreases with D for both inliers and outliers, as variance is distributed among more components, making harder to capture significant variance in the first few components.
C.2 PRDC AS LOCAL TRANSFORMATIONS
A local transformation is function applied to vector in a space which depends not only the vector itself but also on its neighboring vectors. This allows for contextual dependence. where the transformation depends on the local environment of the vector, therefore making it sensitive to local variations and patterns. This also allows the local transformations to be non-linear, allowing for complex manipulations that not possible with global linear transformations. This means they can adapt to regions of the data space, capturing variations and features that might be missed by global transformations.
These are obviously very powerful, as evidenced by convolution kernels (depending on neighbour pixels), Graph convolutions, Laplacian smoothing, and wavelet transforms. However, they can be sensitive to parameters like kernel size, weights, and tuming is important to ensure robustness. PRDC metries might look very different at different values, for example.
C.3 KEY PROPERTIES AND CHARACTERISTICS OF THE FORTE ALGORITHM
At its core, Forte leverages locality, basing per-point PRDC (Precision, Recall, Density, Coverage) metrics on local neighborhood structures. This approach makes the algorithm sensitive to local variations in data distribution. Simultaneousll, Forte performs dimensionality reduction, compressing high-dimensional data (D >> 4) into a more manageable g space while preserving essential information for OOD detection. This reduction not only aids in mitigating overfitting but also improves generalization and computational efficiency.
key strength of Forte lies its model-agnostic nature, allowing it to work with any feature extractor that provides meaningful representations in RDD Self-supervised models like CLIP, ViTMSN, and DINOV2 are particularly effective in this context due to their rich feature representations. The algorithm's paramerraetic approach, which avoids assumptions about the data distribution's 's form, contributes to its flexibility and robustness. This is further enhanced by the tunable parameter k (number of nearest neighbors), which allows for balancing sensitivity local structures with noise robustness.
Forte's effectiveness in OOD detection stems from its ability to capture both density (proximity of neighbors) and coverage (sample's position within the support of PID). By focusing on local neighborhood information, the algorithm can detect subtle discrepancies between in-distribution and OOD samples, particularly effective when OOD samples reside in low-density regions. The use of multiple metrics (precision, recall, density, and coverage) provides a holistic view of how each sample relates to the training data Xtrain enhancing the algorithm's 's discriminative power. The local transform employed by Forte also confers robustness to feature space variability, an important consideration when working with different self-supervised learning models. By normalizing differences through focus on relative distances within the feature space, Forte maintains consistency across varied feature representations. Empirical results presented in this paper have comprehensively demonstrated Forte's
33

---

Published as a conference paper at ICLR 2025
superior performance compared to traditional methods, showcasing how the combination of powerful feature extractors and the local transform leads to effective OOD detection.
D APPENDIX: ABLATION STUDY ON ENCODERS
Insight into encoders: The results presented in Table. 3 demonstrate that richer representations significantly improve OOD detection performance, as evidenced by the ranking of encoders: CLIP DINO v2 > MSN when used individually, and CLIP DINO v2 CLIP + MSN > DINO v2 + MSN the two-model combinations. To further investigate this phenomenon, we conducted additional experiments. Specifically, included DeIT with both ViT-B (Base) and ViT-Ti (Tiny) models and evaluated their OOD detection performance under the settings studied in Tables 2. These results, show that DeIT-B achieves 0.87 AUROC on CIFAR-100, while DelT-Ti achieves 0.82 AUROC. This aligns with our hypothesis that more informative representations are essential for effective OOD) detection. DelT, trained with objective approximating supervision, produces less informative embeddings compared to self-supervised encoders like DINO v2. Similarly, the DelT-Ti model performs worse due to its reduced capacity for generating robust representations. We think these findings provide valuable insights into the utility of different encoders for OOD detection and offer guidance for practitioners seeking optimal performance.
Table 8: Comparison of AUROC and FPR95 perfor- mance figures for Base and Tiny DelT models across the tasks
Model
In-Dist
OOD Dataset
AUROC
FPR95
Base-DelT
CIFAR-I0
CIFAR-100
0,8712
0.9926
Tiny-DelT
CIFAR-10
CIEAR-100
0.8261
0,9903
DelT
CIFAR-10
0,9554
Tiny-DelT DelT
CIFAR-
SVHN
0,9296
0.6195
Base DelT
CIFAR-10
Celeb-A
0,9871
0.0015
Tiny DelT
CIFAR-10
Celeb-A
0,9929
0,0007
E APPENDIX: MEDICAL IMAGE DATASETS
In medical imaging research, studies are often done using one in-house dataset. Conclusions and models drawn from these studies are then applied to new data, with poor results. In particular, MRI datasets exhibit strong batch effects that prevent them from being in-distribution relative each other because some acquisition protocol is bound to be different between them. Moreover, dataset sizes are severely limited clinical applications, which means separate model cannot be trained for each batch. A single model cannot be robust to all MRI datasets of the same subject matter, even if they have similar acquisition parameters. Such datasets still carry enough differences to impact model performance.
To simulate this scenario. two public datasets are used for the experiments in Section 5.2 coronal knee MRI from FastMRI Zbontar et al. (nnol et al. (2020) with two subsets and Ostcoarthritis Initiative (OAI) Nevitt al. (2006) with three subsets. The acquisition parameters, including sequence and fat suppression are detailed in Table samples are shown in Figure [4] Treating the FastMRI dataset as in-distribution and assuming that models have been trained on them, Forte used to determine the next course of action when confronted with the OAI dataset: 1) which subsets of the new dataset can be aligned with the existing subsets/ models? 2) To what degree do these subsets diverge from each other? Using the FastMRI FS subset as in-distribution, the two OAI subsets (OAI T1 and OAI MPR) are tested for OOD detection. Similarly, the FastMRI NoFS subset is used as in-distributios and the OAI TSE subset is tested for OOD detection.
34

---

Published as a conference paper at ICLR 2025
Table 9; Acquisition parameters for MRI, grouped by distributions used in Section
5.2
FastMR NotS
FastMRI PS
OAI
OAI MPR
Sequence
2D TSE PD
2D ISE PD
TSE Tlw
FLASH T
DESS T2w
(mm")
140
Matrix size
320 x 320
320 320
320 x 320
384 384
320 x 320
Slice thickness (mm)
3
0.7
1.5
TR (ms)
2750 3750-3000
2850 2850-3000
800
14.7
TE: (ms)
27-32
9
4.2
Fat suppression
No
Yes
No
Yes
(a) FastMRI FS OAITI
(c) MPR
(d) FastMRI NoFS
(c) OAI TSE
Figure 14: Sample images from five total subsets of the OAI and FastMRI datasets. Fat-suppressed (a-c) and non fat-suppressed (d,e) subsets tested for OOD.
35

---

Published as a conference paper at ICLR 2025
F PSEUDOCODE
Algorithm (OD Detection Using Per-Point PRDC Metrics Forte
Input:
Reference data features {
Input: Test data features {=+]
Input: Number nearest neighbors &
Output: OOD detection performance metrics: AUROC, FPR @95
1: Feature Extraction (Preprocessing):
Use pre-trained models (c.g.. CLIP, ViT-MSN, DINOv2) to extract features for both reference and test data.
Reference features: {-7))=1
Test features: { 12
2: Compute Nearest Neighbor Distances:
3: for j = to do
4:
Compute (T))( distance to its k-th nearest neighbor in {2), mm
1=
5: end for
6: for i 1 do
7: Compute (xf): distance to its k-th nearest neighbor in {2), 1n
8: end for
9: Compute Per-Point PRDC Metries for Test Data:
10: for i = 1 to n do
Compute per-point metrics for xa relative {a})>=
Precision per point:
precision (i) pp
= 11( E S({=))7=)))
where ((x=));=1)
- B NNDD&(F)).
2. Recall per point:
recall pp
Σ m
1
E BCE (x?))
3. Density per point:
1 m Σ
density pp
(a E B NND,
km 1=1
4. Coverage per point:
coverage
(1)
min ||> 2'J1 NNDx j=1....,m
12: Assemble feature vector (6())
precision
( pp recallll (1) density pp: coverage
(8)
end for
14: Prepare Reference Training and Validation Sets:
Split { into training set (a:trin) and validation set {a=yalidg.
16: Compute Per-Point PRDC Metrics for Reference Training Data:
17: for each Irain do
18: Compute per-point metrics ()) following similar steps as above, relative to {a Minis
Jh=1
19: end for
20: Train Anomaly Detection Models:
21: Use the reference per-point metrics (Ø (1) to train unsupervised anomaly detection models:
One-Class SVM (OCSVM), Kernel Density Estimation (KDE), & Gaussian Mixture Model (GMM).
22: Evaluate Models on Test Data:
23: for = 1 to do
24: Compute anomaly scores g(t) for (00) using the trained models. Use it to assign Ground
Truth Labels, and calculate AUROC & FPR@95 @95
25: end for
36

---

Published as a conference paper at ICLR 2025
Notes:
1 ((-) is the indicator function, returning 1 if the condition is true, and 0 otherwise.
[zx r) denotes a ball (in Euclidean space) centered at z with radius r.
NNDx (z) is the distance from point its k-th nearest neighbor. ((("))7=)) = U B (xt, N,,NN represents the union of balls around each refer- ence point o3 with radius (x)).
G COMPARATIVE EVALUATION
To contextualize the strong state-of-the performance achieved by Forte, we present the following infographics showing the evolution of methods and their performance benchmark tasks.
ALROC Novr CINR-100 100
AUMOC over YEARS for CORD-
AUMOC Yoars for Swih
1009
2035
3000
0551
2005
2005
3000
1858
2000
2000
2025
2000
3500
2004
2,00
2370
2005
2000
Figure 15: AUROC performance of various unsupervised OOD detection methods over years for the CIFAR-10 10 (In-distribution) and the CIFAR-100(0O)) Celeb-A(OOD) and SVHN(OOD) dataset. The figure illustrates the progression of anomaly detection techniques, with methods represented using unique markers and colors.
AUROC Saae ISScO
Figure 16: Comparison of AUROC performance across various supervised diitti--iitiiittonn de- tection methods and datasets from the OpenOOD leaderboard. The figure presents results for five datasets (NINCO, SSB-Hard, iNaturalist, Textures, and OpenImage-O) with each subplot showcasing the progression of AUROC scores over the years. Each method is represented with a unique marker and color. Our method "Forte + GMM" is highlighted for its superior performance, demonstrating strong fatte-of-the-eee results across all datasets. The x-axis represents the publication year, while the y-axis denotes the AUROC (%) scores.
H MISCELLANEOUS RELATED WORKS
Data augmentation using generative models, particularly diffusion models, has shown promise in improving model performance and generalization. Methods include using pretrained models to generate variations of existing data (Luzi et al. 2022 Sariyildiz et al. (022). fine-tuning models on
specific subjects or concepts (Gal al. 2022
2023
Kawar al. 2023), and generating synthetic
37

---

Published as a conference paper at ICLR 2025
datasets for downstream tasks (Shipard et al. 2023 Roy et al. 2022
Packhäuser al. 2023
Ghalcbikesabi et al. 2023 Zhang et al. 2024b; Trabucco et al. 2023 Karras et al. 2020
Bansa
& Grover] 2023] Akrout et 2023). Other notable examples include TransMix CCen et (2022)
and MixPro iZhao et al. (2023), which have demonstrated strong performance on the ImageNet
classification task, However, the effectiveness of these methods is lower compared traditional
data augmentation and retrieval baselines (Zietlow et al. 20222 Azizi et al. [022] [urg et al., (223))
Potential biases introduced by synthetic data and the need detect out-of-distribution generated samples should be considered when employing these techniques.
The concept of score-based likelihood maximization is fundamental to diffusion models, inherently guiding the reverse generation process towards maximizing the likelihood. This maximization process likely pushes the generated images closer to the dominant modes of the distribution [Song et al. (202T) :
Song & Ermon (020), which has been discussed but has not been explicitly demonstrated as failure mode by Yamaguchi & Fukuda (023).
38


Reviews:
[{"review_id": "uI15epqkvY", "reviewer": "Reviewer_niAs", "summary": "The authors present a methodology, Forte, which enables the identification of outliers using a rigorous unsupervised algorithm.  The algorithm relies on establishing metrics to identify which samples are in-distribution vs out-of-distribution.  The method is very general, and applicable to many models.  The authors use SVM, KDE, and GMM models with their method, and show that it is capable of distinguishing between real and synthetic data.", "questions": "None.  The appendix cleared them up", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very well written, easy to follow, and highly implementable.  An extensive appendix provides supporting information and data.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LYOQntKykq", "reviewer": "Reviewer_GdRX", "summary": "The paper investigates OOD by modeling distributions in feature space. The paper is poorly written, but my guess is that the proposed method is a variant of DoSE, using models _per-sample metrics_ of the feature space, rather than the feature vectors themselves. Four metrics, including precision and recall, are used. Experiments are performed on real images from distinct classes, synthetic vs. real images, and medical images. Results indicate that the proposed method is effective.\n\n----\n\nThe authors' response have addressed most of my concerns. The changes have significantly improved the manuscript. I decide to raise my score.", "questions": "1. My main question is how the proposed method really works. If the authors could provide a pseudocode of the algorithm, and how it differs with DoSE, it would be great.\n2. See above weaknesses.\n3. Overall poor writing and latex formatting, in addition to the issues listed above. E.g., \"in Figure 2 FIgure 3 Figure 4\", \n3. Minor typos: \"near-ood\" -> \"near-OOD\", \"Table 1 & 2\" -> \"Tables 1 & 2\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "[The following is based on my guess of the proposed method, which is not well described in paper.]\n\n+ Proposed approach is a simple change over feature-space OOD methods, and appears effective.\n+ Experiments seems cover a wide range of scenarios", "weaknesses": "+ The paper is extremely poorly written. I list some major issues here.\n    1. None of the math latex in section 3.2 is well formatted. Subscripts and superscripts are wrong.\n    2. Variables are used without definition, e.g., $\\text{nearest}_k$ in section 3.2. Is it different from the $k$ below?\n    3. No description is given on how the four metrics are used. Are they used as the \"summary statistics\" that the proposed method models?\n    4. The method, referred to as \"Forte\", is never truly defined or mentioned in the method section 3.\n    \n+ If my understanding of the method is correct, is it simple just taking DoSE and run it on the new set of statistics?\n+ Incorrect claims. E.g., GMM is not non-parametric, and I don't think that the four metrics are newly proposed in this paper.\n+ Density definition is inconsistent with Fig. 1. As defined, it is just a scaled \"recall\", which would make it useless to model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zkYpXgYwjD", "reviewer": "Reviewer_2mpC", "summary": "This paper introduces Forte, a novel out-of-distribution (OOD) detection framework that leverages self-supervised learners. Forte enhances detection by combining representation learning methods (e.g., CLIP, ViT-MSN, and DINOv2) with non-parametric density estimators (OCSVM, KDE, GMM) to model the typicality of input samples. The proposed framework emphasizes detecting atypical samples through summary statistics (precision, recall, density, and coverage) to analyze representation distributions. Forte’s performance was evaluated on synthetic datasets generated by Stable Diffusion and various medical image datasets, showcasing its advantages over existing supervised and unsupervised methods.", "questions": "Could the computational demands of Forte’s ensemble approach limit its applicability in real-time OOD detection scenarios?\n\nWould Forte’s effectiveness in OOD detection promote if one apply multiple models by one self-supervised approach in this framework?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- **Performance**: Forte demonstrates superior OOD detection performance compared to state-of-the-art methods across multiple benchmarks, including synthetic data and medical image datasets, which often present significant OOD detection challenges.\n- **Flexibility**: Forte’s unsupervised nature eliminates the need for labeled data or pre-exposure to OOD samples, making it adaptable to various tasks and practical for real-world applications where OOD examples may not be available during training.\n- **Comprehensive Evaluation**: Forte is rigorously tested on both synthetic and medical datasets, demonstrating the framework’s versatility and robustness across vastly different domains.\n- **Insightful Metrics**: The use of novel per-point summary statistics (e.g., precision, recall, density, and coverage) contributes valuable insight into data distribution, enhancing OOD detection beyond standard density-based methods.", "weaknesses": "1. **Paper Structure**: The paper allocates a substantial portion of its Introduction to reviewing existing OOD detection literature and explaining the typicality concept. This approach detracts from an immediate focus on the novel contributions and design of Forte, which may hinder reader engagement and understanding of the primary contributions.\n2. **Complexity in Practical Implementation**: The integration of multiple representation learning techniques, combined with non-parametric density estimators, may lead to a higher computational overhead and increased complexity in practical deployment. The paper lacks an illustration figure to clearly explain the proposed framework how to integrate the representations from diverse models.\n3. **Insight.**: This work integrates representations from diverse models empirically. It lacks insight into the choice of self-supervised models, such as whether any specific attributes of CLIP, ViT-MSN, or DINOv2 contribute uniquely to Forte’s robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bJrZJ2Ja2F", "reviewer": "Reviewer_2YAU", "summary": "This paper proposes a method for identifying OOD and synthetic data created using generative models. The definition of OOD changes with the advent of foundation models that can generate very plausible data. However, even when this data seems real, there can still be distribution shifts that are difficult to detect. In this paper, the concept of typicality is used to assess OOD. However, computing typicality is challenging. This paper proposes Forte, a method for computing typicality, and they evaluate it and compare it with other methods.", "questions": "I miss some comparison with DoSE in some table for completeness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clear to understand. The problem is relevant. The method is sound. The experimental framework is correct and complete.", "weaknesses": "In the related work section I miss some comparison of the current SOTA with Forte.\n\nThis method is based on DoSE. I miss a clear statement in the intro about that and the changes over DoSE with an intuition of why. Also, it would be beneficial to clearly state the method to guide the reader on what will come.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)