Two Is Better Than One: Aligned Representation Pairs for Anomaly Detection
Authors: Alain Ryser
Thomas M. Sutter
Alexander Marx
Julia E Vogt
Venue: Accepted by TMLR ()

Abstract: Anomaly detection focuses on identifying samples that deviate from the norm. Discovering informative representations of normal samples is crucial to detecting anomalies effectively. Recent self-supervised methods have successfully learned such representations by employing prior knowledge about anomalies to create synthetic outliers during training. However, we often do not know what to expect from unseen data in specialized real-world applications. In this work, we address this limitation with our new approach, Con2, which leverages prior knowledge about symmetries in normal samples to observe the data in different contexts. Con2 consists of two parts: Context Contrasting clusters representations according to their context, while Content Alignment encourages the model to capture semantic information by aligning the positions of normal samples across clusters. The resulting representation space allows us to detect anomalies as outliers of the learned context clusters. We demonstrate the benefit of this approach in extensive experiments on specialized medical datasets, outperforming competitive baselines based on self-supervised learning and pretrained models and presenting competitive performance on natural imaging benchmarks.

Published in Transactions on Machine Learning Research (09/2025)
Two Is Better Than One:
Aligned Representation Pairs for Anomaly Detection
Alain Ryser
moinnryyseriinfette
Deportment of Computer Science
ETH Zurich
Thomas M. Sutter
Department of Computer Science
ETH Zurich
Alexander Marx
Research Center Trustworthy Data Science and Security of the University Alliance Ruler Department of Statistics TU Dortmand University
Julia E. Vogt
Department of Computer Science ETH Zurich
Reviewed OpenReview: https: http:///ppeeeeeeeeeeeeeee wete://weee//eeeeeeee forum? id=BtOzdsnWWYe
Abstract
Anomaly detection focuses on identifying samples that deviate from the norm. Discovering informative representations of normal samples crucial to detecting anomalies effectively. Recent self-supervised methods have successfully learned such representations by employing prior knowledge about anomalies create synthetic outliers during training. However, we often do not know what to expect from unscen data in specialized real-world applications. In this work, we address this limitation with our new approach CON2, which leverages prior knowledge about symmetries in normal samples to observe the data in different contexts. CON₂ consists of two parts: Context Contrasting clusters representations according to their context, while Content Alignment encourages the model to capture semantic information by aligning the positions of normal samples across clusters. The resulting represent atinn space allows us detect anomalies as outliers of the learned context clusters. We demonstrate the benefit of this approach in extensive experiments on specialized medical datasets, outper- forming competitive baselines based self-supervised learning and pretrained models and presenting competitive performance natural imaging benchmarks.
1 Introduction
Reliably detecting anomalies is essential in many safety-critical fields such as healthcare (Schlegl et al., 2017; Ryser al., 2022), finance (Golnohammadi & Zaiane, 2015), industrial fault detection (Atha & Jahanshahi, 2018; Zhao et al,, 2019), cyber-security (Xin al., 2018). A common real-world example of anomaly detection is the standard screening scenario, where doctors regularly examine general population for anomalies that would indicate a health risk. Standard screening datascts predominantly comprise samples from healthy people, as most screened individuals do not exhibit any diseases. Detecting anomalies in this setting is challenging, as anomalies can arise from an arbitrary set of potentially rare diseases or measurement errors. At the same time, we predominantly encounter normal samples from healthy people the datasct. Anomaly detection methods tackle such problems by learning representations that reflect normality during training and detect anomalies deviations from the learned normal structure test time.

---

Published in Transactions on Machine Learning Research (09/2025)
One way of characterizing the anomaly detection problem is to view it a one-class classification (OCC) problem (Schölkopf et al., 2001; Tax & Duin, 2004). The idea of OCC is to estimate a tight decision boundary around all normal representations and to detect anomalies as samples that do not lie inside this boundary (sce Figure (a)). However, such a decision boundary requires a dense cluster of normal representations, whereas anomalous representations should lie outside this cluster. This requirement is sometimes called the concentration assumption (Ruff et al., 2021). It is generally difficult to formalize an objective that learns representations fulfilling the concentration assumption without observing anomalies during training, as a trivial shorteut is to collapse all representations onto single point (Ruff et al., 2018). While earlier works have proposed various regularization techniques to circumvent this shortcut (Perera & Patel, 2019; Ghafoori & Leckie, 2020), it has recently become popular to learn a decision boundary more explicitly by carefully designing synthetic anomalies (Oza & Patel, 2018; Sabokrou et al., 2020; Tack et al., 2020; Wang et al., 2023). However, anomalies can be diverse and unexpected, making it difficult to simulate them in real-world settings. A more recent line of work focuses on using or adapting the representations from pretrained models (Reiss et al., 2021; Liznerski et al., 2022; Zhou el al., 2024) instead of learning specifie representations for anomaly detection. While these methods are vastly successful on natural imaging benchmark datasets, they may not generalize well to more specialized domains that are underrepresented in the pretraining data, as we will demonstrate in our experiments.
In this work, we present how to learn informative, concentrated representations with CON2', a novel objective to learn representations for anomaly detection targeting specialized domains, where anomalies are challenging to simulate, and large datasets for pretraining are difficult to obtain. CON₂ contrastively learns two separate, concentrated representation spaces from normal training data by leveraging natural symmetries in normal create contert augmentations (examples in Figure 5), allowing to set
data. These symmetrics help samples into distinct contexts. CON₂ clusters representations according to their contexts to create context clusters while encouraging a symmetrical structure of the space (Figure 1). This approach leads to informative representations that are structured according to the properties of normal data. The structure of anomalous samples is typically different from normal samples, which lets us detect these outliers in the representation space learned by CON2- Our method is particularly valuable for specialized datasets, such as the medical domain, where anomalies may be difficult to obtain or simulate.
Our main contribution is CON2, a new approach to representation learning for anomaly detection. We further present context augmentations, which allow us to put samples into different contexts by leveraging symmetries observed in the normal training dataset. Additionally, we show how to use the representations learned by CON2 to detect anomalies using two anomaly score functions. The score function SNND measures sumple-anomaloussees through the nearest neighbor distance to normal training representat ions, whereas the more efficient SLH anomaly score provides a simple likelihood-based alternative. Finally, extensive evaluation on diverse medical-imaging benchmarks demonstrates that learning concentrated representations of normal data with CON2 yields superior anomaly detection performance compared to popular self-supervised and pretrained approaches that depend on assumptions about anomalies or simulated examples.
2 Related Work
Learning useful normal representations of nigh-dimensional data for anomaly detection has recently become a popular line of research. Early works have tackled the problem using hypersphere compression (Ruff et al., 2018). Other popular mothods define pretext tasks such as learning reconstruction models (Chen et al., 2017; Zong et al., 2018; You et al., 2019) or predicting data transformations (Golan & Eaniv, 2018; Hendrycks et al., 2019b; Bergman & Hoshen, 2019). Although these approaches have had some success in the past, the learned representations are often not informative enough for reliable anomaly detection, as there is typically discrepancy between the pretext task and learning to characterize normal samples. More recently, progress in self-supervised representation learning led to new methods that learn more expressive normal representations through contrastive learning (Sun et al., 2022; Sehwag et al., 2021), improving upon prior work. Methods such as CSI (Tack et al., 2020) and UniCON (Wang et al., 2023) further refine these representations for anomaly detection by introducing simulated anomalies as negative samples.
We provide our code https://gåthhbe htt.://....oo.laaaiineerrrrraaaa
2

---

Published in Transactions on Machine Learning Research (09/2025)
Normal Trainset
Anomalous Testset
Trainingset Context
Normal Context 0
Anomalous Context
Normal Testset
Decision Boundary
Trainingset Context
Normal Context 1
Anomalous Context
x
(a)
(b)
(c)
Figure I: Structure of the representation space in traditional one-class classification (a). Figures (b) and (c) contain two-dimensional PCA embeddings of representations of the train (+), normal test (+), and anomalous test (x) samples after training CON₂ on normal samples of the BR35H dataset using the invert context angmentation. CON₂ creates two compact, distinct, and aligned context clusters by leveraging symmetries of normal data. Fach line corresponds to the position of an original BR35H sample (left.) and its context. augmented counterpart (right). Parallel lines indicate alignment of the representations of normal test samples, which anomalies fail to achieve.
A separate line of work focuses on estimating the training density with the help of generative models, detecting anomalies as samples from low probability regions (An & Cho, 2015; Schlegl et al., 2019; Nachman & Shih, 2020; Mirzaci et al., 2022). However, these methods tend to generalize better to unseen distributions than to the observed training distribution (Nalisnick et al., 2018), which often harms anomaly detection performance.
Recently, leveraging existing large models pretrained on big, usually unrelated datasets has become a popular approach to tackle anomaly detection. Some methods have been introduced that use representations from such models directly in a zero-shot fashion (Bergman et al., 2020; Liznerski al., 2022; Jeong et al., 2023), while others demonstrate the benefit of fine-tuning (Cohen & Avidan, 2022; Reiss & Hoshen, 2023; Li et al., 2023; Zhou et al., 2024).
In addition to the traditional setting, where we assume training datasets without any labels, some works started to assume having access to a limited number of labeled samples from the same distribution as the training data. This setting is called anomaly detection with Outlier Exposure (OE) (Hendrycks et al., 2019a), and it has been shown that already a few labeled samples can sometimes greatly boost performance over an unlabeled dataset (Ruff et al., 2020; Qiu et al., 2022; Liznerski et al., 2022). OE has been very successful in the past, often outperforming methods operating in the traditional anomaly detection setting across many benchmarks, though at the cost of requiring labeled samples, which are often not available or hard to obtain in more specialized domains.
Another setting that has recently gained popularity is titt----iitribation (OOD) detection. In OOD detection, we have additional information about our dataset the form of class labels. Anomaly detection is a special case of OOD detection with only a single label. While the problem is similar, most approaches that tackle OOD detection make specific use of a classifier trained on the dataset labels (Hendryeks & Gimpel, 2017; Lee et al., 2018; Wang et al., 2022), which camnot directly be applied in the anomaly detection setting, training a classifier on a single class is not straightforward.
In cantrast, our method operates in the traditional anomaly detection setting and leverages only the information we have about our normal training samples without making additional assumptions about the nature of anomalics. Further, while we assume access to a datasct containing mostly normal samples, our method does not rely on additional labels, as they can be difficult and expensive to obtain, particularly in more specialized settings.
3

---

Published in Transactions on Machine Learning Research (09/2025)
Input
Representations
Context Contrasting
Heads
Context:
Encoder
Content Alignment
Contexta
Heada
Figure 2: We provide an overview of the training with CON2. We observe each input sample, dashed/dotted lines marking different samples, in two distinct contexts, and pass through encoder to extract its represenlation. A projection head maps representations into a projection space where apply condent contrasting to learn context-specific clusters in the representation space (= and clusters). Another projection head projects representations to a different space to conduct content alignment, encouraging structural aligument between the context clusters. We mark positive and negative pairs with + and respectively.
3 Methods
In the following, we recap some background on contrastive learning. We then present our novel representation learning objective CoN2, which allows us to learn tightly clustered, informative representations for anomaly detection when observing samples two different contexts. Consequently, we introduce the concept of context. augmentations, which allow us create new contexts for arbitrary datasets by leveraging symmetrics within normal samples. Finally, showcase how to use representations learned with CON₂ to detect anomalies al. test time.
3.1 Contrastive Learning
In this section, introduce terminology of contrastive learning (van den Oord et al., 2019), which we later use in CON2 objective. Contrastive learning relies the definition of positive and negative pairs of samples and learns to maximize the similarity between positive representation pairs while pushing apart representations negative pairs. Popular contrastive approaches, such as SimCLR (Chen et al., 2020), achiove this by incorporating an instance discrimination objective in their loss function. Here, we define the instance discrimination loss as
exp (sim xx, xx)/T) exp (sim(xx (x, x/)/T)
(x, x)
-log ae'
(1)
x"eX,x"
where we consider sim(ær, ar) to be the cosine similarity between two samples x, e X of a dataset X. We
provide additional background on contrastive learning Appendix A.1.
3.2 Context Contrasting with Content Alignment
The instance discrimination loss from Section 3.1 requires negative samples to prevent degenerate solutions. However, we typically do not have access to anomalous negative samples in the anomaly detection setting. Previous work instead relied simulating synthetic anomalies to circumvent this problem (see Section 2).
4

---

Published in Transactions on Machine Learning Research (09/2025)
However, designing synthetic anomalies is often not feasible, especially in more specialized domains. Our work addresses this limitation with the new CON₂ objective, which leverages additional views contexts of normality instead of creating negative pairs from anomalies. CON2 learns concentrated representations and circumvents collapse by simultancously learning two separate but connected normal representations cach sample. We achieve this by combining two contrastive building blocks: Contert Contrusting, which learns distinct context-dependent clusters of normality, and Content Alignment, which ensures that representations capture the semantic information of normality by aligning the positions of context-augmented samples across clusters. This approach allows to group normal representations in an informative manner and tailor them for anomaly detection in a contrastive way. In Figure 1, we compare the desired structure in the traditional one-class classification setting with the representation space we are learning with CoN2.
First, let's assume that we can observe a dataset Xtrain from a second perspective, or context, retaining the information content of all samples. Let us denote this dataset with X train The idea behind CON₂ to let our model learn distinct, concentrated representation clusters of both Xtran and X Since we know which sample in Xtrain corresponds to which sample in X train CON₂ additionally encourages the model to learn symmetrical structure across these context clusters, effectively aligning the representations between them. In Section 3.3, we will see how to create X from Xerain and assume they are available for the remainder of this section.
Given Xusin and XC trains we define a new dataset and label each sample according to its context as follows:
x == {(0, 0) x Xeram u {(x,, 1) a Xtrain)
Then, let T be set of augmentations that model sample invariances similar to Chen et al. (2020). We apply two different augmentations fxx l/2 ~ T for each (æ, in X and define
x U (z), (t ()).
(3)
(ax))ee
Further, for case of notation, we denote
((f()))))))) (x, e xx
(4)
for any function f. Next, we introduce context contrasting and content alignment, the main building blocks of our CON2 objective.
Context Contrasting As described carlier, we want a model to learn two distinct normal clusters, one for Xtrain and one for XC X We achieve this in a contrastive manner with the context contrusting loss Ccontext()). For given sample æ, we derive its representation go(x) using an encoder go- We can then define the context contrasting loss as
LContexe LContexc(X
Σ (x)) o' xaz stz'/y=y'
, Za),
(5)
K
where K := 4N(2N-N11 is the normalization constant, Z h ho (x²)), he is a projection head that gets discarded after training similar to Chen et al. (2020), and l as in Section 3.1.
Intuitively, context contrasting builds positive and negative sample pairs by matching context labels (see Figure 2). Thus, LContext encourages dense context clusters by maximizing the similarity of positive pairs while ensuring distincliveness between clusters by maximizing dissimilarity between negative pairs of sample representations.
Content Alignment While CContext (.) allows us to learn context-dependees representation clusters, we also want to leverage our knowledge about sample correspondences between Xtrain and X to align the structure between the context clusters. Similar to LContext (:), we can accomplish this in a contrastive manner by building positive pairs across clusters, associating all instances that correspond to the same sample,
5

---

Published in Transactions on Machine Learning Research (09/2025)
independent of their context, while negatively associating all pairs of samples that correspond to different sample. In specific, given sample a Xtrain and its corresponding of e x Grain let
(x) = (Sv (1))) (1 (z)) X ALET) u (Jw (1 ( 1 ALET).
(6)
where (x) = ho (x) and be denotes projection head that is independent of hgg Intuitively, A(x)
contains the 4 projections that are associated with the content augmented samples of and x. We then define the content alignment loss as
LContent (x9)= = 12N
Σ DE
8(z,2'7
(7)
2,2'
242
where Zo fo (XC) contains the projections of samples from the augmented dataset, and & again in Section 3.1.
Content alignment ensures that all representations of the same normal sample get matched across different contexts, encouraging alignment of the representations between context clusters.
Our final objective CON₂ is then a linear combination Coontext (-) and CContent This way, CON₂ allows a model learn context-specific, content-aligned representations of normality:
Lcong (XC)= LContoxt ( (8) ar Co (o (XC)
Note that LContext and LContent contain different numbers of positive and negative pairs. Hence, they have different scales and we thus introduce a weighting factor e IR+ (see Appendix C for more details). Figure provides a visual overview of how CON₂ learns representations using context contrasting and content alignment. We provide empirical evidence for the existence of context clusters after training in Appendix E.4.
3.3 Context Augmentation
In Section 3.2, we saw how CON₂ applies context contrasting to distinguish between the original training dataset Xtrain and the dataset in distincl context X At the time, content alignment leverages the fact that we match, align, each original sample with its counterpart in the new context. To create X we observe that, for most datasets, we can find sample symmetries that allow us to create distinct new context of its samples without altering their information content. Let Xtrain C X' and X = 4c(X (Xttain)) ) where X' denotes the dataspace and te : X + is a data transformation. We call te a context augmentation for Xtrain if it fulfills two heuristic requirements.
Assumption 1 (Distinativeness). Let x ~ PX.rin and α ~xx be any two samples from the original and the transformed data distribution respectively. The transformation le satisfies the distincliveness assumption for Xtrain if, for any two samples æ and x. it holds that:
Pxc (x)) s 0 and PX (a ) ss
Assumption 2 (Alignment). Let x, x' E Xurain and let d(x, x') denote some similarity measure for samples in the imput space. Transformation te satisfics the alignment assumption, for any two samples x, x', it holds that:
d(x,x') d(te(z),
(10)
Intuitively, Assumption 1 ensures a clear distinction between the distributions PX and pxc which is necessary to learn soparated clusters with context contrasting. Similarly, Assumption requires originally similar normal samples stay similar in the new context to prevent potential misalignments when applying content alignment. The idea behind distinctiveness and alignment is help find reasonable transformation te that is symmetric with respect to the normal data distribution PX in the sense that
PXtramn (x) =xx (x)), and PX # Pxc for all x ~ PXeram"
(11)

---

Published in Transactions on Machine Learning Research (09/2025)
Hence, we call be contert augmentation if it satisfies both Assumptions 1 and 2 for a given dataset Xtrain- In the following, we introduce some examples of context augmentations that we will use in our experiments in Section 4.
Invert This transformation exchanges every pixel value a with the value 1- z. Consider a dataset of lung X-rays. Normal tissue and bones cannot lead the inverse of any normal X-ray image, so distinctiveness is satisfied. Additionally, inversion does not remove semantic information, ensuring alignment.
This transformation corresponds to mirroring sample vertically. Consider brain MRI dataset. MRIs are typically recorded in standardized world coordinates, such that all samples have the same orientation. Flip changes this orientation, satisfying distinctiveness, while keeping the content of the image unchanged (alignment).
Flip
Equalize The histogram equalization transformation ensures that the histogram of pixel intensities of an image uniform. Consider dataset containing pictures of Mclanoma that can be benign or malignant. On such images, histogram equalization changes the color distribution considerably (see Figure 5), ensuring distinctiveness. However, one can still clearly see the same skin features, albeit colored differently, keeping the semantics of the images intact and ensuring alignment.
Among these three, invert satisfies our assumptions for practically all datasets considered in our experiments. We will see representations learned by CON₂ using the invert context augmentation in Section 4.1 and discuss the alternatives in Section 4.3.
3.4 Anomaly Detection
Similar to other works in the field (Ruff et al., 2021), we define an anomaly score function that maps a given sample's 's reprosentation onto a scalar determine its anomalousnoss and detoct anomalics at test time. We can then define threshold on this anomaly score, predicting anomaly for samples above the threshold and normal for samples below. See Appendix A.2 for additional background and related work the anomaly detection setting.
To detect anomalies using the representations of CON2, we present two anomaly score functions that measure how well a test sample adheres to the context representation clusters. One of the most popular and straightforward approaches to achieve this a non-parametric nearest neighbor distance approach (Bergman et al., 2020; Sun et al., 2022). Our first score adopts similar procedure using the cosine similarity, though explicitly leveraging the augmentations used when training CON2- Specifically, let us define the cosine distance between the training set Xtrain and given test sample æ with transformation t as
(o x'e
sNND(2;t):= := max
(12)
Intuitively, the better a new sample aligns with the context cluster given by augmentation t, the more likely it is to be normal. Conversely, a lower cosine similarity indicates that a sample is misaligned with its context. cluster, effectively allowing us to flag it as anomalous. While this approach works well in practice, it is rather memory-ineffiiien as we need to store the representations of all samples in Xtrain-
To address this limitation, we introduce a likelihood-basee based score function SLh to adapt our approach to eesource-constrainee settings. For simplicity, assume that representations within each context cluster are distributed according to multivariate Gaussian distribution. This assumption allows us to efficiently estimate the empirical mean and covariance from the training set and evaluate the probability density to derive an anomaly score without requiring a lot of compute or memory. Note that contrastive approaches typically tend to learn representations with relatively large norms, which may lead to numerical instabilities when estimating the covariance matrix. Our SLH thus estimates the empirical mean and covariance on the normalized representations. In particular, let
( [x0((())) [T=))
x X Xtrain
(13)
train
7

---

Published in Transactions on Machine Learning Research (09/2025)
be the normalized representations of the training set augmented with some augmentation t. We then compute the density of a multivariate normal distribution based on the empirical mean and covariance,
F4, = 7o (Z and Σ == E (2(1)
(14)
We then define
1 =o (N(1 90 s0(((a))) Fe. Fr.EE E
sLH(#;t) =
(1)()())))
(15)
We further leverage that our model can differentiate between the two contexts and learns invariances across different augmentations from T by applying test-time augmentations, similar to previous works (Tack et al., 2020; Wang et al., 2023), which further improves our anomaly detection performance. More specifically, let Tisst (ti....,t.l ta} = T be a set of A test time augmentations. For a given sample æ and its corresponding context. augmented sample x., we define our final anomaly score functions S(NND.LH) X R as
S(NDDLH) (x) 1
1/122 A/2
Σ
LNN
Σ (EF : A/2
(16)
We will see in our experiments how both scores reliably lead to competitive anomaly detection performance, though exhibiting a slight performanc--fficiencyy trade-off.
4 Experiments
In the following, compare anomaly detection representations learned by CON2 to various anomaly detection approaches based on pretrained foundation models and popular self-supervised methods across various medical imaging datasets, a specialized domain where prior knowledge about anomalies is typically hard to obtain. We further analyze the performance trade-off of SNND and SI. and explore different context augmentations, discussing their effect on anomaly detection performance. Finally, we examine the impact of context contrasting and content alignment on the performance of CON2. We refer to Appendices to D for more details regarding compute, code, the choice of perrparameesrs, and our datasels.
Baselines We compare our work to various recent contrastive anomaly detection baselines, including SSD (Sehwag et al., 2021), CSI (Tack et al., 2020), and UniCon-HA (Wang et al., 2023). To ensure comparability between CON₂ and other self-supervised methods, we conduct all experiments with randomly initialized ResNet18 architecture (He et al., 2016). Additionally, we compare our method against CLIP-AD (Liznerski et al., 2022), AnomalyCLIP (Zhou et al., 2024), anomaly detection with SNND on I-JEPA (Assran et al., 2023) representations, MVFA (Huang et al., 2024), MediCLIP(Zhang et al., 2024) and PANDA (Reiss et al., 2021), which build on large, pretrained models such as CLIP (Radford et al., 2021) or ResNet (He al., 2016). Both MediCLIP and MVFA-AD are methods specifically proposed for anomaly detection on medical datasets.
4.1 Anomaly Detection with CON₂ Representations
We demonstrate the capabilities of CON₂ across six different medical datasets. We train CON₂ on the healthy population of the training datasets of BreastMNIST (Al-Dhabyani et al., 2020) and OctMNIST (Kermany et al., 2018b) of the MedMNIST collection (Yang et al., 2021; 2023), containing breast ultrasound and retinal optical coherence tomography images, respectively, the KVASIR dataset (Pogorelov et al., 2017) which contains endoscopic images of the gastrointestinal tract, the BR35H brain MRI datasct (Hamada, 2020), a chest. x-ray datasel for Pneumonia detection (Kermany et al., 2018a) and a Melanoma detection dataset (Javid, 2022). We use the invert transformation to put each training dataset into a new context. As discussed in Scction 3.3, this transformation satisfics both Assumptions 1 and on most imaging datascts and is thus usually a valid context augmentation. We first perform anomaly detection with CoN, using the SNND anomaly score function and later provide comparison to SLT. We run all experiments across three seeds,
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: We apply the SNND anomaly score to representations of CON₂ using invert as the context aug- mentation and compare various baselines that cither use large pretrained networks (Pretrain) or learn normal representations through sule-supeniision (SSL). We train all methods on normal training samples of six real-world medical imaging datasets and evaluate them held-out test set with normal and anomalous samples. Except for the zero-shot baselines, we each experiment with three different seeds and report the mean # standard deviation of the area under the receiver operating characteristic curve (AUROC).
Method
BreastMNIST
OctMNIST
Kvasir
BR35H
Pneumonia
Melanoma
CLIP-AD
55.1
41.3
57.0
66.1
71.2
77.2
Pretrain
AnomalyCLi CLIP
63.0
68.9
68.1
96.5
70.3
62.1
(Zero-shot)
I-JEPA-SAD
70.8
82.3
90.3
99.9
82.1
93.5
MVFA
55.7
91.3
74.4
78.1
45.9
72.8
Pretrain
MediCLIP
69.
2.5
69.4+1.5
6.0
53.7
73.5 3.1
(Fine-tuned)
PANDA
63.
9..
91.0=0.0
99.8 ; 0.0
a.0
93.5 0.0
I-JEPA-AD
70.4
6.7
81.6+1.7
99.8
76.4+17
9.3
SimCLR
74.7
0.2
85.2
99.8
91.0=0.9
7
SSL
SSD
44.6
8.4
81.8+0.7
99.8
90.9
79.0 2,2
CSI
77.3
0.1
88,1
95.1=0.6
7..9
92.3 0.2
UniCon-HL. HA
76.2
6..
64.6
98.6
86.4+0.1
91.1+0.8
Ours
CoN₂ (SNND)
81.7
91.4+002
100 0.0
91.1.007
94.1 0.4
SAND
Sin
CoN;
Learen
Leatos
96
80-
AUROC (%)
AUROC (%)
90
60
85-
40
80
70
BreasMNIST
OOMNIST
Ksavir
BR3SH
Preumonia
Melanoma
BreHMNIST
OcINNIST
Kvairr
BR35H
Posumonia
Melanom
Figure 3: Comparison between anomaly detection with SNND and SLH- The figure shows the AU- ROC in percentage on all datasets after training CoNg, and evaluating the anomaly scores on the resulting representations.
Figure 4: Evaluating the impact of the individual loss terms CContent and LContext (.) on CON2- The bars in the figure show the AUROC in per- centage all datasets after training CON₂ and applying SNND the resulting representations.
training on a healthy train split and applying our anomaly score functions to the representations of samples of a held-out test set to detect anomalics. We report the mean and standard deviation of the resulting area under the receiver operating characteristic curves (AUROC) Table 1. We report only mean values without. standard deviations for our zcro-shot basclines, as these methods do not involve randomness.
Anomaly detection using representations learned with CON, consistently outperforms baselines across all datasets. When comparing method to other self-supervised learning baselines, we see clear performance gap. demonstrating the advantage of leveraging symmetrics that are present in the normal training dataset as opposed to learning representations in a traditional contrastive way, such as SimCLR and SSD, or making assumptions about the expected anomalies, like CSI or UniCon-HA. Further, while CLIP-based zero-shot methods like CLIP-AD or AnomalyCI.IP have previously demonstrated impressive performance across many natural imaging datasets, we can see that these methods are not yet able reach the performance levels of specialized self-supervised approaches. Surprisingly, found that MVFA and MediCLIP, which
9

---

Published in Transactions on Machine Learning Research (09/2025)
Original
Dataset
Flip
Equalize
Invert
Invert
BreSSt
8..
1.8
81.7+0.9
OctMNIST
0.5
0.3
0.8
Kvasir
2.1
932
+.2
Equalize
BR35H
99.8=0.3
99.9
100 =0.0
Pncumonia 92.8=
9.3
91.1
Flip
Melanoma 93.4
94.6+
94.1
Table 2: Comparison of different context augmenta- lion candidates. We report the mean AUROC on all datasets after training CON, across three seeds. Augmentations that satisfy Assumptions and ex- hibit robust performance.
Figure Examples of different context augmenta- tion candidates on BreastMNIST, OctMNIST, Keasir, BR35H, Preumonia, and Melanoma, respectively. In- vert replaces each pixel value with 1- x, Equalize stands for histogram equalization, and Flip denotes vertical flipping.
specifically tailored for anomaly detection on medical datasets, did not consistently outperform broader CLIP- methods like CLIP-AD and AnomalyCLIP. Further, we found that PANDA, which individually fine-tunes a pretrained ResNet50 on each dataset using a domain adaptation technique, exhibits much better performance, sometimes reaching AUROCs close to what observe when training with CONg. Similarly, our I-JEPA-ZSAN baseline, which applies SNND on top of representations of pretrained I-JEPA model, performs surprisingly well. Interestingly, training I-JEPA our datasets directly (L-JEPA-AD) yields much worse results. Nonetheless, further exploration of anomaly detection using I-JEPA may provide an interesting direction for future work. We provide more details regarding our baselines in Appendix B and additional ablations and experiments in Appendix E.
While anomaly dctection with SNND on CON₂ representations exhibits impressive performance across all datasets, we need to store the whole datasel to compule the nearest. neighbor distance, which is often not feasible for larger datasets. We thus want to compare the performance of SNND the more efficient. alternative SLH from Section 3.4. When comparing the mean AUROCs (see Figure 3), we can see that SLH typically performs very similar to SNND- We thus suspect that CoN₂ typically learns elliptical context clusters, allowing a Gaussian likelihood function to effectively detect anomalies in low probability regions of the representation space. However, for some datasets like BreastMNIST, we observe a significant performance drop, which suggests that elliptical clusters are not always guaranteed. conclusion, SNND exhibits better results overall and should be preferred over SLH. However, resource constraints do not permit the usage of SNND; SLH provides efficient alternative with only minor performance degradation. We compare compute efficiency between SNND and Sth in Appendix E.3.
4.2
Impact of LContent and LContext
We evaluate the effect of the context contrasting and content alignment on CON₂ by applying LConteut (+) and Lcontexa(.) individually and using the SAND score for anomaly detection. As in Section 4.1, we apply the invert context augmentation to all samples in these experiments and present the ablation results in Figure 4.
We observe that LContent (.) often performs fairly well. We suspect the structure learned by the content alignment is quite similar to CON2, though less concentrated and with the context clusters overlapping, which may lead to lower performance. Conversely, LContext does not seem to perform well on its own on most datasets. Without content alignment, we suspect that context contrasting collapses the context clusters onto single points similar to the hypersphere collapse in (Ruff et al., 2018). Finally, this experiment demonstrates that combining both terms in CON₂ improves overall anomaly detection performance.
10

---

Published in Transactions on Machine Learning Research (09/2025)
CoN₂ (Invert)
CoNy (Flip)
CSI
UniCon-HA
SimCLR
SSD
AUROC
0.8
0.7
0.6
Airplane
Bird
Cat
Deer
Dog
Horse
Ship
Truck
Average
Normal Class
Figure 6: AUROCs of CIFAR10 when setting class as normal and detecting the rest as anomalous. We compare CON₂ with the invert and flip context augmentations with SNND to other contrastive anomaly detection methods. Both the invert and flip context augmentations fulfill our assumptions, resulting in good performances across all classes. Our method further outperforms our basclines in most classes. CON, with flip has the highest average across all methods considered.
4.3 Alternative Context Augmentations
In our previous experiments, we trained CON, representations using the invert context augmentation. However, this transformation may not always satisfy Assumptions 1 and 2 for all datasets. We thus want to explore alternative transformations that could serve as context augmentations in certain scenarios. In particular, we find that vertical flipping and histogram equalization can fulfill our distinctiveness and alignment assumptions on many of our datasets. Figure 5 provides examples of these transformations on each dataset. We compare the performance of all three candidate transformations in Table 2.
While the three transformations typically perform rather similarly across datasets, we observe clear drop in performance with the flip transformation on MNNSST OctMNIST, and Kvasir, and with the equalize transformation on MNIST and OctNNIST. For the flip transformation, we observe a clear violation of distinctiveness (Assumption 1), as these datasets seem to record samples from an arbitrary angle. On OctMNIST, the equalize transformation seems to introduce some noise artifacts, which can lead to a violation of alignment (Assumption 2). Additionally, some original samples of both BreastMNIST and OctMNIST may look very similar histogram equalized samples, violating distinctiveness (Assumption 1). Further, note that the flip transformation on Melanoma violates distinctiveness but still performs well, indicating that violation of Assumptions 1 and 2 does not necessarily imply bad performance. Finally, these experiments demonstrate how proper context augmentations achieve similar performance, validating the definition of our assumptions in Section 3.3.
4.4 Natural Imaging
In addition to the results on the more specialized medical imaging domain, our method also exhibits robust performance on more traditional natural imaging benchmark datasets. Here, we train CON₂ on the CIFAR10, CIFAR100 (Krizhevsky et al., 2009), ImageNet30 (Russakovsky et al., 2015; Hendrycks al., 2019b), Dogs vs. Cats (Cukierski, 2013), and Muffin vs. Chimuahua (Cortinhas, 2023) datasets in the one-class classification setting (Ruff et al., 2021). In the one-class classification setting, typically work multi-class classification datascts where we consider one of the classes as the normal class and the rest as anomalics. In particular, we train our model on the training samples of the normal class and want to differentiaae between unseen samples of this normal class and all other classes at test time. Similar to our previous experiments, we train each model across three soods for cach class of cach datasct. Note that we do not compare pretrained models here, because standard pretraining datasets typically include the samples of these datasets, leading leakage of the anomaly class during pretraining.
11

---

Published in Transactions on Machine Learning Research (09/2025)
Cos, (Invert)
CoN; (Flip)
Coly (Best)
UniCon-HA
SimCLR
0.8
AUROC
CIFAR100
ImageNet200
Cats
Muffin Chihuahua
One Class Dataset
Figure 7: One class classification results for CIFAR100, ImageNet30, Dogs vs. Cats, and Muffin vs. Chihuahua. Our method consistently outperforms our baselines on CIFAR100 and Dogs vs. Cats while exhibiting more robust. performance across different normal classes with similar average performance and Muffin vs. Chihuahua. Additionally, provide results including CON2 (Best), which demonstrates how carcfully selecting context augmentations satisfying the assumptions of Section 3.3 further improves anomaly detection capabilities of CoN2.
CSI on ImageNet30
Much like our experiments on medical imaging datasets, we can typically consider the invert transformation as valid context augmentation in these datasets. Additionally, vertical flipping often satisfies distinctiveness, as natural images are usually not taken from a birds-eye view and adhere to gravity, e.g., a plane of CIFAR10 will typically not fly upside down. Vertical flipping also satisfies alignment since it neither adds removes any information from the image, but instead reorders pixel positions. On the other hand, histogram oqualization often does not satisfy distinntivenesss as this transformation may result in scenes that seemn slightly differently illuminated. We thus only present results of CON2 with flip and invert context augmentations, We provide examples of cach context augmentation in Figure 8 in Appendix E.1.
In Figure 6, we compare the performance of CON₂ and our baselines across the different classes of CIFAR10. For both the invert and the flip context augmentation, CON₂ outperforms our baselines on almost all classes. Here, the flip context augmentation achieves slightly better average AUROC of 95.3 than the invert transformation, which exhibits average AUROC of 94.6.
We further provide results on one-class CIFAR100, ImageNet30, Dogs vs. Cats, and Muffin vs. Chihuahua in Figure 7. In addition to the invert and flip context augmentation, we also provide results for CON₂ (Best), which selects the context augmentation individually for each class, depending on which satisfies alignment and distinctiveness better for the current normal class. We report the mean and standard deviation of the AUROCs aggregated over seeds and classes of the respective datasets. Our method compares well against established baselines natural images, matching improving the state-of-the-art in self-supervised anomaly detection. Similar to what we
CIFAR10, CON₂ displays robust performance across the board. Our approach outperforms basclines on CIFARI00 and Dogs vs. Cats while matching the performance on ImageNet30 and Mulfin vs. Chihuahua, while exhibiting much more consistent performance across different. normal classes. We can also see that selecting the context augmentation that best fits our assumptions for each normal class improves the performance. In Appendix E.1, we provide numerical results of CON, on all natural imaging datasets and context augmentations, including the histogram equalization context. augmentation.
12

---

Published in Transactions on Machine Learning Research (09/2025)
5 Conclusion
In this work, we presented the method CONg, which learns representations suited for anomaly detection by leveraging symmetries in the normal training data. Learning representations without making particular assumptions about anomalous data is particularly useful in specialized domains such healthcare, where anomalous data can be rare and hard to simulate accurately.
We demonstrated the efficacy of our method on real-world medical imaging datasets, showcasing impressive results when compared to competitive baselines. Our experiments highlighted the applicability of CoN₂ in safety-critical applications where robust anomaly detection is essential. We further introducod a likelihood- based alternative to the widely used neighbt distance anomaly score function. This approach leverages that context clusters tend to be elliptical and usually achieves similar anomaly detection performance to nearest neighbor distance approach while requiring much less memory. We further demonstrated CoN2 's robustness to the choice of context augmentation, validating the distinctiveness and alignment assumptions of context augmentations. Finally, we showcase how the combination of context contrasting and content. alignment with CoN₂ lcads to the overall improvement of anomaly detection performance.
In conclusion, CON₂ represents significant advancement in anomaly detection by learning concentrated representations from the normal data without relying on anomalous data. Our approach offers a particularly valuable and effective solution in specialized, high-stakes application domains.
Limitations Our current work focuses exclusively on image-based anomaly detection, and we do not include experiments involving other modalities like time-series or multimodal data, where finding appropriate context. augmentations could prove more challenging. However, the definition of CON2 is broad, and could be interesting to explore whether the symmetries in time-series, graphs, or multinodal data could naturally serve context angmentations, though finding appropriate content augmentation may prove difficult for some modalities. Additionally, while we empirically show that CON₂ leads to highly informative representations of normality, we do not provide formal theoretical guarantees for our embeddings. Investigating how our method compares to other representation learning techniques outside of anomaly detection would be an interesting direction for future research. Finally, extending our approach to settings such as outlier exposure or dut-of-distribution detoction presents another promising direction. These scenarios would further test the robustness and flexibility of CONa in handling more complex anomaly detection tasks across wider range of domains.
Broader Impact While anomaly detection methods offer significant societal benefits, such as supporting doctors in standard screening procedures or identifying adverse samples safety-critical systems, careful consideration is needed when defining normal data. Biases or the underrepresentation of certain groups within these datasets could inadvertently lead to discrimination, especially in sensitive domains like healthcare. Ensuring that normal datasets are representative and unbiased is crucial to avoid unintended harm.
Acknowledgement
AR is supported by the StimuLoop grant #1-007811-002 and the Vontobel Foundation. TS is supported by the grant #2021-911 of the Strategic Focal Area "Personalized Health and Related Technologies (PHRT)" of the ETH Domain (Swiss Federal Institutes of Technology). AM was funded by ETH Zurich for part of the project.
References
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound
images. Data in brief, 28:1 18:664 2020. 8, 22
Jinwon An and Sungzoo Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special lecture IF, 2(1):1-18, 2015. 3, 20
13

---

Published in Transactions on Machine Learning Research (09/2025)
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezeano, Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang, Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorcch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programmming Languages and Operating Systems, Volume 2, volume 2 of ASPLOS '24, pp. 929 947, New York, NY, USA, April 2024. Association for Computing Machinery. ISBN 9798400703850. doi: 00.1455/622065336440
20," 21
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Sclf-supervised learning from images with joint-ombedding prodictive archilecture. arxiv preprint arXiv:2301.082. 2023. 8, 21
Deegan J Atha and Mohammad R Jahanshahi. Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection. Structural Health Monitoring, 17(5):1110-15 1128, September 2018.
ISSN 1475-9217. doi: 10.117//4452211177711 1
Liron Bergman and Yedid Hoshen. Classification-Ban Based Anomaly Detection for General Data. In International Conference on Learning Representations, 2019. 2
Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly delection. orXio preprinl WxXi:2202.104443 2020. 3,
C. M. Bishop. Novelty detection and neural network validation. IEE Proceedings Vision, Image and Signal Processing, 141(4)2117222, August 1994. ISSN 1359-7108. doi: 10.1049/ip-vis:11 vis:19941330. 20
Markus M. Breunig, Hats-Peter Kriegel, Raymond T. Ng, and Jörg Sander. LOF: identifying density-basod local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of dota, SIGMOD '00, pp. 93 93-104, New York, NY, USA, May 2000. Association for Computing Machinery. ISBN 978-1-58113-217-5. doi: 00.1145/32000.3335383 33353888 20
Jinghui Chen, Saket Sathe, Charu Aggarwal, and Deepak Turaga. Outlier Detection with Autoencoder Ensembles. In Proceedings of the 2017 SIAM International Conference on Data Mining (SDM), Proccedings, pp. 90-98.099000 Society for Industrial and Applied Mathematios, June 2017. 2
Ting Chen, Simon Kornblith, Mohammad Norowii, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. 37th International Conference on Machine Learning, 1CML 2020, 1ArTF1168147-31155--15888 February 2020. doi: 10.48550/ /axix 2002.00709. 4, 5, 19, 20, 21, 22
Matan Jacob Cohen and Shai Avidan. Transformaly-two (feature spaces) are better than one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion, pp. 4060-4069 2022. 3
Samuel Cortinhas. Muffin vs. Chihuahua, 2023. 11, 24
Will Cukierski. Dogs vs. Cats, 2013. 11, 24
Imant Daunhawer, Alice Bizeul, Fmanuele Palumbo, Alexander Marx, and Julia E. Vogt. Identifiability results for multimodal contrastive learning. In The Eleventh International Conference Learning Representations,
2023. 19
William Falcon and The PyTorch Lightning team. PyTorch Lightning, March 2019. 20, 21
Zahra Ghafoori and Christopher Leckie. Deep multi-sphere support vector data description. In Proceedings of the 2020 M International Conference on Data Mining, pp- 109-117. SIAM, 2020.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Izhak Golan and Ran El-Yaniv. Deep Anomaly Detection Using Geometric Transformations. In Advances in Newral Information. Processing Systems, volume 31. Curran Associates, Inc., 2018. 2
Koosha Golmohammadi and Osmar R. Zaiane. Time series contextual anomaly detection for detecting market
manipulation in stock market. In 2015 IEFE International Conference Data Science and Advanced
Analytics (DSAA), pp. 1-10, October 2015. doi: http:://www...gggg..o//////ttttt
A. Hamada. Br35h: Brain tumor detection 2020. 2020. URL http:////ww.llaggg///aaaaaatataa
detection. Online. 8, 23
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis F. Oliphant. Array programming with NumPy. Nature, 585(7825): 357 362, September 2020. doi: 10.1038/s41588 0221556-00002699-22 21
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 770-778, 2016. 8, 22
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazcika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In
International Conference on Learning Represenlalions, 2019a. 3
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019b. 2, 1,11,22 24
Chaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang, Xinchao Wang, and Yanfeng Wang. Adapting visual-language models for generalizable anomaly detection in modical images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11375-11385, 2024.
Muhammad Hasnain Javid. Melanoma skin cancer dataset of 10000 images, 2022. 8, 23
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winelip: Zero- Pero-/few-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 19606-19616, 2023. 3
Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C. S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Magdalene Y. L. Ting, Jie Zhu, Christina Li, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A. N. Huu, Cindy Wen, Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimmin Xia, and Kang Zhang. Identifying Medical Diagnoses and Treatable Diseases by Image-Basse Based Deep Learning. Cell, 172(5):1122-1133 1131.e9, February 2018a. ISSN 0092-8674, 1097-4172. doi: 10. (o.1016/j.coll.200 2018. 201...2..00.000 02.00.0. 8, 23
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKcown, Ge Yang, Xiaokang Wi, Fangbing Yan, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. cell, 172(5):1122-1133 1131, 2018b. 8, 22
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18677 2020. 19, 21
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. 11, 24
15

---

Published in Transactions on Machine Learning Research (09/2025)
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confiddenco-calibrated classifiers for detecting out-of-distribution samples. In International Conference on Learning Represenhations, 2018. 3
Jingyao Li, Pengguang Chen, Zexin He, Shaozuo Yu, Shu Liu, and Jiaya Jia. Rethinking out-of-distribution (ood) detection: Masked image modcling is all you need. In Proceedings of the IEEE/CV conference on compuler vision and pallern recognition, pp. 11578 11589, 2023. 3
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation Forest. 2008 Fighth TEEE International Conference on Data Mining, pp. 413-422, December 2008. doi: 10.1109/ICDM.2000
20
Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Klaus Robert Muller, and Marius Kloft. Exposing Outlier Exposure: What Can Be Learned From Few, One, and Zero Outlier Images. Transactions on Machine Learning Research, August 2022. ISSN 2835-8856. 2, 3,
Πlya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Represenlations, 2017. 22
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. 22
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 56 61, 2010. doi: 10. 10.25080/ Majora-92b/11922-000 21
Hossein Mirzaei, Mohammadreza Salehi, Sajjad Shahabi, Efstratios Gavves, Cees GM Snoek, Mohammad Sabokrou, and Mohammad Hossein Rohban. Fake It Until You Make It: Towards Accurate Near- Mia--iitribbtion Novelty Detection. In The Eleventh International Conference on Learning Representations, 2022. 3
Benjamin Nachman and David Shih. Anomaly detection with density estimation. Physical Review D, 101(7): 075042, April 2020. doi:
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do Deep Generative Models Know What They Don't Know? 7th International Conference on Learning Representations, ICLR 2019, October 2018. doi: (055500/rxxv11110000011 3, 20
Poojan Oza and Vishal M. Patel. One-class convolutional neural network. IEEE Signal Processing Letters, 26 (2):277-281, 2018.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and others. Scikit-learn: Machine learning in Python. Journal of machine learning research, (8(0cc):2825-295))) 2011. 21
Pramuditha Perera and Vishal M Patel. Learning deep features for one-class classification. IEEE Transactions Image Processing, 28(11):5450 5463, 2019. 2
Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen Mathias Lux, Peter Thelin Schmidt, et al. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, pp. 164-169, 2017. 8, 23
Chen Qiu, Aodong Li, Marius Kloft, Maja Rudolph, and Stephan Mandt. Latent outlier exposure for anomaly detection with contaminated data. In International Conference on Machine Learning, pp. 18157-18167. PMLR, 2022. 3
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Tal Reiss and Yedid Hoshen. Mcan-Shifted Contrastive Loss for Anomaly Detection. Proceedings of the AAAT Conference on Artificial Intelligence, 37(2):2155-2D1 2162, June 2023. ISSN 2374-3468. doi: 10.1609/ v3..........25309. 3
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2806-2814, 2814, 2021. 2,
Peter J Rousseeuw. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53 65, 1987. 27
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep onc-class classification. In International conference on machine learning, pp. 4393 4402. PMLR, 2018. 10
Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In International Conference on Learning Representations, 2020. 3
Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermoulen, Grégoire Montavon, Wojeicch Samek, Marius Kloft, Thomas G. Dietterich, and Rlbbrt Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEFF, 109(5):756-795, 2021. 2, 7, 11, 20, 22
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-22)) 252, December 2015.
doi: 00007//s1263-01--0811 ISSN 0920-5691, 1405.
11,
Alain Ryser, Laura Manduchi, Fabian Laumer, Holger Michel, Sven Wellmann, and Julia E. Vogt. Anomaly Detection in Echocardiograms with Dynamic Variational Trajectory Models. In Proceedings of the 7th Muchine Learning for Healthcare Conference, pp.
PMLR, December 2022.
Mohammad Sabokrou, Mahmood Fathy, Guoying Zhao, and Ehsan Adeli. Deep end-to-end one-class classifier. IEEE transactions on neural networks and learning systems, 32(():675566)) 684, 2020. 2
Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt-Frfurth, and Georg Langs. Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery. In Marc Niethammer, Martin Styner, Stephen Aylward, Hongtu Zhu, Ipek Oguz, Pew-Thian Yap, and Dinggang Shen (eds.), Information Processing in Medical Imaging, pp. 146 157, Cham, 2017. Springer International Publishing. ISBN 979-3-319-59050-.. doi:
12.
Thomas Schlegl, Philipp Secböck, Scbastian M. Waldstein, Georg Langs, and Ursula Schmidt-Erfurth. F-noGAA: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Analysis, 54:30 44, May 2019. ISSN 1361-8415. doi: http::/.mmmia..2110.......
3, 20
Bernhard Schölkopf, John C. Platt, John Shawe-Taylos Alex J. Smola, and Robert C. Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 447):1143-1471, 2001. 2, 20
Vikash Schwag, Mung Chiang, and Prateck Mittal. SSD: A unified framework for sclf-supervised outlier detection. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.ned 2021. 2, 8, 20, 21
Kihyuk Sohn. Improved Deep Metric Learning with Multi-class N-pair Loss Objective. In Advonces in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. 19
Yiyou Sun, Yifei Ming. Xiaojin Zhu, and Yixuan Li. Distrf--istiibution Detection with Deep Ncarest. Neighbors. In Proceedings of the 39th International Conference on Machine Learning, pp. 20827-20840. PMIRR June 2022. 2, 7, 21
17

---

Published in Transactions on Machine Learning Research (09/2025)
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances. In Aduances in Newral Information Processing Systems, volume 33, pp. 11839 11852. Curran Associates, Inc., 2020. 2, 8, 19, 20
David M.J. Tax and Robert P.W. Duin. Support Vector Data Description. Machine Learning, 54(1):45-60) 66, January 2004. ISSN 1573-0565. doi: 10.1023
2, 20
The pandas development team. pandas-der/pandaas Pandas, February 2020. 21
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. 4, 19
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Exgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Tarson, J Carey, Maan Polat, Yu Feng, Fric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henrikson, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 272, 2020. doi: 10-008//41199-01----90 2. 21
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Beruhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style. In Advances in Neural Information Processing Systems, volume 34, pp. 16451-16467. Curran
Associates, Inc., 2021. 19
Guodong Wang, Yunhong Wang, Jie Qin, Dongming Zhang, Xiuguo Bao, and Di Huang. Unilatcrally aggregated contrastive learning with hierarchical augmentation for anomaly detection. In IEEE/CVF International Conference on Computer Vision, 1CCV 2023, Paris, France, October 1-6, 2023, pp. 6865 6874. 2, 8, 19, 20, 21
IEEE, 2023. doi:
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Outtof-distribution with virtual-logit matching. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp.
IEEE, 2022. doi: D..1109/CVR526665 2022.00022 3
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- con-paracctric instance discrimination. In 2018 IEEE Conference on Compuler Vision and Pallern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18- 22, 2018, pp. 3733 3742. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR.2000 20VPP..011.00393, 19
Yang Xin, Lingshuang Kong, Zhi Liu, Yuling Chen, Yanmiao Li, Hongliang Zhu, Mingcheng Gao, Haixia Hou, and Chunhua Wang. Machine Learning and Deep Learning Methods for Cybersecurity. IEEE Access, 6:35365 35381, 2018. ISSN 2169-3536. doi: M.1109/ACCESS..0 201.. 2018.2836950.
Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. Tn IEFE 18th International Symposium on Biomedical Imaging (ISBI), pp. 191-195, 2021. 22
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Dala, 10(1):41, 2023. 8, 22
Suhang You, Kerem C. Tezcan, Xiaoran Chen, and Ender Konukoglu. Unsupervised Lesion Detection via Image Restoration with a Normative Prior. In Proceedings of The 2nd International Conference on Medical Imaging with Deep Learning, pp. 540 556. PMLR, May 2019.
Ximiao Zhang, Min Xu, Dehui Qiu, Ruixin Yan, Ning Lang, and Xiuzhuang Zhou. MediCLIP: Adapting CLIP for Few-shot Modical Image Anomaly Detoction In proceedings of Medical Image Computing and Computer Assisted Intervention MICCAI 2024, volume LNCS 15011. Springer Nature Switzerland, October 2024. 8
18

---

Published in Transactions on Machine Learning Research (09/2025)
Rui Zhao, Ruqiang Yan, Zhenghua Chen, Kezhi Mao, Peng Wang, and Robert X. Gao. Deep learning and its applications to machine health monitoring. Mechamicul Systems and Signal Processing, 115:213-28 -37, January 2019. ISSN 0888-3270. doi:
1
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. AnomalyCLIP: Object-agnostic prompt learning for zero-shot anomaly detection. In The Twelfth International Conference on Learning Representations, 2024. 2, 3, 8
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dae-ki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. 2
A Background
This section provides some terminology for contrastive learning and background about the anomaly detection setting.
A.1 Contrastive Learning
Recently, contrastive learning has emerged as popular approach for representation learning (van den Oord et al., 2019; Chen et al., 2020). By desigu, contrastive learning can learn representations that are agnostic to certain invariances (von Kügelgen et al., 2021; Daunhawer et al., 2023), which makes contrastive learning a particularly interesting choice to learn informative representations of normal samples (Tack et al., 2020; Wang et al., 2023), as it allows us to incorporate prior knowledge about our data into the representing learning process in the form of data augmentations. More specifically, invariances are learned by forming positive and negative pairs over the training dataset by applying data augmentations that should retain the relevant content of sample.
The goal of contrastive learning is to learn an encoding function go(x). where representations of positive pairs of samples are close and negative pairs are far from each other. For a given pair of samples æ, e X, we can define the instance discrimination loss as (Sohn, 2016; Wn et al., 2018; van den Oord et al., 2019)
exp(sim((x, x')/T) Σ
(x X) loog
(17)
(x, x" (1((sim(x.."))
As mentioned in Section 3.2, we consider the function sim x²) to correspond to the cosine similarity between the two input vectors, as this is one of the most popular choices in the contrastive learning literature.
One of the most prominent contrastive methods is SimCLR (Chen et al., 2020), which creates positive pairs through sample augmentations. There exists supervised extension called SupCon (Khosla et al., 2020), which incorporates class labels into the SimCLR loss. For a given set of augmentations T, a dataset X = ( (();,,))) and an augmented dataset X where |X] 2N and (02. Bi), (#22+1, 9:) ε X denote two transformations of the same sample using random augmentations from T, SimCLR and SupCon introduce the following loss functions:
1 N
LsimCLn(
Σ (( + 2N ml1
Σ (eo
(18)
2N
Σ 1
LoSupCon (x)
Σ
( Ez. 7)
(19)
N(y:)
(E
(E,,)))&
Here, we denote
2 = ([f(((()) x).
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table Average compute hours for the SSL experiments for each dataset and method per run. SimCLR and SSD use the same representations, so we can evaluate both methods in one go and list their compute hours together.
Dataset
BreastMNIST
OctMNIST
Kvasir
BR35H
Pneumonia
Melanoma
Method
SimCLR/SSD
0.6
25
4
2
3
5
CSI
2
112
6
4
8
6
UniCon-HL. HA
4
120
8
8
12
18
CON₂
1
36
6
3
5
6
where [e(*) = hor (x)), (a)) is a feature extractor, and hg (z) is a projection head that is typically only used during training (Chen al., 2020). Further, we define fe(X) ) (fe(a) (œ. e X3 and
(()) =(((),))))) (æi, X n =
is the number of samples in X with label y.
A.2 Anomaly Detection
the anomaly detection setting, we are given an unlabeled dataset fæI...., , = X c X, while assuming that most samples are normal, i.e., the dataset practically free of outliers (Ruff al., 2021). The goal is learn model from the given dataset that discriminates between normal and anomalous data at test time.
In this work, we assume the challenging case where our dataset is completely free of anomalies. Hence, we aim to discriminate between the normal class and a completely unobserved set of anomalies at test time. This setting is sometimes called one-class classification or novelty detection.
To achieve this goal, one straightforward approach to approximate the distribution px (x: directly using generative models (An Cho, 2015; Schlegl al., 2019). Because we assume normal data lie in high-density regions of px, we can discriminate between normal and anomalous samples by applying a threshold function Px T, where 7 IR is often task-specific threshold (Bishop, 1994). As density-based approaches are often difficult to apply to high-dimensional data directly (Nalisnick et al., 2018), we follow slightly different. line of work.
In this paper, we focus on lcarning a function X that provides us with representations that capture the normal attributes of samples in the dataset (Sehwag el. al., 2021; Tack al., 2020; Wang al., 2023), by mapping normal samples close each other in representation space. On the other hand, anomalies that lack the learned normal structure should be mapped to a different part of the representation space.
Given go (w), popular approach to detect anomalies is by defining scoring function S Z IR (Breunig et al., 2000; Schölkopf et al., 2001; Tax & Duin, 2004; Liu et al., 2008). The score function maps representation onto a metric that estimates the anomalousness of a sample. To identify anomalies at test time, we can use similarly to the density px, i.e., we consider a new sample æ to normal (z)) S T, whereas ((go(=)) t means x an anomaly.
B Compute & Code
We run all our experiments on single GPUs on compute cluster using either an RTX2080Ti, RTX3090, RTX0 GPU for training. Each experiment can be run with 4 CPU workers and 16 GB of memory. We provide an overview of the compute for our SSL experiments in Table 3. We omit the runtime for experiments with Pretrain methods, as these usually never run for more than an hour. Our experiments are written using PyTorch (Ansel et al., 2024) with Lightning (Falcon The PyTorch Lightning team, 2019).
In the following, list for each method and baseline how we arrive results and which code we
20

---

Published in Transactions on Machine Learning Research (09/2025)
CONz: We implement CON₂ using PyTorch (Ansel et al., 2024) together with Lightning (Falcon & The PyTorch Lightning team, 2019). To evaluate our method, we use various open-source Python libraries such as NumPy (Harris et al., 2020), scikit-learn (Pedregosa et al., 2011), Pandas (McKinney, 2010; team, 2020), or SciPy (Virtanen et al., 2020). We base the implementation of the instance discrimination loss & on the implementation provided Khosla et al. (2020)
SimCLR: For this baseline, we implement SimCLR (Chen al., 2020) and compute anomaly scores in a similar fashion as (Sun al., 2022). For this baseline, we rely on similar packages CoN2-
SSD: We use the same representations for SimCLR but evaluate by following the procedure outlined in Sehwag et al. (2021).
CSI: To run experiments for CSI, we used the code provided in https://ggthub..om/llmmaaa imple- menting new dataloaders for the missing datasets.
UniCon-HA: We conducted experiments by running code provided by Wang al. (2023) implementing new dataloaders for the missing datasets. We thank the authors for sharing their code with us.
CLIP-AD: We ran CLIP-AD analoguous to the CLIP-AD experiments described by Liznerski et al. (2022),
using the following prompts to describe normal images: BriAsMMNIST:
an image of healthy breast tissue
OctMNIST:
image of healthy retinal tissue
BR35H:
mri of healthy brain
Kvasir:
an image of a healthy cecum, pylorus, or z-line
Pneumonia:
an xray image of normal lung
Melanoma:
photo of benign melanoma
AnoMalyCLIP: We ran AnomalyCLIP with the code provided in wttpg//ttt....cgcogggo///e/
implementing new dataloaders for the missing datasets.
IJEPA-ZSAD:
This baseline is using I-JEPA (Assran et al., 2023) for zero-shot anomaly detection. We took wwtpo////tttasoooooooo/ooo/oo//e arransffrmers/ee//enee /ttps////emmmdeeddd/////aaaaa and
the pretrained model provided in performed anomaly detection with SNND on the average-pooled embeddings, using the normal training set build the nearest neighbor index.
I-JEPA-AD: This baseline trains I-JEPA (Assran et al., 2023) on the normal training data of our datasets to later perform anomaly detection. We trained the model using the original I-JEPA codebase in https: //github.comm ccca///aaacboooeeearrc/iiiiiiai embeddings after training. We train I-JEPA using the ViT-Tiny configuration to keep parameter counts comparable to our ResNet18 backbone. To make sure the performance gap between I-JEPA-AD and IJEPA-ZSAD is not due the smaller architecture, we also provide results for ViT-Base in Appendix E.5.
and performed anomaly detection with SNND on the average-pooled
MVFA: We ran MVFA with the code provided in http:///ggtttacccooo//aaaaaaaaaa imple- menting new dataloaders for the missing datasots.
MediCLIP: We ran MediCI.I with the code provided in http:://ggthuub.commmmmaaabbblb imple-
menting new dataloaders for the missing datasets.
PANDA:
We ran PANDA with the code provided in htttps//ggihuub.cm//aaaaesssss/s imwllmmnntiiil dataloaders for the missing datasets.
21

---

Published in Transactions on Machine Learning Research (09/2025)
C Experimental Details
Setting We evaluate our method in the so-called one-class classification setting (Ruff et. al., 2021). More specifically, we assume to have access to only the normal (healthy) class during training. At test time, the goal is to detect whether a new sample from held-out testsct stems from the normal class seen during training or whether it seems anomalous, i.., deviates from the training distribution.
Metrics Typically, there is a high-class imbalance between normal and anomalous samples in the one-class classification setting. Further, sctting an appropriate threshold for the anomaly score is often task-dependent. Therefore, popular approach to evaluating the performance of anomaly detection methods is to use the area under the receiver operator characteristic curve (AUROC) (Ruff et al., 2021). This metric is threshold agnostic and robust to class imbalance.
Hyperparameters We conduct all experiments using a ResNet18 (He et al., 2016) without the last linear layer as the encoder ge. Additionally, we set the two projection heads h& and h to a standard MLP with one hidden layer, analogous to SimCLR (Chen et al., 2020).
Similar to our method, all baselines make use of test-time augmentations. By default, CST and UniCon-HA use 40 test time augmentations, which we adopt for all baselines. In our experiments, we set the augmentation class to the augmentations introduced by Chen al. (2020). For the context augmentation, we experiment with vertical flips (Flip), inverting the pixels of an image (Iuvert), i.e., Luvert (Ziz) Tij. and histogram equalization (Equalize), see Figure for illustration.
We choose hyperparameters for CON2 based on their performance on the CIFARI0 dalaset and keep them constant across all experiments to ensure we have no exposure to the anomaly class of the medical datasets. We lincarly anneal the hyperparameter a Lcos, from 0 to over the course of training to encourage the model to first learn the context-specilic cluster structure while gradually aligning representations over the course of training. We optimize our loss using the AdamW optimizer (Loshchilov & Hutter, 2019) with 31 0.9, Bg 0.999, weight decay À 0.001, and using a learning rate of 10-3 with cosine annealing (Loshchilov & Hutter, 2017) schedule. We run all experiments for 2048 epochs.
Datasets
In the following, we provide details about preprocessing, sources, and licenses of the datasets we use in our experiments.
BreastMNIST
The BreastMNIST dataset (Al-Dhabyani et al., 2020) is part of the MedMNIST (Yang et al., 2021; 2023) collection. It consists of 780 ultrasound images of breast tissues, which are labeled for breast cancer with Malignant and Benign/Normal labels. We first resize images to 256 and apply center-cropping to feed 224 x 224 images to our model. We ran all our experiments on BreastMNIST with a batch size of 64. The dataset is part of the medmnist package, which can be installed with pip and is published under the CC BY 4.0 license.
OctMNIST
The OctMNIST datasct (Kermany et al., 2018b) is part of the McdMNIST (Yang et al., 2021: 2023) collection and consists of 109'309 optical coherence lomography images, which are labeled for blinding diseases with either the Normal label or any of Choroidal Neovoscularization. Diabetic Maculor Edema, or Drusen as anomalics. We first resize images to 256 and apply conter-cropping to feod 224 x 224 images to our model. We ran all our experiments on OctMNIST with batch size of 128. The dataset part of the medmnist package, which be installed with pip and is published under the CC BY 4.0 license.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Original
Flip
Invert
Equalize
Figure Examples of different transformations that can serve as context augmentations on ImageNe
Kvasir
The Kvasir dataset (Pogorelov et al., 2017) consists of 4000 endoscopic images of the gastrointestinal tract, which are labeled for various abnormalities with the labels Normal Cecum, Normal Pylorus, and Normal z-line for normal images and any of Polyps, Dyed Lifted Polpps, Dyed Resection Margins, Esophagitis, or Ulcerative- Colitis for anomalies. We resized images 224 224 and ran all our experiments on Kvasir with a batch size of 128. The dataset can be downloaded from https: htpp:://www.tt www.kk/ggl..co///tatteeeeeeeeeee kvasir bvasir-dataset dataset and is published under the Open Database license.
BR35H
The BR35H dataset (Hamada, 2020) consists of 3865 brain MRI images and is labeled for brain tumors with binary labels. We first resized images to 256 and applied center-cropping to feed 224 x 224 images to our model. We ran all our experiments on BR35H with a batch size of 128. The dataset can be downloaded from https:/ http//ww.kttkk www.kaggle.com www..aaglle.coo/ttattttese////// http/d/ataaetts/mmmeaaaaaeeeeee wwwrrrrtnnnnneeeeeeeeeeeeeeeeeee detectionn and is published under the CC BY 4.0 license.
Pneumonia
The Pneumonia dataset was originally published by Kermany et al. (2018a) and consists of 5'863 lung X-rays, which are labeled with Preumonia and Normal labels. We first resize images to 256 and apply center-cropping to feed 224 x 224 images to our model. We ran all our experiments on the Pneumonia dataset with a batch size of 128. The dataset can be downloaded from https: wtw..aaglle...o/tttttts///////// immot/oooommtttmmmoooooooo///// chest-xray- seat--xray-nnemmoii and published under CC BY 4.0 license.
Melanoma
We the Melanoma dataset of Javid (2022), which consists of 10'600 images of Melanoma labeled with being benign malignant. We resize all images to 128 x 128 before passing them to the model with batch size 128. The dataset is publicly available at https: http://ww///////// www.kkawgggga.................ee com/datasets/ taatasstsshassaalee hht./aanaannnnnaaa aaed/ manomaa images and is published under the CCO: Public Domain license.
a
CIFARIO/CIFARIO
CIFAR10 and CIFAR100 are natural image datascts with 32 samples. Both datascts consist of 60'000 samples, totaling 10 and 100 classes for CIFAR10 and CIFAR100, respectively. As CIFAR100 comes with only 600 samples per class, the dataset authors additionally define set of 20 superclasses, aggregating 5
23

---

Published in Transactions on Machine Learning Research (09/2025)
Table One class classification results for CIEAR100, ImageNet30, Dogs vs. Cats, and Muffin vs. Chihuahua. With all three context augmentations and both scores.
Method
Score
CIFAR10
CIFAR100
Image.Net30
Dogs vs. Cats
Muffin vs. Chihuahua
SLH
91,1
86.1
85.2 12.6
1..0=111
83.
CoN₂ (Equalize)
SNND
51.5
8..5
86.0= 12.0
8..2
87.
SLH
93.7+413
89.5
90.9= 8.8
87.8=100
91.4+12
CoN₂ (Invert)
SNND 94.6+3.6
90.6
91.2
88.7+15
93.8+3.0
SLH 94.7
89.1
88.9 11.9
90.0
92.6
CoN2 (Flip)
SNND
95.3
8..7
89.8 11.1
90.3
94.0
labels each. In our one-class classification experiments on CIFAR100, use the superclasses to ensure manageable number of runs and sufficient training data. We ran all our experiments on CIFAR10 and CIFAR100 with batch size of 512. Both datasets were published by Krizhevsky et al. (2009) and can be downloaded from https: www.....oorrtto.eeb/rrriiaaiiaia datasets come without a license.
To the best of our knowledge, these
Imagenet30
The ImageNNet30 dataset is subset of the original ImageNet dataset (Russakovsky et al., 2015). It was created by Hendrycks et al. (2019b) for one-class classification. The dataset consists of 42'000 natural images, each labeled with one of 30 classes. We preprocess the dataset by resizing the shorter edge to 256 pixels, from which we randomly crop a 224 224 image patch every time we load an image for training. We ran all our experiments on ImageNet with batch size of 128. The dataset can be downloaded from https: https://githnn /ttes://gttttbttt. cwt.bocco/eeeeeeee hoom/hendrykks//ae ssee/sssoooo ood, which comes with the MIT License. Further, while we could not find a license for ImageNet, terms of use are provided on https bttps://imager httt:///eonneee..rg..///
Dogs vs. Cats
The Dogs vs. Cats was originally introduced in Kaggle challenge by Microsoft. Research (Cukierski, 2013) and consists of 25'000 images of cats and dogs. We preprocess the dataset by resizing the shorter edge to 128 pixels and then perform center cropping, feeding the resulting 128 128 image to our model. Dogs Cats with batch size of 256. The dataset can be downloaded We ran all our experiments hwwp://ooetiiio//oogssseeaa catssvaaaas/ To the best of our knowledge, there from https://www.kaggl.ccomm is no official license for the dataset, but the Kaggle page points to the Kaggle Competition rules https: //www.kaggle, waggle.comm.... competit pemppetttiiss/aaee wwwiiionn/nndaeeeeeeeeeeeeeeeeee /d/////o-----sesaeesoe sangg-vv-catssrlle in the license section.
Chihuahua vs. Muffin
The Chihuahua vs. Muffin dataset consists of 6'000 images scraped from Google Images. We prepro- cess the dataset similar to ImageNet30, resizing the shorter edge of the images to 128 pixels while feed- ing random 128 x 128 sized image crops to the model during training. We ran all our experiments on Chihuahua vs. Muffin with a batch size of 256. The dataset was published by Cortinhas (2023) and can be downloaded from https: //www//wwwwww/// wwwglk.ccg...............ooooo. com/datasets/ wtwee/ottttaeeiessss.....oo wwwufffi----cihhheeee huahhuuaa--- mmag-clasiifcationnaaa According the datasets Kaggle page, the dataset is licensed under CCO: Public Domain.
In addition to the preprocessing mentioned above, normalize each image with mean and standard deviation of 0.5 after applying the augmentations of CoNg.
24

---

Published in Transactions on Machine Learning Research (09/2025)
SanD
SLH
1.00
0.95
0.95
0.90
0.00
AUROC
AUROC
9..55
0.85
0.80
0.80
0.75
0.73
0.70
2
5
Number of Context Augmentations
Number of Context Augmentations
ImageNet33 'Toaster'
ImageNet33 'Ambulance'
CIFARIO 'Cat'
CIFARIO 'Car'
Figure Ablation illustrating the effect of adding more context augmentations. While the performance of well-performing normal classes, such ImageNet30 Ambulance CIFAR10 Car, stays consistent when adding more augmentations, we see decrease for normal classes such ImageNet30 Toaster CIFAR1O Cat that already perform poor, to begin with.
E Ablations
In this section, we present some additional results for the natural image benchmark datasets (Appendix E. provide an ablation that explores adding more than one context (Appendix E.2), quantify the efficiency our anomaly score functions (Appendix E.3), and quantify the presence of context clusters using the silhouette score (Appendix E.4).
E.1 Natural Image Benchmarks
We provide examples of context augmentations on ImageNet Figure 8. Table shows detailed results of CON₂ on all natural imaging benchmarks and anomaly score functions. As we can see, SNND consistently outperforms SLH. Further, we can see that invert and flip, which usually satisfy distinctiveness and aligument the natural imaging datasets, outperform the equalize context augmentation, which fails to satisfy our assumptions many samples, can be seen in Figure 8.
E.2 Multiple Context Augmentations
Our formulation Soction 3.3 can be extended beyond only one additional context by slightly adjusting CContext However, in addition to loss in efficiency due requiring more memory, did not find additional context augmentations to provide performance benefit, seen in Figure 9. There, different classes of CIFAR10 and ImageNet30. In particular, we trained the adapted CON, loss for 2, 3, 4, 5, 6, 7, and 8 context augmentations, which we derived by combining Flip, Invert, and Equalize from our previous experiments. Adding more augmentations does not seem to harm cases where we experience good performance in the first place. However, we observe a diminishing performance for slightly more challenging classes. Additionally, combining context augmentation may violate distinctiveness alignment in pairwise comparison between the different contexts, potentially leading to unintended structure in the representation space.
ablation with
different numbers of context augmentations
E.3 Anomaly Score Efficiency
Assume representations Z e Bxd of our normal training datasot, where n the number of samples and d the dimension of the representation space. SNND requires us store all samples perform nearest neighbor search, resulting in memory complexity of O(nd). On the other hand, Stn only needs to store
25

---

Published in Transactions on Machine Learning Research (09/2025)
Runtime vs Training Set Size
Runtime Number of Test Batches
TTst Batches-10 10, Test Batch Size- 32)
(Training Set Size=10000, Test Batch Size Size-128)
NND
NND
10a
LH
LH
10
Runtime (seconds)
Runtine (seconds)
10
10a
10-2
10-1
10-3
100
1000
5000
10000
25000
500000
100000
10
10a
10a
Number of Test Batches
Training Set Size
Figure 10: Left: SNND and SLH when scaling n between 1 and 100000, while evaluating batches 32
samples each. Right: Evaluating to 1000 batches of 128 samples each when keeping 10000.
Silhouette Scores by Dataset (Mean = Standard Deviation)
0.7
0.6
Silhouette Score
0.496=0.030
0.5
0.421
0.4
0.358=0.019
0.3
0.249=0.030
0.219=0.038
0.211
0.2
0.1
0.0
BreastMNIST
OctMNIST
Kvasir
BR35H
Pneumonia
Melanoma
Dataset
Figure 11: The silhouctte averaged different sceds for cach datasct. The score is for all datasets, indicating clear presence of context clusters after training with CoN2-
the parameters of a multivariate Gaussian in the representation space, which results in O(d)) due to the covariance matrix. Hence, SLH is more memory efficient at scale than SNND, as typically n >> d. As for runtime, naive implementation of SNND(*) would result in a runtime of O(nd), we would have to compare to each sample of the training set, while StH is again O(d) due to the matrix multiplication when computing the log probability of gaussian. However, clever implementations in today's 's compute framework can narrow this gap, and we want to provide additional empirical evidence that compares runtimes between SNND and St.-- In Figure 10, we provide two figures that compare the runtime between SNND and SLH when varying n and the number of evaluated batches, respectively. As indicated by the asymptotic runtimes provided before, we see that SNND faster for smaller n, due the d within SLи- However, when increasing the number samples, both when fitting and for evaluating the scores, SLH soon becomes much more efficient than SNND-
26

---

Published in Transactions on Machine Learning Research (09/2025)
Table 5: Comparison of I-JEPA-AD when training on VIT Tiny versus ViT Base. We find that scaling up the number of parameters does not consistently lead to improved performance, and I-JEPA-AD clearly falls short of the performance of I-JEPA-ZSAD.
Method
BNeAST
OctNNIST
Kvasir
BR35H
Pneumonia
Melanoma.
1-JEPA-ZSAD (ViT-H/14)
70.8
82.3
90.3
99.9
82.1
93.5
I-JEPA-AD (ViT-T)
70.4+ 0.2
53.3= 6.7
81.6
9.1
76.4+ 1.7
92.1
I-JEPA-D (ViT-B)
55.6 12.7
38.5= 15.1
8..7
2.8
71.3 12.5
9 0.2
E.4 Context Clusters
This section provides ablation that analyzes whether the learned representation space exhibits the context. clusters, claimed in Section 3.2. For each dataset from Section 4.1, compute the representation of each sample for both contexts. We then label each representation with its corresponding context. This labeling allows us to evaluate how well our representations are clustered by context by calculating the silhouette score (Roussceuw, 1987). This score function evaluates how well a given cluster assigmment relates to the geometry of the dataset by computing a normalized fraction between the intra- and inter-cluster distances. The silhouette score takes values in [-1,1], where a score 0 indicates wrong labels, indicates overlapping clusters, and indicates clustering structure with correctly associated labels. Results of this ablation are in Figure 11. We see that all values are consistently well above clearly indicating clustering structure within our representation space.
E.5 I-JEPA-AD Backbone
In Table 5, compare the results between I-JEPA-AD with VIT-Tiny (5.5 million parameters) and ViT-Base (85.7 million parameters). As can be seen from the large standard deviations, scaling the number of parameters generally leads to inconsistent results and mostly performs worse than training with smaller backbone. We suspect this may be due to the small number of training samples in our datasets (see Appendix D). However, the impressive performance of T-JEPA-ZSAA suggests that tailoring the I-JEPA training paradigm more specifically for anomaly detection could be an interesting direction for future research in self-supervised anomaly detection.
27


Reviews:
[{"review_id": "urGoQIg5Ns", "reviewer": "Reviewer_LXz2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper is that the experimental results look very good.\n\nWeaknesses.\n* From a distant point of view, this paper is in some sense like many others the authors have cited. The overall theme of these papers are as follows. There exists a thing called self-supervised learning, which promises to learn a good latent representation. We will use it to learn latent representation and detect anomalies in the latent, frequently assuming normal distribution. The self-supervised method usually relies on contrastive learning and differs in how they define augmentations and possibly incorporating other tweaks. This is fine, there are already few papers on this.\n* What I see as the weakness is that the absolute lack for formal rigor. While reading the paper, authors offer a rationale why their augmentation and tweak to loss function is better than that of the others, it is more of a speculation.  I mean, it is better to have a rationale rather than nothing (and authors of prior art do the same), but how this is formally related to the density level detection / modelling of probability distribution / or anomaly detection.  Take as a example [1], which explains why learning a classifier between data and their noised version models distribution. Or [2], which tries to explain a particular phenomenon of self-supervised learning. I would like to understand, why the proposed improvements formally leads to better model.\n* How the would the method work on other data modalities? Like tabular data or graphs? How can we define the transformations?\n* Why the experimental comparison on datasets used by prior art is in the appendix?   Judging by Figure 7, the proposed method is marginally better than the prior art. \n* Is the proposed method of self-supervision better than for example JEPA, [4], which follows the same scheme.\n* Why the multi-variate gaussian is a good model for the data in the latent space, when they are mapped on the sphere (by dividing the projected point by its norm)? \n* Adding pretrained latents, e.g. CLIP is nice, but when that model was pretrained on very different dataset, is it comparable to the method, which was pretraied on the dataset, where one is performing the anomaly detection. Looking at Figure 5, the images looks very different to the normal image used to train CLIP. This comparison is therefore a bit unfair.\n\n\n[1] Gutmann, Michael, and Aapo Hyvärinen. \"Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.\" Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010.\n[2] Tian, Yuandong, Xinlei Chen, and Surya Ganguli. \"Understanding self-supervised learning dynamics without contrastive pairs.\" International Conference on Machine Learning. PMLR, 2021.\n[3] Han, Songqiao, et al. \"Adbench: Anomaly detection benchmark.\" Advances in neural information processing systems 35 (2022): 32142-32159.\n[4] Assran, Mahmoud, et al. \"Self-supervised learning from images with a joint-embedding predictive architecture.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an anomaly detection based on first learning latent representation and then detecting anomalies in latent while assuming that the data in latent follows normal distribution.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "* I think it would be fair to show that on the usual benchmark the method does not work all that great.\n* I would like to see theoretical grounding of the paper.", "broader_impact_concerns": "I do not see any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vghpWLYxbt", "reviewer": "Reviewer_SuCd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is clearly written.\n- Approach is interesting and novel.\n- Experimental results are strong.\n\n**Weaknesses**\n1. I think Figure 2 can be improved by color-coding the 4 input images or add corresponding symbols in Section 3.2. I was initially confused on which pair corresponds to same sample under different context, as it is introduced 2 pages later in Section 3.3 that context augmentation is invert.\n2. It is not very clear why \"The structure of anomalous samples is typically different from normal samples, which lets us detect them as outliers in the representation...\", especially because the context augmentation is color invert, which transforms any image under a fixed rule. It is helpful to elaborate this point with support from empirical evidence.\n3. Is Figure 1 using the actual representation learned by Con2? If yes, it is useful to highlight this. If not, I suggest visualizing the representation and checking if it corresponds to Figure 1, especially because leveraging representation symmetries is a big part of the motivation.\n4. From the discussion in Section 4.3, it does seem that an appropriate context augmentation is difficult to identify. This can be a limitation, e.g., data from different domain may require non-trivial design of the chosen augmentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Con2, a self-supervised learning method for anomaly detection that leverages symmetries in normal data to create two distinct but aligned representation spaces. Con2 enforces context-contrasting with content alignment. Hence anomalies which violate the learned symmetrical structures can be easily detected as outliers. Extensive experiments on medical imaging datasets demonstrate that Con2 outperforms existing anomaly detection methods, including self-supervised baselines and pretrained model approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Visualize the learned representation\n- Discuss W2 and W4 in paper.", "broader_impact_concerns": "Sufficiently addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IWS5kjJrN3", "reviewer": "Reviewer_ECNy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- The proposed method is simple, yet effective in the evaluation presented. The approach has a wide reaching impact if proven to be generalisable. \n- The idea itself is rationale and reasonable in the contrastive framework. It plays off the concepts of invariance to transformations to enforce contextual contrasting, which is a nice use of the framework. I particularly like figure 2, it does a good job of explaining the context alignment and contrasting idea.\n- The authors introduce the background and technical concepts well, introducing the topic of contrastive objectives, one-class classification and prior works.\n- Ablations studies demonstrate the impact and effectiveness of each component, clearly demonstrating the need of the whole system as presented.\n\n**Weaknesses**:\n\n- The main weakness of the work is that the writing is arguably inconsistent, missing some details, and makes claims that are not substantiated. I do however, feel that this can be simply addressed in revisions.\n- The results are limited in scope, while I agree that the medical domain is a suitable use case, the work is presented as a generalised solution. Hence, it would be beneficial to see performance on more generalised datasets, this would mean at the very least moving the results presented in the supplementary into the main manuscript. However, some further analysis would be appreciated. \n- Transformations to adjust the context are narrow in scope and do not reflect well real-world (flip, invert, equalise). Additionally, the types of transformation should be introduced more formally within the text, justified, and then analysed to ensure they are doing as stated in the assumption section 3.3. This section introduces the concept, but does not provide any clear implementation details, or justification.\n- Following on from the prior, there is little experimentation or analysis on the transformations themselves. This is arguably the most vital factor in the contrastive SSL objective. The choice of what to be invariant too has significant impacts on performance and generalisation to other datasets and domains.\n- It is not clear if figure 1 is a real, learned representation space or a visualisation. Some analysis of the clusters/distributions of the representations would be needed to confirm the assumptions of the objective functions. Simply put, is the objective doing what you want it to do?\n- The authors make many assumptions and references to symmetries in the representations but no theoretical or empirical evidence is given to support these claims. How do the authors know these symmetries exist and it is not clear what type of symmetries they refer to (from figure 1 I assume some type of translational symmetry).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a new objective function for self-supervised anomaly detection. The first contribution is the use of dual contrastive losses aimed at enforcing invariance to sample transformations and context transformations. The former enforces content alignment within the sample being invariant to the context changes, while the latter ensures distinct distributions given different contexts. The contexts themselves are created through heuristically derived transformations. The work also presents two scoring functions to determine threshold for anomalous results. The method is then evaluated on medical datasets, with some general benchmark datasets reported in the supplementary. The proposed method outperforms compared methods by some good margin in places, and some ablations to support the decision making is provided.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major**\n- From the weakness identified previously about symmetries, either (i) remove the reference to symmetries, or (ii) provide evidence, empirically or theoretically that supports these claims.\n- Compare against other existing scoring functions, rather than just comparing your own. This would give better justification for why the proposed method is superior.\n- Move the generalised dataset results into the main manuscript, and if possible, give some numerical measure of performance for comparison and benchmarking.\n- The authors mention that there is a small efficiency trade-off, but there seems to be no comparative results given that evaluates such efficiency, please add this.\n- The abstract could be re-written to introduce the problem statement, and solution better. It does not summarise the problem statement, research question, or the findings.\n- Please address the limitations of the work.\n\n**Minor**\n- Figure 1 could have an improved caption to assist in readability, for example it is not entirely clear (b), and (c). While it is fine it can be confusing rather than supporting the text, \n- There are no equation numbers in the manuscript.\n- The last Equation of page 5, where does the denominator '12' appear from?\n- The two projection heads should be formally defined in the text, additionally, a more clear and structured introduction to the two tasks, losses, and projection heads would improve readability.\n- There are no Standard Deviation given for CLIP-AD, AnomalyClip, and MVFA in Table 1.", "broader_impact_concerns": "I would advise the authors to add a broader impact statement, not only addressing the impact of the work, but also to reflect on the limitations. While I have no major concerns, it would be beneficial for the authors to reflect on this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)