Data Augmentation is a Hyperparameter: Cherry-picked Self-Supervision for Unsupervised Anomaly Detection is Creating the Illusion of Success
Authors: Jaemin Yoo
Tiancheng Zhao
Leman Akoglu
Venue: Accepted by TMLR ()

Abstract: Self-supervised learning (SSL) has emerged as a promising alternative to create supervisory signals to real-world problems, avoiding the extensive cost of manual labeling. SSL is particularly attractive for unsupervised tasks such as anomaly detection (AD), where labeled anomalies are rare or often nonexistent. A large catalog of augmentation functions has been used for SSL-based AD (SSAD) on image data, and recent works have reported that the type of augmentation has a significant impact on accuracy. Motivated by those, this work sets out to put image-based SSAD under a larger lens and investigate the role of data augmentation in SSAD. Through extensive experiments on 3 different detector models and across 420 AD tasks, we provide comprehensive numerical and visual evidences that the alignment between data augmentation and anomaly-generating mechanism is the key to the success of SSAD, and in the lack thereof, SSL may even impair accuracy. To the best of our knowledge, this is the first meta-analysis on the role of data augmentation in SSAD.

Published in Transactions on Machine Learning Research (07 (07/2023)
Data Augmentation is Hyperparameter: Cherry-picked Self-Supervision for Unsupervised Anomaly Detection is Creating the Illusion of Success
Jaemin Yoo
juemingoo@emu.ccm edu
Heinz College of Information Systems and Public Policy Carnegie Mellon University
Tiancheng Zhao
tianchenSandrew.com
School of Architecture Curnegie Mellon Universily
Leman Akoglu
lakkogha&hendrew.comm
Heinz College of Information Systems and Public Policy Carnegie Mellon University
Reviewed on OpenReview:
hilps:/ wwwparreiee..nl//orrrttto
Abstract
Self-supervised learning (SSL) has emerged as promising alternative to create supervisory signals to real-world problems, avoiding the extensive cost of manual labeling. SSL par- ticularly attractive for unsupervised tasks such as anomaly detection (AD), where labeled anomalies are rare or often nonexistent. A large catalog of augmentation functions has been used for SST--based AD (SSAD) on image data, and recent works have reported that the type of augmentation has significant impact accuracy. Motivated by those, this work sets out to put image-based SSAD under larger lens and investigate the rolc of data augmentation in SSAD. Through extensive experiments on 3 different detector models and across 420 AD tasks, we provide comprehensive numerical and visual evidences that the alignment between data augmentation and anomaly-generaiee
mechanism is the key to the success of SSAD, and the lack thereof, SSL may even impair accuracy. To the best of our knowledge, this is the first meta-analysis the role of data. augmentation in SSAD.
1 Introduction
Machine learning has made tremendous progress in creating models that can learn from labeled data. How- ever, the cost of high-quality labeled data is a major bottleneck for the future of supervised learning. Most recently, self-supervised learning (SSL) has emerged as a promising alternative; in essence, SSL transforms an unsupervised task into a supervised one by self-generating labeled examples. This new paradigm has had great success advancing NLP (Devlin al.) 2019 Conncau al. 2020 Brown al.
2020) and has
also helped excel at various computer vision tasks (Goyal et al., [022 Ramesh al. 2021 He et al. 2022
Today, SSL is arguably the key toward "unlocking the dark matter of intelligence" (LeCun & Misra
2021
SSL for unsupervised anomaly detection. SSL is particularly attractive for unsupervised tasks such anomaly detection (AD), where labeled data is either rare or nonexistent, costly to oblain, or nontrivial to simulate in the face unknown anomalies. Thus, the literature has seen a recent surge of SSL-based AD
(SSAD) techniques (Golan El- Yaniv 2018 Bergman & Hoshen 2020 Li et al.) 2021 Schwagg ct al.) 2021 Cheng et al. 2021 Qiu et al. 2021). The common approach they take incorporating self- celf-generated pseudo anomalies into training, which aims to separate those from the inliers. The pseudo-anomalies are often created

---

Published in Transactions on Machine Learning Research (07/2023)
Aligned Flip, con-Flip)
Controlled testbed
Rutto of samplee
Training data
alg
20 normal classes
Test Anoma
Test Normal
3 known anomalies
AUC=0.98
Normal x
allg( (x)
gen E {[uttOut, Flip, Invert]
Anomely sixl
Detector f aug
Unaligned (aug-CutOut, gen-Flip)
Test data
2. In-the-wild testbed
40 normal classes
Ratio of samples
Training Aug.
Normal x
Test Normal Tast Anama y
semantic anomalles
gen(x)
8
AUC=0.67
gen E {Cat, Dog, T..c
Anomaly score sixt
(a) SSAD Framework
(b) Testbeds
(c) Main finding: alignment matters
Figure 1: (best in color) A visual summary of our work. (a) In SSAD, we train a detector Saug from normal data x and pseudo anomalies aug(x) generated from an augmentation function aug, and feed the out of faug a score function 8. We denote the underlying anomaly- gennaaatgnn mechanism in test data by gen. (b) We use two testbeds in experiments: 1) 20 x tasks with known (or controlled) anomalies, and 2) 40 x 9 tasks for real-world semantic class anomalies. (c) We illustrate two cases with and without alignment between aug and gen: (left) data embeddings and (right) anomaly score histograms. SSAD succeeds when aug mimics gen, showing almost perfect AUC (top), while it fails when aug is misaligned with gen (bottom).
in one of two ways: (i) by transforming inliers through data augmentation or (ii) by "outlier-epposing" the training to external data sources (Hendrycks et al. 2019 Ding et al. 2022a The former synthesizes artificial data samples, while the latter uses existing wealdworld samples from external data repositories.
While perhaps re-branding under the name SSL, the idea of injecting artificial anomalies to inlier data to create a labeled training set for AD dates back carly 2000s (Abe et al. 2006; (Seinwart et al., 2005 Theiler & Cail 2003). Fundamentally, under the uninformative uniform prior for the (unknown) anomaly -ennualll-gneatnngg distribution, these methods are asymptotically consistent density level set estimators for the support of the inlier data distribution (Steinwart et al. 2005). Unfortunately, they are ineffective and sample-ineffcieen in high dimensions as they require a massive number of sampled anomalies to properly fill the sample space.
SSL-based AD incurs hyperparameters to choose. With today's SSL methods for AD, we observe a shift toward various non-uniform priors on the distribution of anomalies. In fact, current literature on SSAD is laden with many different aforementioned forms of generating pseudo anomalies, cach introducing its own inductive bias. While this offers means for incorporating domain expertise detect known types of anomalies, in general, anomalies are hard to define apriori one is interested to detect unknown anomalies. As consequence, the success on any AD task depends on which augmentation function is used or which external dataset the learning is exposed to as pseudo anomalies. In other words, SSL calls forth hyperporam- eter to choose carefully. The supervised MT, community has demonstrated that different downstream tasks benefit from different augmentation strategies and associated invariances (Ericsson et al.) 2022), and thus integrates these " "data augmentation hyperparameters" into model selection (MacKay et al. 2019 Zoph al., 2020 Ottoni et al., 2023 Cubuk et al. 2019). whereas problematically, the AD community appears to have turned blind eye the issuc. Tuning hyperparameters without any labeled data is admittedly challenging, however, wnsupervised AD does not legilómiiz cherry-picking crilical SSL hupergorauelers,, which creates the illusion that SSL is the "magic stick" AD and pver-represents the level of true progress in the field.
Evidence from the litcrature and simulations. Let us provide illustrative examples from the literature as well as our own simulation showing that SSL hyperparamelers have significant impact on AD performance. Golan & E1- Yaniv (2018) have found that geometric transformations (e.g. rotation) outperform pixel-level augmentation for detecting semantic anomalies (e.g. airplanes vs. birds). In contrast, Li et al. (2021) have
F.g.. rotation, blurring, masking, color jittering, CutPaste Li
2021 as well as "cocktail" angmentations like GEOM
Golan & Paniv 2018: and SimCLR (Chen al. 2020
2

---

Published in Transactions on Machine Learning Research (07 (07/2023)
cutout
p=0.000
Invert
p=0.000
cutpaste
p=0.000
color
p-0,003
cp-scar
p=0.000
flip
p=0,008
blur
0.869
rotate
p=0.020
noise
0.998
simcir
p=0.251
mask
1.000
geom
p-0,327
rotate
0.997
noise
p-0.699
geom
0.993
crop
p=0.916
p 1,000
cutpaste
p=0,991
crop
p- 1.000
mask
p-0,997
simelr
0.998
cp-scar
p-0,998 /
invert
p= 1.000
p=0,999 /
1.000
p=0.998
-02
-0.1
0.0
0.1
-0.2
-0.1
0.0
0.1
0.2
0.3
Difference in AUC
Difference in AUC
(a) DAE on gen:=CutOut
(b) DeepSAD on gen:-Invert
Figure 2: (best in color) Relative AUC (x-axis) of SSL-based DAE and DeepSAD in comparison to their un- supervised counterparts, respectively AE and DeepSVDD, on the controlled testbed with different anomaly- generating function gen in (a) vs. (b). Color augmentation names (y-axis) depicts their categories. (a) Local augmentations (in red) perform well thanks to the high alignment with Cen:=CutOut, while others (o.g. in green) can even hurt the accuracy significantly (p-values in gray). (b) Similarly, color-basod augmen- (ations (in green) improve the baseline AUC significantly since they agree with gen:=Iuvert, while several others even impair the accuracy. See our Finding 1 in Sec. for further details.
shown that such global transformations perform significantly poorly at detecting small defects in industrial object images (c.g. scratched wood, cracked glass, etc.), and thus proccoding to propose local augmentations such as cut-and-paste, which performed significantly better than rotation (90.9 vs. 73.1 AUC on average; see their Table 1). We replicate these observations in simulations; Fig. 2 shows that local augmentation functions (in red) improve performance over the unsupervised detector when test anomalies are simulated to be local (mimicking small industrial defects), while other choices may even worsen the performance (!).
On the other hand, Ding et al. 2022a consider three different angmentation functions as well as two external data sources for outlier-exposure (OE), with significant differences in performance (see their Table 4). They have compared baseline methods by picking CutMix augmentation, the best one in their experiment, on all but three medical datasets which instead OF from another medical dataset. Similarly, in evaluating OE-based AD (Hendrycks et al. 2019 Liznerski et al. 2022), the authors have picked different external datasets to training depending on the larget/lest dataset; specifically, they use as OE data the 80 Million Tiny Images (superset of CIFAR-10) to evaluate on CIFAR-10, whereas they use ImageNet-222K (superset of ImageNet-1K) to evaluate on ImageNet. Both studics provide evidence that the choice of self-supervision matters, yet raise concerns regarding fair evaluation and comparison to baselines.
As we argue later in our study, the underlying driver of success for SSAD is that the more the pseudo anomalies mimic the type/nature, of the true anomalies in the test data, the better the AD performance. This is perhaps what one would expect, i.c. is unsurprising, yet it is essential to emphasize that true anomalies are unknoun at training time. Any particular choice of generating the pseudo anomalies inadvertently leads to an inductive bias, ultimately yielding biased outcomes. This phenomenon was recently showcased through an eye-opening study (Ye et al.) 2021); a detector becomes biased when pseudo anomalies are sampled from a biased subset of true anomalies, where the test error is lower on the known/sampled type of even
anomalics, at the expense of larger errors on the unknown anomalics even when they are casily detected by an unsuperoised detector. We also replicate their findings in our simulations. Fig. [aj shows that augmenting the inlier Airplane images by rotation leads to improved detection by the self-supervised DeepSAD on average; but with different performance distribution across individual anomaly classes: compared to (unsupervisod) DeepSVDD, it better detects Bird, Frog, and Deer images as anomalies, yet falters in detecting Automobile and Truck-despite that these latter are easier to detect by the unsupervised DeepSVDD (!). Results in Fig. 3b for DAE are similar, also showcasing the biased detection outcomes under this specific self-supervision.
Our study and contributions. While the aforementioned studies serve as partial evidence, the current literature lacks systematic scrutiny of the working assumptions for the success of SSL for AD. Importantly,
3

---

Published in Transactions on Machine Learning Research (07 2202))
0.9
0.9
Deep SVDD
AE
0.8
Deep
0.8
AUC
0.7
AUC
0.7
0.6
0.5
0.6
0.4
automobile
truck
ship
horse
dog
cat
bird
frog
deer
ALL
truck
automobile
ship
horse
frog
deer
bird
dog
cat
ALL
(a) DeepSAD (vs. DeepSVDD)
(b) DAE (vs. AE)
Figure 3: AUC performances different classes of semantic anomalies, comparing SSL-basod DeepSAD and DAE with their unsupervised counterparts, respeclively DeepSVDD and AE, on CIFAR-10. We use Airplane as the inlier class and Rotate as the augmentation function. Self-supervision improves accuracy average (last column) but impairs it on certain individual anomaly classes that are easily detected by unsupervised models. This shows that the negative effect of biased supervision as observed by Ye et al. (2021) also applics to SSAD, leading to biased detection outcomes across anomaly types. See Obs. in Sec. for details.
hyperparameter selection and ht design choices for SSAD are left unjustified in recent works, and many of them even make different choices for different datasets in the same paper, violating the real-world setting of unsupervised AD. In this work, we sot out to put SSAD under a larger investigative lens. As discussod, Fig.s 2 and g give clear motivation for our research; comparing SSL-based models DAE and DeepSAD with their unsuperoised counterparts, respectively AE and DeepSVDD, showcases the importance of the choice of self-supervised augmentation. Toward deeper understanding, models over 420 different AD tasks, studying why and how self-supervision succeeds or otherwise fails. We show the visual summary of our work in Fig. [ including our main finding on the alignment.
run extensive experiments on 3 detector
Our goal to uncover pitfalls and bring clarity to the growing field of SSL for AD, rather than proposing yel-another SSAD model. To the best of our knowledge, this the first mela-analysis study on SSAD with carefully designed experiments and statistical tests. Our work is akin to similar investigative studies on other aspects of deep learning (Erhan et al., 2010 Ruff al. 2020a Mesquita et al. 2020) that aim to critically review and undersland trending research areas. We expect that our work will provide a better understanding of the role of SSL for AD and help steer the future directions of research on the topic. We summarize our main contributions as follows:
Comprehensive study of SSL image AD: Our study sets off to answer the following ques- lions: How do different choices for pseudo anomaly generation impact detection performance? Can data augmentation degrade performance? What is the key contributor to the success or the failure of SSL on image AD? To this end, we conduct extensive experiments on controlled as well as in-the-wild testbeds using 3 different types of SSAD models across 420 different AD tasks (see Fig. D
Alignment between pseudo and true anomalies: Our study presents comprehensive evidence that augmentation remains to be hyperparameter for SSAD the choice of which heavily influences performance. The "X factor" is the aligmment between the augmeniation function aug and the true coomall--gnerating function gen, where alarmingly, poor aligmment even hurts performance (see Fig, 2 or leads to a biased error distribution (see Fig. Bottom line is that effective hyperparameter selection essential for unlocking the game-changing potential SSL for AD.
For reproducibility, we open-source our implementations at Mitips://githnn com hommjininnyo/////omoommmo /SST=
2 Related Work and Preliminaries
2.1 Related Work on Self-supervised Anomaly Detection (SSAD)
Generative Models. Generative models learn the distribution of normal (i.e, inlier) samples and measure the anomaly score of example based its distance from the learned distribution. Generative models for
4

---

Published in Transactions on Machine Learning Research (07/2023)
AD include autoencoder-basse models (Zhou & Paffenroth 2017: Zong et al. 2018 generative adversarial networks (Akcay et al.]
2018: Zenati et. al. 2018), and flow-based models (Rudolph et al. 2022). Recent works Cheng et al. 2021 Ye et al. 2022) proposed denoising autoencoder (DAE)-based models for SSAD by
adopting Comainn-speciffe augmentation functions instead of traditional Gaussian noise or Bernoulli masking
(Vincent et al.) 2008: 2010). SSAD addresses the limitation of generative models, where out-of-disuribution
data exhibit higher likelihood than the training data, itself (Theis et al. 2015 Nalisnick et al. 2018).
Classifter-based Models. Angmentation prediction is to learn classifier by creating pscudo labels of samples from multiple augmentation functions. The classifier trained to differentiate augmentation functions is used to generate better representations of data. Many augmentation functions were proposod for AD in this sense, mostly for image data, including geometric transformation (Golan & El- Yaniv 2018), random affine transformation (Bergman & Hoshen 2020), local image transformation (Li et al. 2021 ) and learnable neural network-based transformation (Qiu et al. 2021). Outlier exposure (OE) (Ding et al. 2022a Hendrycks et al. 2019) uses an auxiliary datasct as pscudo anomalies for training a classifier that separates it from normal data. Since the choice of an auxiliary dataset has a large impact on the performance, previous works have chosen suitable dataset for each target task considering the nature of true anomalies.
Semi-supervised Models. Semi-supervised models assume that a few samples of true anomalies are given at training (Ruff et al. 2020b Ye et al.] 2021). A recent work (Ye et. al. 2021) has shown that observing a biased subset of anomalies induces a bias also in the model's predictions, impairing its accuracy on unseen types of anomalies even when they are easy to detect by unsupervised models. Motivated by this, we pose supervisd AD as a form of self-supervised learning in that limited observations of (pseudo) anomalies and its proper alignment with the distríbution of true anomalies play an essential role in the performance.
2.2 Related Work on Self- supe-supe Learning and Automating Augmentation
While our work focuses on SSL for anomaly detection problems in the absence of labeled anomalies to learn from, we also outline related work on SST. in other areas more broadly.
Self-supervised Learning. Self-supervised learning (SSL) has seen surge of attention for pre-training foundation models [Bommasani et al. 2021) like large language models that can generate remarkable human- like text (Zhou et al., 2023). Self-supervised representation learning has also offered an astonishing boost to a varicty of tasks in natural language processing, vision, and recommender systems (Liu et al. 2021 J. SSL
has been argued as the key toward "unlocking the dark maiter of intelligence" (LeCun & Misra 2021
Automating Augmentation. Recent work in computer vision (CV) has shown that the success of SSL relies heavily well-designed data augmentation strategies (Tian et al. 2020 Steiner et al. 2022 Touvron
et al. 2022 Fricsson et al. 2022) as in recent study on the semantic alignment of datasets in semi-supervise
learning Ralderon-Ramirez et al. 2022). Although the CV community proposed approaches for automating 2020), such approaches are not applicable to SSAD, where labcled data
augmentation (Cubuk et al. 2019:
are not available at the training. While augmentation in CV plays a key role in improving generalization by accounting for invariances (e.g. mirror reflection of a dog is still a dog), augmentation in SSAD presents the classifier with specific kinds of pseudo anomalies, solely determining the decision boundary. Our work is the first attempt to rigorously study the role and effect of augmentation in SSAD.
2.3 Related Work on Unsupervised Outlier Model Selection
Through this work, we show that data augmentation for SSAD is yet-another hyperparameter and thus more broadly relates to the problem of unsupervised (outlier) model selection. Unsupervised hyperparameter tuning (i.e. model selection) is nontrivial in the absence of labeled data (Ma et al. 2023). where the literature is recently growing (Zhao et al. 2021 2022 Zhao & Akoglu, 2022 Ding et al. 2022b Zhao et al., 2023). Existing works can be categorized into two groups depending on how they estimate the performance of an outlier detection model without using any labeled data. The first group uses internal performance measures that are based solely on the model's output and/o input data (Ma et al. 2023). The second group consists of meta-learning- based approaches (Zhao et al. 2021 Zhao & Akoolu 2022), which facilitate model selection for new unsupervised task by leveraging knowledge from similar historical tasks.
5

---

Published in Transactions on Machine Learning Research (07/ (20)))
2.4 Problem Definition and Notation
We define the anomaly detection problem Li et al. 2021 Qiu et al. 2021 as follows:
Given: Set D = fx, of normal data, where N is the number of training samples, and E Rd;
Find: Score function s(-) E R + R such that s(x) < ((x)) if x is normal and x' is abnormal.
The definition of normality (or abnormality) is different across datasets. For the generality of notations, we introduce an anomaly-gencrating function gen(.) R + Rd that creates anomalics from normal data. We denole a test set that contains both normal data and anomalies generated by gen(.), D gen-
The main challenge of anomaly detection is the lack of labeled anomalies in the training set. Self-supervised anomaly detection (SSAD) addresses the problem by generating -enudo-nnomalies through a data augmen- lation function aug(-) E IRd + Rd. SSAD trains a neural nelwork ()::0) E Rd + +R using aug(-), where 0 is the set of parameters, and defines a score function s(.) on top of the data representations learned by f. We denote a network f trained with aug by faug when there is no ambiguity.
2.5 Representative Models for SSAD
The main components of an SSAD model are an objective function I and an anomaly score function s. These determine how to utilize aug in the training of saug and how to quantify anomalies. We introduce three models from the three categories in Sec. respectively, focusing on their definitions of ( and s.
Denoising Autoencoder (DAE). The objective function for DAE (Vincent et al., 2008 is given as ((0) Exeoll/(aug(x)) #) x112. That is, Saug aims to reconstruct the original x from aug(x). We use the mean squared error between and the reconstructed version f(x; 0) as an anomaly score. The training of a vanilla (no-SSL) autoencoder (AE) is done by employing the identity function aug(x)
Augmentation Predictor (AP). The objective function for AP (Golan & El-Yaniv 2018 is given <x)=>XEE ExED Ek= (x); k), where K is the number of separable classes of aug, aug, is the k-th class of aug which is an augmentation function itself, and NLL(ý, y) = log is the negative loglikelihood. The idea is to train a K-class classifier that can predict the class of augmentation used to generate auga (x). The scparable classes are defined differently for cach aug. For example. [Golan & El- Yaniv (0018) sets K when Sang is an image rotation function and sets each aug, to the e-degree rotation with E (0, 90, 180, 270). Unlike DAE and DeepSAD, there is no vanilla model of AP that works without aug. The anomaly score s(x) is computed as sx) = EK [faug (x))]k, such that receives a high score if its classification is failed.
The objective function for DeepSAD Rull et al. 2020b) is given as ((0) (Ilf(aug(x); 0) cl))-1, where the hypersphere center e is set as the mean of the outputs obtained from an initial forward pass of the training data D. We then re-compute c every time the model is updated during the training. The anomaly score s(x) is defined as cl2, which is the squared distance between x and c. We adopt DeepSVDD (Ruff et al. 2018) as the no-SST, vanilla. version of DeepSAD by using only the first term of the objective function; it only makes the representations of data close to the hypersphere center.
DeepSAD.
(x; +
3 Experimental Setup
Models. We run experiments on the 3 SSL-based detector models introduced in Sec. 2.5 DAE, DeepSAD, and AP. We also include the no-SSL basclines AE and DeepSVDD for DAE and DeepSAD, respectively, with the same model architectures. Details on the models are given in Appendix A
Augmentation Functions. We study various types of augmentation functions, which are categorized into five groups Bullet colors are the same as in Fig.s
and 5
Geometric: Crop (Chen et al. 2020), Rotate, Flip, and GEOM (Golan & El- Yaniv 2018
Local: CutOut (Devries & Taylor 2017) CutPaste and CutPaste-scar (Li et al. 2021
Elementwisc: Blur (Chen et al. 2020). Noise, and Mask (Vincent et al. 2010
Color-bascd Invert and Color (jittering) (Chen et al. 2020).
Mixed ("cocktail"):: SimCLR (Chen et al. 2020
6

---

Published in Transactions on Machine Learning Research (07/2023)
Table 1: AUC of three detector models on CIFAR-10C and CIFAR-10; the first three gen functions are from CIFAR-1OC, where Cut. means CutOut, while Sem. mcans the anomalies of different semantic classes in CIFAR-10. Every model performs best when aug and gen are matched, supporting Finding | Different aug functions take the first place in Semantic, where the alignment with gen is unknown.
Auomaly-gencrating function (= gen)
Augment
DAE
DeepSAD
AP
aug)
Cut.
Flip
Invert
Sem.
Cut.
Flip
Invert
Sem.
Cut.
Flip
Invert
Sem.
CutOut
0.974
0.593
0.654
0.604
0.999
0,580
0.556
0.592
0.797
0.503
0.508
0.505
Flip
0.771
0.865
0.772
0.691
0.727
0.876
0.731
0.691
0.639
0.959
0.836
0.780
Invert
0.746
0.663
0.970
0.690
0.674
0.659
0.973
0.695
0.645
0.717
0.994
0.753
GEOM
0.813
0.726
0.724
0.621
0.938
0.758
0.699
0.682
0.760
0.941
0.881
0.863
Geometric functions make global geometric changes to the input images. Local augmentations, in contrast, modify only a part of an image such as by erasing a small patch. Flementwise augmentations modify every pixel individually. Color-basod augmentations change the color of pixcls, while mixed augmentations combine multiple calegories of augmentation functions. Detailed information is given in Appendix B
Datasets. Our experiments are conducted two kinds of testbeds, containing 420 different tasks overall. The first is in-the-wild testbed, where one semantic class is selected as normal and another class is selected as anomalous. We include four datasets in the testbed: MNIST (Garris) 1994; LeCun et al. 1998), Fash-
jonMNIST (Xiao et al. 2017). SVHN (Netzer et al. 2011), and CIFAR-10 10 (Krizhevsky et al. 2009 Since each dataset has 10 different classes, we have 90 tasks for all possible pairs of classes for each dataset. The second controlled testbed, where we adopt a known function as the momaly-generatho have full control of the anomalies. Given two datasets SVHN and CIFAR-10, we use three aug functions gen: CutOut, Flip, and Invert, making 30 different tasks (10 classes 3 anomalies) for each dataset. We denote these datasets by SVHN-C and CIFAR-10C, respectively.
function gen to
Evaluation. Given a detector model f and a test set Dgen containing both normal data and anomalies for each task, we compute the anomaly score s(x) for each x E Dgen- Then, we measure the ranking performance by the area under the ROC curve (AUC), which has been widely used for anomaly delection.
The extent of alignment is determined by the functional similarity between aug and gen. They are perfectly aligned if they are exactly the same function, and still highly aligned if they are in the same family, such aug: Rotate and gen:=Flip both of which are geometric augmentations.
4 Success and Failure of Augmentation
Our experiments are geared toward studying two aspects of augmentation SSAD: when augmentation succeeds or otherwise fails (in Sec. 4 and how augmentation works when it succeeds fails (in Sec. 5). We first investigate when augmentation succeeds by comparing variety of angmentation functions.
4.1 Main Finding: Augmentation is a Hyperparameter
We introduce the main finding on the relationship between aug and gen for the performance of SSAD, which we demonstrate through extensive experiments presented in Fig. and Table I
Let D ggen be a test set with the models with aug, aug', and without augmentation, respectively. Then, (i) faug surposses faug' if aug is better aligned with gen than aug' is, and (ii) f surpasses Saug if the alignment between aug and gen is poor.
Finding 1.
function gen, and faug, Saug', and f be detector
In Fig. we compare the performances of DAE and DeepSAD with their no-SSL basclines, AE and DeepSVDD, respectively. Recall that AP has no such vanilla baseline. We run the paired Wilcoxon signed- rank test (Groggel 2000) between faug and f. Each row summarizes the AUCs from 20 different tasks across two datasets (CIFAR-IOC and SVHN-C), ten classes each. We report the (pscudo) medians, 95% confidence intervals, and p-values for each experiment. The r-axis depicts the relative AUC compared with that of f. We consider aug to be helpful (or harmful) if the paluue is smaller than 0.05 (or larger than 0.95).
7

---

Published in Transactions on Machine Learning Research (07 2202)
rotate
p=0,000
cutout-0.1
p=0.000
p=0,000
p=0,000
cutout-0.2
p=0.000
cutpaste
p=0,000
cutout:0.3
p=0.000
p=0,000
cut
p=0.003
cutout
p=0,000
cutout-0.3 0.5
p=0.131
crop
p=0,000
cutout-0.6
p=0.969
Invert
p=0,000
cutout 0.7
p=1.000
noise
p=0,000
c
p=1.000
mask
p=1,000
p
cu9
p=1.000
color
p=1.000
-0.2
-0.1
0.0
0.1
simdlr
p= p=1.00
Difference in AUC
0.00
0.02
0.04
0.06
Difference
(a) Different aug hyperparameters
(b) Different model hyperparameters
Figure 4: (best in color) Relative AUC of DAE with different hyperparameters. (a) Controlled testbed with
Out:-CulOutcc where E [0.1, 0.9] depicts the size of removed patches. Anomaly function gen:=CulOut sets the average patch size to 0.19. (b) In-the-wild testbed summarizing 16 different hyperparameter settings of DAE (and AE); each row is generated from 5760 values (360 tasks x 16 models). See Obs and
In Fig.s 2a and 2b aug functions with the best alignment with gen significantly improve the accuracy of supporting Finding 1 the local augmentations in Fig. 2a and the color-based ones in Fig. 2b In costrast, the remaining aug functions make negligible changes even cause a significant decrease in accuracy. That is, the alignment between aug and gen determines the success or the failure of Saug on D Our observations are consistent with other choices of gen functions as shown in Appendix D.
In Table D we measure the accuracy of all three models on CIFAR-I0C and CIFAR-10 for multiple combi- nations of aug and gen functions. In the controlled tasks, aug gen performs best in all three models, even though their absolute performances
different. On the other hand, different aug functions work best for the semantic anomalies: Flip for DAE, GEOM for AP, and Invert for DeepSAD. This is because different semantic classes are hard to be represented by single aug function, making no aug achieve the perfect align- ment with gen. In this case, different patlerns are observed based on models: DAE shows similar accuracy with all aug, while AP shows clear strength with GEOM, as shown also in (Golan & EI- Yaniv 2018
Our Finding 1 may read obvious but has strong implications for selecting testbeds for fair and accurate evaluation of existing work. The literature does not concretely state, recognize, acknowledge the im- portance of alignment between aug and gen, even though it determines the success of given framework. Our study is the first to make the connection explicit and provide quantitative results through extensive experiments. Moreover, we conduct diverse types of qualitative and visual inspections on the effect of data augmentation, further enhancing our understanding in various aspects (discussed later in Sec. 4.3 and
4.2 Continuous Augmentation Hyperparameters
Next, we show through experiments that Finding T is consistent with augmentation functions with different continuous hyperparameters, and robust to different choices of model hyperparameters.
Observation The alignment between aug and gen determined not only by the functional type of aug, but also by the amount of modification made by aug, which is determined by its continuous hyperpanameterrs). Fig. 4a compares the effect of aug:=CutOut on the controlled testbed, varying the size e of crased patches in augmented images. For example, CutOut-O.2 represents that the width of an erased patch 20% of that of each image, making their relative area 4%. Note that the original CutQut used as gen selects the patch width randomly in (0.02, 0.33), making an average of 0.19 and thus aligning best with c= 0.2.
The figure shows that CutOut-c performs better with smaller values of c, and it starts decrease the AUC of the unsupervised baseline when e 2 0.6. This is because CutOut-c with small achieves the best alignment with gen, which removes small patches of average size Rs 0.2. Although the functional type of aug is the same as gen, the value of e determines whether faug succeods or not.
28ee Everything Obvious: *Once You Know the Answer. Duncan Watts. Crown Business, 2011.
8

---

Published in Transactions on Machine Learning Research (07 (002))
p=0.000
rotate
p=0,000
p=0.000
p=0,000
p=0.000
geom
p=0,000
geom
p=0.000
simdlr
p=0,000
cutpaste
p=0.00
crop
p=0,000
crop
p=0.000
cutpaste
p=0.251
cutout
p=0.000
noise
p=0,852
invert
p=0.000
cutout
p=0,814
noise
p=0.000
cD-scar
p=0,985
p=0.220
color
p=1,000
simdlr
p=0.652
p=1,000-
blur
p=1.000
blur
p=1.000
color
p=0.975
mask
p=1.000
0.00
0.02
0.04
0.06
0.08
0.050 0.025 0.000 0.025 0.050 0.075
Difference AUC
Difference
(a) DAE (vs. AE)
(b) DeepSAD (vs. DeepSVDD)
Figure 5: (best in color) Relative AUC on the in-the-wild testbed in which anomalies are different semantic classes. Color represents the category of augmentation. Geometrie augmentations (in blue) perform best in both cases while others make insignificant changes impair the baselines. See Obs.
Observation 2. Finding 7 is consislent with diffeernn lepperperometers of delector model f-
Our study aims to make observations that generalizable across different settings of detector models and hyperparameters. To that end, we run experiments on in-the-wild testbed with 16 hyperparameter settings of DAE, making total of 5,760 tasks (4 datascts x 90 class pairs x 16 scttings): the number of epochs in [64, 128), the number of features in {128, 256}, weight decay in {10-5, 5 10-9), and hidden dimension size in {64, 128]. Note that the hyperparameters of AP and DeepSAD are directly taken from previous works.
Fig. presents the results, which are directly comparable to Fig. 5a Both figures show almost identical orders of aug functions only with slight differences in numbers. This suggests that although the absolute AUC values are affected by model hyperparameters, the relative AUC with respect to the vanilla AE is stable since we use the same hyperparameter setting for DAE and AE. A notable difference is that the confidence intervals are smaller in Fig. 4b than in Fig. 5a due to the increased mumber of trials from 90 to 5,760.
4.3 Additional Observations: Error Bias and Semantic Anomalies
Based on our main finding, we introduce additional observations toward betler understanding of the effect of self-supervision on SSAD: error bias (in Obs. B and the alignment on semantic anomalies (in Obs. g
Observation 3. Given dataset containing multiple types of anomalies, self-supervision with aug creates bias in the error distribution of faug compared with that of the vanilla f-
Fig. compares DAE and DecpSAD with their no-SSL basclines, AE and DeepSVDD, respectively, on multiple types of anomalies on C0F In Fig. 3al Saug decreases the accuracy of ʃ on Automobile and Truck, which are semantic classes that include ground (or dirt) in images and thus can be easily separated from Airplane by unsupervised learning. The self-supervision with Rotate forces faug to detect other semantic classes including sky as anomalies, such as Bird, by feeding rotated airplanes as pseudo anomalies during the training. Such bias is observed similarly in Fig. 3b with DAE, while the amount of bias is smaller than in Fig. 3a This result shows that the "bias" phenomenon existing in semi-supervised learning (Ye et al. 2021 is observed also in SSL, and emphasizes the importance of selecting a proper aug function especially when there exist anomalies of multiple semantic types.
Observation 4. Geometric angmentations work best when anomalies are differrntt semantic classes, consis- tent with similar observations in previous works [Golan & El- Yanio 2018 Li et al. 2002)).
Fig. shows the results on 360 in-the-wild tasks across four datascts and 90 class pairs cach, whose anomalies represent different. semantic classes in the datasets. The alignment between aug and gen is not known a priori unlike in the controlled testbed. The geometric aug functions such as Rotate and Flip work best with both DAE and DeepSAD, showing their effectiveness in detecting semantic class anomalies, consistent with the observations in previous works (Golan & El- Yaniv many classes in those datasets, such as dogs and cats in CIFAR-10, sensitive to geometric changes such
2018 Li et al. 2021). One plausible explanation is that

---

Published in Transactions on Machine Learning Research (07 (07/2023)
ingut Image
imput image
vanilla (failure)
vanills (failure)
cutout (success]
cutouut
(a) Normal images on CIFAR-10C
(b) Anomalous images on CIFAR-100
input image
36
inpur
Vanilla
rotate (success) (failure)
6
36
Vanilla (failur))
16
rotate
(c) Normal images SVHN
(d) Anomalous images SVHN
Figure 6: Images from CIEAR-10C and SVHN, where the three represent original images and those re- constructed by AE and DAE faug, respectively. (a, b) aug:=CutOut and gen:=CutOut. (c, d) aug:=Rotate, and the digits and are normal and anomalous classes, respectively. The success and failure represent whether the images are assigned accurately to their true classes (normal vs. anomaly). DAE preserves the original normal images (on the left) while applying aug to anomalies (on the right), making them resemble normal ones with high reconstruction errors. As a result, DAE faug produces high AUC of (a, b) 0.986 and (c, d) 0.903, while those of AE f are low (a, b) 0.865 and (c, d) 0.549, respectively. See Obs.
rotation. Thus, aug creates plausible samples outside the distribution of normal data, giving faug an ability to differentiale anomalies that may look like augmented (i.e. rotated) normal images.
5 How Augmentation Works
Based our findings and observations in Sec. 4 perform case studies and detailed analyses to study how augmentation helps anomaly detection. We adopt DAE as the main model of analysis due to the following reasons. First, DAE learns to reconstruct the original sample x from its corrupted (i.e., augmented) version aug(x), which helps with the interpretation anomalics. Second, DAE is simply its unsupervised counterpart AE when aug = Identity, which helps study the eflect of other augmentations on DAE.
5.1 Case Studies on CutOut and Rotate Functions
We visually inspoct individual samples to observe what images DAE reconstructs for different inputs. Fig. shows the results on CIFAR-10C and SVHN with different aug functions.
Observation 5. Given normal sample DAE faug approrimates the identity function f(x) se Given an anomaly gxn faug approzimates the inverse aug if gen and aug are aligned, i.e., faug (gen(x)) ~ x.
We observe from Fig. 0 that both DAE and vanilla AE produce low reconstruction errors for normal data, but AE fails to predict them as normal since the errors are low also for anomalies. Given anomalies, DAE recovers their counterfactual images by applying aug-1, increasing their reconstruction errors to be higher than those from the normal images and achieving higher AUC than that of AE. is noteworthy that the task to detect digits anomalics from digits naturally aligned with the Rotate augmentation function because, in effect, DAE learns the images of rotated be anomalies during training. We show in Appendix D that similar observations derived from other pairs of normal classes and anomalies.
10

---

Published in Transactions on Machine Learning Research (07/2023)
100
Test Normel Anomaly
100
Tainiin Aug. Normal Test Anomaly
100
Training Aug.
Training Aug.
Ratio of samples (1)
80
80
Normal
Ratio of samples (%)
80
Test Normal
0.645
Ratio samples
Ratio of samples
Test Anomalyy
Test Anomaly
60
60
AUC: 0.869
60
AUC: 0.978
60
AUC: 0.900
40
40
40
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Reconstruction error
Reconstruction error
Reconstruction error
Reconstruction
(a) aug:=Identity
(b) aug:=CutOutt
(c) Fug:=Flip
(d) aug:=
Figure (best color) Reconstruction error distributions on CIFAR-10C with Automobile as the normal class and gen:=Flip. The distributions gradually shift to the right as aug changes the input images more and more: (a) Identity, (b) CutOut (local), (c) Flip (global), and (d) GEOM ("cocktaill" angmentation). The distributions of augmented samples and anomalies are matched the most in (c) when aug gen, which also achieves the smallest MMD: (a) 0.063, (b) 0.277, (c) 0.031, and (d) 0.399. See Obs
Training Aug.
Anomaly
Test Normal
Test Anomaly
1.0
1.0
0.8
0.8
Dimension
Dimension
0.6
Dimension2
Dimenzton22
0.6
0.4
0.4
0.2
Dimension
Dimension
Dimension 1
Dimension
(a) Flip (by groups)
(b) Flip (by scores)
(c) CutOut (by groups)
(d) CutOut (by scores)
Figure 8: (best in color) t-SNE visualization of data embeddings on CIFAR-10C under perfect alignment: = gen Flip and (c, d) aug = gen CutOut. The colors represent either (a, c) data categories or (b, d) anomaly scores. DAE achieves high AUC of 0.978 and 0.973 in both cases, respectively, despite the distributions of embeddings being different goometrically. See Obs.
(a, b) aug
5.2 Error Histograms with Varying Degrees Modification
Next, study the effect of augmentation the distribution of reconstruction errors normal data and anomalies. Fig. shows the error histograms on CIFAR-10C with different aug functions.
Observation 6.
The overall reconstruction errors are higher in DAF Saug than in vanälla AE f, and increase with the degree of modification that aug employs.
We compare DAE with three different aug functions and AE in Fig, 7 recall that faug with aug = Identity is equivalent to AE, In general, DAE shows higher errors than those of AE, leading to more right-shifted error distributions. This because the training process of DAE involves a denoising operation that increases the roconstruction errors for augmented data by mapping them normal oncs. The improved AUC of DAE is the result of increasing reconstruction errors for anomalies than those for normal data.
In Fig. 7 the amount of change incurred by aug increases from Fig. shifted to the right accordingly, as augmented data become more different from the normal ones. Notably, the distribution for anomalies is more right-shifted in Fig. 7c than in Fig. 7 resulting in higher detection AUC of 0.978, due to the better alignment between aug and gen. This showcases that the amount of alignment between aug and gen is the main factor that determines the errors (i.c., anomaly scores) of anomalics, while the general distributions are affected by the amount of modification made by aug. We show in Appendix that our observation is consistent in different gen functions for various tasks.
7d
The error distributions are
11

---

Published in Transactions on Machine Learning Research (07 (07/2023)
5.3 Embedding Visualization: Clusters and Separability
We visualize data embeddings generaled by DAE to understand how the embedding distributions of aug- mented training samples, test normal, and test anomalous samples affect the detection performance.
Observation 7. Data embeddings generated by DAE faug for normal data and anomalies are seporated when aug makes global changes in images, whereas mixed when aug makes local changes.
Fig. 8 illustrates data embeddings on the controlled testbed when aug and gen are the same. The first two and the last two figures exhibit different scenarios. In Fig.s Sa and [bb] there exist separate clusters: one for normal data, and another for augmented data and anomalies. In Fig.s Sc and [d] all data compase a single cluster without a separation between the different groups, even with the high AUC of 0.973. The difference between the two scenarios is mainly driven by the characteristic of the aug function, especially the amount of modification made by aug: Flip makes global changes, while CutOut affects only a part of each image.
Fig. supports Obs. through embeddings with different patch sizes of CutOut as in Fig. 4a Augmented
data start to create dense clusters as the patch size increases, and they are completely separated from the normal data and anomalies when c = 0.9 in Fig. 9d One notable difference from Fig. g is that Fig.s [9c and Dd represent failurcs, as the training augmented data are separated from both training normal data and test anomalies, while Fig. Sa represents a success due to the alignment between aug and gen.
6 Discussion
Summary of Findings and Take-aways. In this study, we took a deeper look at the role of augmentation in self-supervised anomaly detoction (SSAD) through extensive carcfully-designed experiments. Our findings and observations are summarized as follows:
Success and failure: The aligmment between augmentation aug and anomaly- generating mechanism
gen determines the success of SSAD (Finding) D and Obs 2). The degree of alignment is determined not only by the functional type of aug, but also by its continuous hyperparameters (Obs. D Empirical performance and bias: Geometric augmentation works best on datasets in which anomalies are different semantic classes (Obs. 4 When there exist multiple types of anomalies, self-supervision leads to a bias in the error distribution as observed also in semi-supervised learning (Obs. B).
Case studies: Given an anomaly, DAE approximates the inverse aug of augmentation if gen and aug are aligned (Obs. 5) and it increases overall reconstruction errors (Obs. 6). The distribution of data embeddings can be the key to estimate the degree of alignment between aug and gen (Obs. 7
Such findings clearly demonstrate that SSL for AD emerges as a data- or task-specific solution, rather than a cure-all panacca, effectively rendering it a critical hyperparameter. What makes this a nontrivial problem is that finding a good augmentation function is challenging in fully unsupervised settings if there is no prior knowledge of unseen anomalies. Our findings pave path toward possible solutions as discussed below.
Future Research Directions. We suggest transductive learning assume unlabeled test data are given at training time. Vapnik (2006) has been an advocate of transductive learning, according to whom, one should not solve a more general and harder intermediate problem and then try to induce to a specific one, but rather solve the specific problem at hand directly. In our setting, the use of unlabeled test data transductively, containing the anomalies to be identified, opens a possibility to tune the augmentation without accessing any labels. At the same time, it creates fundamental difference from existing approaches that imagine how the actual anomalics would look like or otherwise haphazardly choose an augnentation function, which may not well align with what is be detected.
future direction of SSAD, which is to
In transductive SSAD setting, the distribution of data embeddings can offer the grounds for an unsupervised measure of semantic aligmment between aug and gen. Fig.s S and g| show that the embeddings of augmented data are overlapped with those of anomalies under the perfect alignment, while they are separated without alignment. Transductive learning allows us to approximate the distance between aug and gen in the form of dist (Drm U D Daug: Drea)), where dist((..) is a set distance function, and Dern: D Daug, and Diest are sets of training normal, training augmented, and unlabeled test data, respoctively. Since Drest contains both normal data and (unlabeled) anomalies, its distance to Dtm U Dang is expected to be small when they align well.
12

---

Published in Transactions on Machine Learning Research (07/2023)
Training Anomaly
Ditversion
Dimension
Dimerston
Dimension
Dimersion 1
Dimension 1
Dimension
Dimension
(a) aug:=CutOut-0.1
(b) aut=:cnt
(c) aug:=CutOut-0.7
(d) aug:=CutOut-D..
Figure 9: t-SNE visualization of data embeddings on CIFAR-10C where aug:=CutOut-c and Cut.u.. The value of e represents the patch size of CutOut as in Fig. 4a The augmented samples and anomalics are matched well (a), but start to be separated as the aligument between aug and gen becomes weaker from (b) to (d). The result consistent with Fig. and supports Obs.
Towards this end, recent. work has designed an unsupervised validation loss for model selection for SSAD based on a quantitative measure of alignment in the embedding space (Yoo et al.) 2023b). They have later improved this loss into a diffcrentiable form toward augmentation hyperparamcter tuning in an end-to-end framework for image SSAD (Yoo al.] 2023a). Future research could continue to design new unsupervised losses well as differentiable augmentation functions amenable for end-to-end tuning. When equipped with those, one can then systematically apply modern HPO techniques (Bischl et al. 2023 to SSAD.
7 Conclusion
In this work, we studied the role of self-supervised learning (SSL) in unsupervised anomaly detection (AD). Through extensive experiments on SSL-based detectors across 420 AD tasks, showed that the alignment between data augmentation and anomaly-goneratinn and importantly, SSL can even huri detection performance under poor alignment. Our study is motivated (and our findings are also supported) by partial evidences weaknesses reported in the growing body of SSAD literature, and serves as the first systematic meta-analysis to provide comprehensive evidence on the success or failure of SSL for AD. We expect that our work will trigger further research on better understanding this growing area, in addition to new SSAD solutions that can aptly tackle the challenging problem of tuning the data augmentation hyperparamcters for unsupervised settings in principled fashion.
mechanism plays an essential role in the success of SSL;
Acknowledgements
This work is partially sponsored by PwC Risk and Regulatory Services Innovation Center at Carnegie Mellon University. Any conclusions expressed in this material are those of the author and do not necessarily reflect the views, expressed or implied, of the funding parties.
References
Naoki Abe, Bianca Zadrozny, and John Langford. Outlier detection by active learning. In KDD, pp. 504 509, 2006.
Samet Akcay, Amir Atapour Abarghouei, and Toby P. Breckon. GANomaly: Semi-supervised anomaly detection via adversarial training. In ACCV, 2018.
Liron Bergman and Yedid Hoshen. Classification-based anomaly detection for general data. In ICLR, 2020.
Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Mare Becker, Anne-Taure Boulesteix, Difan Deng, and Marius Lindauer. Hyperparam-
13

---

Published in Transactions on Machine Learning Research (07/2023)
eter optimization: Foundations, algorithms, best practices, and open challenges. WIREs Data. Mining. Knowl. Discon., 13(2), 2023.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 072558 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Aduances in neural information processing systems, 33:1877-1901, 2020.
Saul Calderon-Ramirez, Luis Oala, Jordina Torrents-Barrena, Shengxiang Yang, David Elizondo, Armaghan Moemeni, Simon Colreavy-Donnelly; Wojeiech Samek, Miguel A Molina-Cabello, and Ezequiel Lopez- Rubio. Dataset similarity to assess semisupervised learning under distribution mismatch between the labeled and unlabeled datasets. TEEE Transactions Artificial Intelligence, 4(2):282-25) 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for con- trastive learning visual representations. In ICML, 2020.
Zhen Cheng, En Zhu, Siqi Wang, Pei Zhang, and Wang Li. Unsupervised outlier detection via transformation invariant autoencoder. IEEE Access, 9:43991-44002, 2021.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaune Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In ACI, 2020.
Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Lc. Autoaugment: Learning augmentatiin strategies from data. In CVPR, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshop, 2020.
Jacob Devlin, Ming- Men Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidírec- tional transformers for language understanding. In NAACL-HLT, 2019.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with 15/p//0abb//10.0455525
cutout. CoRR, htss/778.00452, 2017. URL
Choubo Ding, Guansong Pang, and Chunhua Shen. Catching both gray and black swans: Open-set supervised anomaly detection. CoRR, abs/ abss/2203.rr 14506, 2022a.
Xueying Ding, Lingxiao Zhao, and Leman Akoglu. Hyperparameter sensitivity in deep outlier detection: Analysis and a scalable hyper-ensemble solution. Advances in Newral Information Processing Systems, 35: 9603 9616, 2022b.
Dumitru Erhan, Yoshua Bengio, Aaron C. Courville, Pierro-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res., 11:625 2010.
Linus Eriesson, Henry Gouk, and Timothy M. Hospedales. Why do self-supervised models transfer? on the impact invariance on downstream tasks. In BMVC, pp. 509. BMVA Press, 2022. URL https: http::2222.miiiiff.........
Michael D Garris. Design, collection, and analysis of handwriting sample image databases. 1994.
Izhak Golan and Ran El-Yaniv, Deep anomaly detection using geometric transformations. In NewrIPS, 2018.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivck Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self-supervised pretraining of visual features in the wild. CoRR, abs/2103.0198s 2021.
14

---

Published in Transactions on Machine Learning Research (07/2023)
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. A kernel method for the two-sample-problem. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann (eds.), NIPS, 2006.
David J. Groggel. Practical nonparametric statistics. Technometrics, 42(3) 317-318, 2000. doi: 10.1080/ 11401706:2000104800000 URL https://doi.org/ http://dd..rgg000000//00000 01000/00000000000000000000000000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun and Ishan Misra. Self-supervised learning: The dark matter of intelligence, 2021. URL https: dor-ddrrk--tttte ofteimataao----nllliigecc//////
/ //i..accbbook.om //tl:://el..sspeeriisdllaaaa
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proc. JEEE, (2111))22782224 1998.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In CVPR, 2021.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Dota Engineering, 35)) 177-876,20 2021.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Robert Müller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. arXiv preprint ar (rXii:2205.111 11474, 2022.
Martin Q Ma, Yue Zhao, Xiaorong Zhang, and Leman Akoglu. The need for unsupervised outlier model se- lection: A review and evaluation internal evaluation strategies. ACM SIGKDD Explorations Newsletter, 25(1), 2023.
Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse. Self-tuning net- works: Bilevel optimization of hyperparameters using structured best-response functions. arx preprinl arXiv:1903. 03088, 2019.
Diego P. P. Mesquita, Amauri H. Souza Jr., and Samuel Kaski. Rethinking pooling in graph neural networks. In NewrIPS, 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint Tfi:1810000118 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco,, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
André Luiz C Ottoni, Raphael M de Amorim, Marcela S Novo, and Dayana B Costa. Tuning of data augmentation hyperparameters in deep learning to building construction image classification with small datasets. International Journal of Machine Learning and Cybermetics, 14(1):171-186, 2023.
Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. Neural transformation learning for deep anomaly detection beyond images. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot. text-to-image generation. Tn TCML, 2021.
15

---

Published in Transactions on Machine Learning Research (07/2023)
Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Fully convolutional cross-scalc-ftows for image-based defect detection. In WACV, 2022.
Lukas Ruff, Nico Görnitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Robert A. Vandermeulen, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In TCML, 2018.
Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Glaus-Robert Müller, and Marius Kloft, Rethinking assumptions deep anomaly detection. CoRR, (bss/2006.0033// 2020a.
Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In ICLR, 2020b.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. SSD: A unified framework for self-supervised outlier detection. In ICLR, 2021.
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lueas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Ingo Steinwart, Don Hush, and Clint Scovel. classification framework for anomaly detection. JMLR, 6(2), 2005.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: novelty detection via contrastive learning on distributionally shifted instances. In NeurIPS, 2020.
James P Theiler and D Michael Cai. Resampling approach for anomaly detection in multispectral images. Proceedings of the SPIE, volume 5093, pp. 230-240, 2003.
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models.
arXiv preprint ar arXic:1511.01..0 2015.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? NewrIPS, 33:6827-6839, 2020.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In ECCV 2022, pp. 516 533, 2022.
Vladimir Vapnik. Estimation of dependences based on empirical data. Springer Science & Business Media, 2006.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. J. Mach. Learn. Res., 11:3371 3408, 2010. doi: (t55///17700011533339
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image datasct for benchmarking machine learning algorithms. CoRR, abs/ 1708.07747 2017.
Fci Ye, Chaoqin Huang, Jinkun Cao, Maosen Li, Ya Zhang, and Cown Lu. Attribute restoration framework for anomaly detection. IEEE Trans. Mullim., 24:116 127, 2022.
Ziyu Ye, Yuxin Chen, and Haitao Zheng. Understanding the effect of bias in deep anomaly detection. In IJCAI, 2021.
Jacmin Yoo, Lingxiao Zhao, and Leman Akoglu. End-to-end augmentation hyperparameter tuning for self- supervised anomaly detection. arXiv preprinl, 2023a.
Jaemin Yoo, Yue Zhao, Lingxiao Zhao, and Leman Akoglu. DSV: an alignment validation loss for self- supervised outlier model selection. In ECML. PKDD, 2023b.
16

---

Published in Transactions on Machine Learning Research (07 (0722)23)
Sergey Zagoruyko and Nikos Komodakis. In BMVC, 2016.
Houssam Zenati, Manon Romain, Chuan-Sheng Foo, Bruno Lecouat, and Vijay Chandrasekhar, Adversarially learned anomaly detection. In ICDM, 2018.
Yue Zhao and Leman Akoglu. Towards unsupervised HPO for outlier detection. arXiv:2208. 11727, 2022.
ar preprinl
Yue Zhao, Ryan Rossi, and Leman Akoglu. Automatic unsupervised outlier model selection. In Advances in Newral Information Processing Systems, pp. 4489-4502, 2021.
Yue Zhao, Sean Zhang, and Leman Akoglu. Toward unsupervised outlier model selection. In 2022 TEEE International Conference on Data Mining (ICDM), pp. 773 782. IEEE, 2022.
Yue Zhao, Xueying Ding, and Leman Akoglu. Fast unsupervised model tuning with hypernetworks for deep outlier detection. arxiv, 2023.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. CoRR, abs/1708.0488) 2017. URL http://arxiv.org/abbee http://bbs/111.....1 1708.04896
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from Bert to Chatgpt. arXiv preprint ar arXit:2302.09/11 2023.
Chong Zhou and Randy C. Paffenroth. Anomaly detection with robust deep autoencoders. In KDD, 2017.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dae-ki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018.
Barret Zoph, Ekin D Cubuk, Golnaz Ghiasi, Tsung- Yi Lin, Jonathon Shlens, and Quoc V Le. Learning data augmentation strategies for object detection. In ECCV, pp. 566 583. Springer, 2020.
17

---

Published in Transactions on Machine Learning Research (07 (07/2023) /023)
A Detailed Information on Detector Models
In our experiments, we use the model structures used in previous works. For AE and DAE, we adopt the structure used in (Golan El-Yaniv 2018). The encoder and decoder networks consist of four encoder and decoder blocks, respectively. Each encoder block has convolution layer of 3 kernels, batch normalization, and ReLU activation function. A decoder block is similar to the encoder block, except that the convolution operator is replaced with the transposed convolution of the same kernel size. The number of epochs and the size of hidden features are both set to 256, and the number of convolution features is (64, 128, 256, 512) for the four layers of the encoder block, respectively.
For AP (Golan & Yaniy 2018) and DeepSAD (Ruff et al., 2020b), we use their official implementations in experiments The structure of AP based on Wide Residual Network (Zagoruyko Komodakis 2016 and DeepSA A D is based on LeNet-lype convolutional neural network.
B Detailed Information on Augmentation Functions
We provide detailed information augmentation functions that we study in this work. We use the official Py'Torch implementations of augmentation functions and their default hyperparameters when available.
Geometric augmentations modily images with geometric functions:
Rotate (random rotation) makes a random rotation an image with a degree in {0, 90, 180, 270}. Crop (random cropping) randomly selects a small patch from an image whose relative size is between 0.08 and 1.0, resizes to the original size, and uses it instead of the given original image.
Flip (vertical flipping) vertically flips image.
GEOM (Golan & El-Yaniv 2018) applies three types of augmentations at the same time (and in this order): Rotate, Flip, and Translate, where Translate denotes a random horizontal or vertical translation by 8 pixels.
Local augmentations change only a small subset of image pixels without affecting the rest.
CutOut (random erasing) (Devries & Taylor 2017 Zhong et al. 2017) erases a small patch from an image, replacing the pixel values zero (i.c., black pixcls). The patch size is chosen randomly from (0.02, 0.33). Paste (Li et al. 2021 copies small patch and pastes it into another location in the same image. The difference from CutOut is that CutPaste has no black pixels in resulting images, making them more plausible. The patch size is chosen from (0.02, 0.15).
Cut Paste-scc eL al. 2021) is variant of CutPaste, which augments thin scar-like paiches instead of rectangular ones. The patch width and height are chosen from (10, 25) and (2, 16), respectively, in pixels. The selected patches are rotated randomly with a degree in (-45, 45) before they are pasted.
Elementwise augmentations make a change in the value of each image pixel individually (or locally).
Noise (addition of Gaussian noise) |Vincent et al. 2010 is traditional augmentation function used for denoising autoencoders. It adds a Gaussian noise with a standard deviation of 0.1 to each pixel. Mask (Bernoulli masking) (Vincent et al. 2010) conducts a random trial to each pixel whether to change the value to zero or not. The masking probability 0.2.
Blur (Gaussian blurring) (Chen et al. 2020) smoothens image by applying a Gaussian filter whose kernel size is 0.1 of the image. The of the filter is chosen randomly from (0.1, done in the SimCLR function.
Color augmentations change the color information of an image without changing actual objects.
Color (Color jittering) (Chen et al. 2020) creates random changes image color with brightness, contrast, saturation, and huc. The amount of changes the same as the SimCLR function.
18

---

Published in Transactions on Machine Learning Research (07 2022)
p=0.000
Invert
p-0.000
rolate
p=0.000
color
p=0.000
p=0.000
simclr
p=0.00
cp-scar
p=0,000
rotate
p=0,001
cutpaste
p=0.000
flip
p=0,001
invert
pm0,000
geom
pm0.116
crop
p=0,000
CD-scar
p=0,001
cutout
9-0.000
crop
p00.778
p=0.005
cutpaste
p=0.050
pm0.383
p-0.2
color
p=0.411
cutout
p 0.206
mask
p=0.63
noise
p=0.470
blur
pm0.825
blur
P- 0.539
0.1
0.2
0.3
0.2
0.3
Difference AUC
Difference AUC
(a) DAE (gen:=Flip)
(b) DAE (gen:=Invert)
fip
p=0.000
cutout
p=0,000
p=0.000
geom
p=0,000
geom
p-0.000
cutpaste
p-0,000
p=0.001
p-0.000
noise
p-0.545
p=0.002
color
p=0.761
color
p=0.001
crop
p-0.876
rotate
p-0,046
simdlr
p=0.980
noise
p=0.148
cutout
p=0,978
flip
0=0.397
cutpaste
p-0,916
Invert
p=0.805
p=
crop
p=0.932
cp scar
p=0.982
0.974
p=0.996
p= 0.963
-0.
0.0
0.1
0.2
0.3
-0.1
0.0
0.1
0.2
0.3
Difference
Difference AUC
(c) DeepSAD (gen:=Flip)
(d) DeepSAD (gen:=CutOut)
Figure 10: (best in color) Relative AUC of (a, b) DAE and (c, d) DeepSAD with respect the no-SSL baselines on the controlled testbed with anomaly-generating functions (a, e) gen:=Fliip, (b) gen:=Invert, and (d) gen:=CutOut. Color in augmentation names denotes their categories. Angmentation functions aligned with the anomaly-generating functions perform well, supporting Finding
Invert (Color inversion) inverts the color information of an image. In the actual implementation, it returns x for each pixel x E [o, 1].
Mixed augmentations combine augmentations multiple categories, making unified changes.
SimCLR (Chen et al. 2020 has been used widely in the literature as general augmentation function TTak et al. 2020 It conducts the following at the same time (and in the given order): cropping,
horizontal flipping, color jittering, gray scaling, and Gaussian blurring.
C Full Results on Individual Datasets for Main Finding
We present more results of relative AUC on DAE and DeepSAD, compared with their no-SSL baselines. Our additional experiments support Finding and Observation which are presented informally as follows:
Finding 1 Self-supervised detection performs better with better alignment between aug and gen functions, and fails (i.c., performs worse than the unsupervised bascline) under poor alignment.
Obs. Geometric aug works best with synthetic class anomalies (i.e., in-the-wild testbed).
Fig. [0 shows the relative AUC on the controlled testbed across two datasets CIFAR-10C and SVHN-C and ten classes each. The missing combinations, DAE with gen:=CutOut and DeepSAD with gen:=I are given in the main paper. All four cases in the figure with different models and anomaly types support our main finding, showing the generalizability of work into various combinations of aug and gen.
19

---

Published in Transactions on Machine Learning Research (07 2022)))3
p=0,000
geom
p-0.000
p=0.000
rotate
p=0.000
rotate
p=0.000
flip
p=0.000
cutpaste
p=0,000
cutpaste
p=0.000
fip
p=0,000
cutout
p=0,000
geom
p=0,001
cp scar
p=0.000
cutour
p=0,000
crop
p=0.167
p-0,000
p-0.764
p=0,000
p=0.982
color
p=0,000
p=0.988
blur
P=0.011
Invert
p=1.000
noise
p=0.609
color
p=1.000
mask
pm0.635
simclr
p=1.000
0.000 0.005
0,010
0.020
-0.02
-0.01
0.01
Difference AUC
Difference AUC
(a) MNIST
(b) INSh
rotate
p=0,000
rotate
p=0,000
p=0.000
p=0,000
cp-scar
p-0.000
invert
p-0,000
geom
p=0.000
color
p-0.002
cutpaste
p=0.000
p=0.000
p=0.000
cutpaste
p=0.000
cutout
p-0,000
p-0,003
simdl
p=0.002
crop
p=0.026
Invert
p=0.000
cutout
p=0,000
noise
p=0,000
noise
p=0.017
mask
p=0,000
mask
p=0.816
p=1,000
p=1.000
color
p=1.000
p=0.922
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
Difference
Difference AUC
(c) SVHN
(d) CIFAR-10
Figure 11: (best in color) Relative AUC of DAE with respect its no-SSL baseline, AE, on the in in-the-wild testbed in which anomalies (d) CIFAR-10. Color in the augmentation names denotes their categories. The geometric augmentations (in blue) perform best. in all datasets, improving the baselines, while others make insignificant changes impair the baselines. The patterns are consistent with Fig. 5a in the main paper. See Obs.
different semantic classes: (a) MNIST, (b) FashionMNIST, (c) SVHN, and
Fig.s 11 and 12 show the relative AUC on the in-the-wild testbed for four individual datasets. That is, Fig.s 5a and 5b are statistical summaries of Fig. [[ [2, respectively, over the four datasets. The results show that every dataset in the testbed exhibits Obs. although the characteristics of datasets are different from each other, supporting the generality of our findings toward different datasets.
Case Studies on Other Augmentation and Anomaly Functions
We conduct additional case studies to support Obs. on different types of gen functions. Our observation is presented informally as follows:
Obs. DAE applies a-) anomalies, while making no changes on normal data.
Controlled Testbed. Fig. 13 ssoow images in CIFAR-10C when (top) gen:=Flip and (bottom) Invert DAE applies the inverse augmentation aug-1 to the anomalies, reconstructing normal-like ones; in Fig. 13b the given images are rotated in the reconstructed ones when aug:=Flip, while in Fig. [3d the color of given anomalous images is inverted back in the reconst ones when aug:=Invert. Tn contrast, the vanilla AE reconstructs both normal and anomalous images close to the input images, as stated in Obs.
In-the- Wild Testbed. Fig. 14 shows the images of SVHN, in which anomalies are images associated with different digits. We study four aug functions (from top to bottom): Flip, CutOut, CutPaste, and SimCLR.
20

---

Published in Transactions on Machine Learning Research (07/202) 2023)
rotate
p-0.014
geom
p-0.000
geom
p=0.116
simclr
p=0,000
p=0.517
crop
p=0.000
crop
p-0,943
cutpaste
p-0.010
p=0,959
p=0.060
noise
p=0.999 /
p=0,003
cutpaste
p=0.987
rotate
p=0,203
cp-scar
p=1.000
cp-scar
P-0.520
cutout
p=1.000
flip
p=0,978
color
p=1000
blur
p=1,000
blur
mask
Invert
p=1.000
color
p=1.000
p=1.000
p=1.000
-0.100 1..0-0.075 075 075-0. -0.0550-0.02 0,000 0.025
-0.15
-0.10
-0.05
0.00
Difference AUC
Difference AUC
(a) MNIST
(b) PashionMNIST
f1pp
p=0.000
rotate
p=0.000
rotate
p=0,000
invert
p-0.000
crop
p=0,000
flip
p=0.000
simdl
p=0,000
geom
geom
p=0,000
color
p=0.001
cutout
p=0.000
simclr
p=0.018
cutpaste
p-0.012
crop
p=0.390
p=0.026
cutpaste
p=0.478
blur
p=0,200
p=0.536
p=0.638
noise
p=0.581
Invert
pm0.851
blur
0=0.813
noise
p=0.975
cutout
p.0.963
color
p=0.992
mask
p=0.999
0.05
-0.05
0.05
Difference in AUC
Diffarance AUC
(c) SVHN
(d) CIFAR-10
Figure 12: (best in color) Relative AUC of DeepSAD with respect to its no-SSL baseline, DeepSVDD, on the in-the-wild testbed in which anomalies are different semantic classes: (a) MNIST, (b) FashionMNIST. (c) SVHN, and (d) CIFAR-10. Color in the augmentation names represents their categories. The geometric augmentations (in blue) perform best in all datasels, improving the baselines, while others make insiguificant changes or impair the baselines. The patterns are consistent with Fig. 5b See Obs.
21

---

Published in Transactions on Machine Learning Research (07 (07/202) 2022)
imput image
input image
vanillaa (failure)
vanilla (failure)
fllip (success)
flip (surcesss)
(a) Normal images on CIFAR-10C (gen:=Flip)
(b) Anomalous images on CIFAR-10C (gen:=Flipp
Input ge
input image
Vanilla (failure)
vanilla (failure)
invert (success)
invert (successs)
(c) Normal images on CIFAR-10C (gen:=Invert)
(d) Anomalous images on CIFAR-LOC (gen:=Invert)
Figure 13: Images from CIFAR-10C, where the three rows represent original images and those reconstructed by the vanilla AE and DAE, respectively: (a, b) aug = gen Flip with Automobile as the normal class, and (c, d) aug gen Invert with Frog as the normal class. The success and failure represent whether the images assigned accurately to their true classes not (normal vs. anomaly). DAE preserves the original images (on the left), while applying aug-1 to anomalies (on the right), making them resemble normal ones with high reconstruction errors. This allows DAE to achieve higher AUC than AE, supporting Obs.
Task 1: 4 vs. 7. When =ut:-Rotate and the task is 4 (normal) vs. 7 (anomalous), DAE successfully generates 4-like images from 7 by applying the inverse of Rotate, which is also a rolation operation ion but with a different degree. This because the digit 7 can be considered or somewhat resembles rotated as shown in Fig. 14b as in the case of vs. Fig.s 6c and 6d
Task 2: O vs. We study another task of (normal) 3 (anomalous) with the remaining aug functions. CutOut and CutPaste generally work well, since the scarred images of can look like 3 by chance. CutOut works best with anomalous images of black background, since the inverse function of CutOut is to fill in the black erased patch; if a given image has a white background, it is difficult to find such a patch to revert by a Cut does not require an anomalous image to have a background of a specific color, since it copies and pastes an existing patch instead of erasing image pixels.
The SimCLR augmentation in Fig.s 14g and 14h unlike the other aug functions, changes the information of color and shape at the same time. This because SimCLR "cocktail" augmentation that creates large degree of modification by combining multiple augmentation functions. The degree of change significant even with normal images in Fig. 14g which makes it perform poorly in terms of detecting anomalies based anomaly scores; both normal and anomalous images are reconstructed into similar forms.
E Error Histograms on Other Augmentation and Anomaly Functions
We present more detailed results on error histograms to support Obs. on differont tasks and different types of gen functions. We informally present our observation as follows:
Obs. Reconstruction errors are higher in DAE than in AE, increasing with the degree change.
Controlled Testbed. Fig. 15 shows the error histograms on CIFAR-10C with two types of gen: CutOut and Invert. The figure supports Obs. by presenting identical patterns in Fig. 7 even with different gen
22

---

Published in Transactions on Machine Learning Research (07/202) 2023)
imput image
4
74
14
4
4
input
vanilla (failure)
A
1
4
4
4
vanilia (failure)
rotate
A
4
4
rotate
(a) Normal images with a Rotate (4 vs. 7)
(b) Anomalous images with aug:= Rotate (4 vs. 7)
Input image
0
input image
3
Vanilla (failure)
50
10
vanilla (failure)
3
cutout (success)
O
0
10
cutout
(c) Normal images with aug: CutOut (0 vs. 3)
(d) Anomalous images with aug:=CutOuu CutOut (0 vs. 3)
Imput image
input image
3
3
vanilla (fallure)
00
0
vanilla (fallure)
3
3
cutpaatte (sucees))
0
cutnaste (success)
0
0
(e) Normal images with aug:- Cat--utttate (0 vs. 3)
(f) Anomalous images with aug: CuttaCeee (0 vs. 3)
Input Image
905
Input image
vanilla (fallure)
O
0
vanilla (fallure)
simclr (Bucees))
simclr (success)
(g) Normal images with aug:=SimCT.R (0 vs. 3)
(h) Anomalous images with aug:=SimCLR (0 vs. 3)
Figure 14: Images from SVHN, where the three rows represent original images and those reconstructed by AE and DAE, respectively. (a, b) Rotate works similarly as in Fig.s 6c and 6d since the digit resembles the rotated digit 4. Given the anomalous images of 7, the DAE recovers the 4-like images by rotation. (c, d, f) DAE with CutOut and CutPaste works well in the task 0 (normal) vs. 3 (anomalous), because the images of 0 with crased patches can mimic those of 3. (g. h) SimCLR changes not only the shapes of digits, but also the color information due to the high degree of augmentation. The results support Obs.
23

---

Published in Transactions on Machine Learning Research (07/2023)
100
100
100
100
Normal Test Anomaly
Training Aug. Test Normal
Training Aug.
Training Aug.
Ratio of samples (%)
80
Ratio of samples, (%)
80
Ratio of samples. (9%
80
Test Normal
Ratio of samples (9%)
80
Test Normal
Test Anomaly
Test Anomaly
Test Anomaly
AUC: 0.780
60
60
AUC: 0.973
60
AUC: 0.907
60
AUC: 0.737
40
20
20
20
20-
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Reconstruction error
Reconstruction error
Reconstruction error
Reconstruction error
(a) aug Identity
(b) aug Out
(c) :ug:=GEOM.
(d) aug: = Invert
100
100
100
100
Test Normel Test Anomaly
Training Aug.
Training Aug.
Training Aug. Normal
Patio of samples (%)
Patio of samples (%)
Normal Test Anomaly
Rallo of samples 1%)
80
Test Normal
Ratio of samples 1%)
80
Test Anomaly
Test Anomaly
AUC: 0.698
60
60
AUC: 0.723
60
AUC: 0.824
60
AUC: 0.990
40
40
40
40
20
20
20
20
01
0.1 Reconstruction error
0.1 Reconstruction error
0.1 Reconstruction error
0.1
Reconstruction emror
(e) adg::Identity
(f) aut:=CutPaaste
(g) aug:=Rotate
(h) GFO:
Figure 15: (best in color) Reconstruction errors on CIFAR-10C with Automobile as the normal class and gen:=CutOut (in the top row) and gen:=Invert (in the bottom row). The distributions gradually shift to the right as the augmentation aug changes the imput images more and more: (a) Identity, (b) CutOut (local), (c) GEOM ("cocktaail" augmentation), and (d) Invert (color-based), which inverts the values of all pixels. The distributions of augmented data and anomalics are matched the most in (b) and (h) when aug = gen, which is the case of perfect aligmment, consistently with Fig.
The results support Obs. 6
100
100
100
100
Teat Nommal
Training Aug. Test Normal
Training Aug. Test Normal
Training Aug. Test Normal
Rabio of samples (%)
60
Test Anomaly
Rabio of samples (%)
80-
Rabio of samples (%)
80-
Ratio of samples (%6
80-
Test Anomaly
Test Anomaly
Test Anomaly
AUC: 0.549
60
60-
AUC: 0.591
60
AUC: 0.903
60
AUC: 0.612
20
20
20-
20-
0-
0-
0.00
0.05
0.10
0.15
0.00
0.05
0.10
0.15
0.00
0.05
0.10
0.15
0.00
0.05
0.10
0.15
Raconstruction error
Reconstruction enror
Reconstruction error
Reconstruction error
(a) aug Identity
(b) aug Cug:=Cutaaste
(c) aug: -Rotaee
(d) aug:=
Figure 16: (best in color) Reconstruction errors on the SVHN data for 6 (normal) vs. 9 (anomalous). Similar observations are derived as in Fig. 7 since the gen function here can be thought of as approximate Rotate. Still, the error distribution of anomalies is not as clearly separated as Fig.s 7 and [5] since the alignment between aug and gen is not perfect. The results support Obs. on the in-the-wild testbed.
functions. notable observation is that the distribution of augmented data is more right-shifted in Invert than in GEOM, even though GEOM is "cocktail" approach that combines multiple augmentations. This is because Invert changes the value of every pixel simultaneously to invert the color of an image, which results in dramatic change with respect to pixel values.
In-the-Wing Wild Testbed. Fig. 16 shows the reconstruction errors on the SVHN dataset with different aug functions. The task is 6 (normal) vs. (anomaly), where gen can be thought of as approximate rotation. Rotate works best among the four aug options thanks to the best alignment with gen. GEOM makes the most right-shifted distribution of augmented data, as in Fig. 7 due to its "cocktail" nature. The result on SVHN shows that. Obs. B is valid not only for the controlled testbed, but also for the in-the-wild bed.
24

---

Published in Transactions on Machine Learning Research (07 (072))
Training
Test Normal
Test Normal
Test Anomaly
1.0
1.0
0.8
0.8
0.6
Dimerss ion
Dimonstonn
0.6
0.4
0.2
Dimension
Dimension
Dimension 1
Dimension
(a) Invert (by groups)
(b) Invert (by scores)
(c) Flip (by groups)
(d) Flip (by scores)
Figure 17: (best in color) t-SNE visualization data embeddings on CIFAR-10C when gen:=Invert: (a, b) aug:=Invert and (c, d) aug:=Flip. The colors represent either (a, c) data categories or (b, d) anomaly scores. The first and the last two cases show different patterns: (a, b) aug:=Invert creates two separate clusters claimed in Obs. 7 achieving AUC of 0.990, thanks to the perfect alignment with gen. (c, d) aug:=Flip still achieves high AUC of 0.889, although its alignment with gen:=Invert is unclear, since it puts anomalies in between normal and augmented data.
F Embedding Visualization on Other Augmentation and Anomaly Functions
We introduce more experimental results on the embedding visualization on both controlled and in-the-wild testbeds, supporting Obs. 7 with different combinations of aug and gen functions. We informally present our observation as follows:
Obs.
Embeddings make separate clusters with global aug, and are mixed with local aug.
Fig. shows the results on CIFAR-10C with gen:=lnvert and two different aug functions. In Fig.s 17a and 17b when aug:=Invert achieves the perfect alignment with gen and creates global changes in the image pixels through augmentation, the points make separate clusters supporting Obs. and result in AUC of 0.990.
In Fig.s 17c and 17d when aug:=Flip exhibits imperfect alignment with gen, it still achieves high AUC of 0.889. This is because it succeeds in putting the anomalies in between normal and augmented samples in the embedding space. This allows f to effectively detect the anomalies.
G Experiments with MMD
For the in-the-wild testbed, where gen is not given explicitly, we can utilize the maximum mean discrepancy (MMD) (Gretton et al. 2006) between augmented data and test anomalies to approximate the functional similarity. We create a set of augmented data by applying aug to the normal data. Then, we generate the embeddings of this set of data and test anomalies using the pretrained ResNet50 (He et al. 2016) as an encoder function, since pixel-level distance in raw images is not quite reflective of semantic differences general. Lastly, compute MMD between the embeddings of the two sets.
Given each task, let Dj and D2 be the sets of augmented data and test anomalies, respectively. First, we randomly sample M instances from D2 and D2, respectively, and denote the results by DÍ and D/2 We set M 256, which is large enough to make stable results over different random seeds. Then, MMD is computed between D' and D' as follows:
MMD(D,,D') M
Σ
((x1,x)) +
Σ
Σ Σ x(x,
(1)
XIED XIED
X2ED/ X2ED
XIED x2ED
where the kernel function k(-,-) is defined on the outputs of a pretrained encoder network di
k(x1, x2) (((o(x1),(xx)) +c)
(2)
25

---

Published in Transactions on Machine Learning Research (07/2023)
Table 2: MMD between aug and gen the in-the-wild testbed. The aug functions are ordered by the average
distance. Geometric aug functions (in blue) exhibit the smallest distances in general, consistent with Fig,
Augment
MNI.
Fash.
CIF.
SVHN
Avg.
Rotate
1.831
1.855
0.267
0.186
1.035
CP-scar
1.653
2.005
0.255
0.259
1.043
CutPaste
1.697
2.100
0.258
0.203
1.065
GROM
1.754
1.774
0.423
0.446
1.099
Flip
1.906
2.082
0.257
0.208
1.113
Color
1.844
2.034
0.336
0.295
1.127
Invert
2.221
2.116
0.257
0.220
1.204
Crop
2.143
1.935
0.348
0.424
1.213
CutOut
1.723
2.066
0.503
0.629
1.230
Blur
2.182
2.372
0.303
0.221
1.270
SimCLR
2.074
1.941
0.498
0.582
1.274
Mask
2.136
2.824
0.585
0.776
1.580
Noise
4.214
3.170
0.427
0.627
2.110
Identity
2.035
2.402
0.265
0.212
1.228
We adopt ResNet50 (He al. 2016) pretrained on ImageNet as Φ, which works as a general encoder function that is independent of detector model. The hyperparameters are set to the default values in the scikit-learn implementation: 7 h-1,,
and d = 3, where h is the size of embeddings from [
Table 2 supports Obs with respect to the MMD between aug functions and the test anomalies. Geometric functions including Rotate, GEOM, and Flip show the smallest distances in general, representing that they are more aligned with the anomalies of different semantic classes than other aug functions are. One difference between TableDane and Fig. is that SimCLR, which shows large MMD on average, works better than the local aug functions (in red) Fig. 5b This is because the large flexibility "cocktail" augmentation induced by the SimCLR is effective for DeepSAD, which learns hypersphere that separates normal data from pseudo anomalies, while DAE aims to learn the exact mappings from pseudo anomalies to normal data.
Shttpss/ learn hew.ottottrrrrrrrraaaaaaaaaa
26


Reviews:
[{"review_id": "BLMKp1Dltq", "reviewer": "Reviewer_SLGE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## strengths\n1. thorough and deep treatment of an overlooked issue: the impact of the choice of augmentation on the ssl ad performance\n1. excellent exposition and contextualization (abe, steinwart, theiler references in section 1) of a complicated concepts (controlled testbed, curse of obviousness -> see watts reference in footnote 2)\n1. thorough quantitative and qualitative experiments\n1. competent, structured and reader-friendly presentation of results (with finding and observation environments in section 5)\n\n## weaknesses\n1. given the richness in concepts, experiments and results of the manuscript as well as the ambition of the authors to consolidate/sort the field, a visual abstract at the beginning of the paper would be very helpful. it would provide others with a blueprint on the experimental design (i.e. the testbed) as well as an illustration of the relationship between [gen, in-the-wild, aug]X[datasets used in testbed (e.g. svhn, mnist, etc.)]X[ads used]X[main results]\n1. as a paper that aims to connect the dots on things ssl, unsupervised learning and ad, the discussion could be enriched further with concepts and findings from existing work work", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "the authors of this article delve into the intricate role of data augmentation in the realm of self-supervised learning (ssl) applied to anomaly detection (ad), especially when handling image data. this technology offers a cost-effective solution for generating supervisory signals, making it a key tool for tasks where anomalies are scant or even non-existent. \n\nalthough recent studies have acknowledged the substantial effect of the type of augmentation on accuracy, the authors strive to expand this understanding by putting image-based ssl for ad under more extensive scrutiny. their research, carried out on three different detection models and across 420 anomaly detection tasks, presents exhaustive numerical and visual proofs. \n\nthe crux of their findings reveals the success of ssl-based ad heavily hinges on the alignment of the data augmentation and anomaly-generating mechanism. when this alignment is absent, ssl might even hamper accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## conceptual\n1. p.6 under \"evaluation\" it says \"aug and gen are perfectly aligned if they are the same function, and still highly aligned if they are in the same family, such as aug:=Rotate and gen:=Flip – both of which are geometric augmentations.\" would it not make more sense to determine \"families\" based on the representation of the (augmented) data in the feature space of the trained ad as this is the notion of similarity that is ultimately of interest (i.e. similarity of the (augmented) data as represented/viewed by the model of interest). i do not expect the authors to adjust their entire testbed as there is value and insights already there. but im curious about a discussion of this conceptual nuance.\n1. similarly, in-the-wild evaluation with mmd on resnet50 features presents a chicken-egg problem. alignment between aug data and normal data is conditioned on resnet50 feature space. which means downstream conclusions based on this alignment score are conditioned on resnet50 feature space. how does this influence the observations in the experiments? how could alignment be more objectively determined? for example, could not the feature space of the ad model in question be used for mmd calculation?\n1. a visual abstract (see details under weaknesses)\n\n## related work\n1. choice of optimal augmentation in ssl there exists plenty of work, a good reference to discuss is the infomin paper https://proceedings.neurips.cc/paper/2020/hash/4c2e5eaae9152079b9e95845750bb9ab-Abstract.html\n1. the impact of \"semantic\" alignment between datasets and downstream model performance has also been investigated for example in semi-supervised learning for example https://www.computer.org/csdl/journal/ai/2023/02/09762063/1CMrqSyVYTC\n1. generative models have also been thoroughly investigated, leading to surprising observations where supposedly ood data has higher likelihood under the generative model than the training data itself (see eric nalisnick's paper https://openreview.net/forum?id=H1xwNhCcYm figure 2b). this has also been treated in more generality as relating to the counterintuitive behavior of scoring functions for generative models when it comes to semantic expectations of humans (see for example lucas theis' great work https://arxiv.org/abs/1511.01844)\n## typos, notation, language, presentation\n1. provide tables from ding et al mentioned at bottom of p.2 in appendices for easier reference\n1. in sentence \"whereas use ImageNet-22K as OE data (superset of ImageNet-1K) to evaluate on ImageNet\" on p.3 missing subject\n1. incomplete mnist citation -> not to dunk on yann lecun and gang but: the mnist dataset was collected, curated and prepared by wilson, wilkinson, garris and colleagues*. yann and colleagues popularized the dataset later.\n1. wording in finding 1 \"(ii) faug impairs f if the alignment between aug and gen is poor.\" seems off. shouldnt it read \"(ii) aug impairs f if the alignment between aug and gen is poor.\" or \"(ii) f surpasses faug if the alignment between aug and gen is poor.\"\n1. p.9 observation 5, notation \"DAEfaug\" looks a bit odd\n\n\\*\nCL Wilson and MD Garris. Handprinted character database. technical report special database 1. Technical\nreport, National Institute of Standards and Technology, 1990. 2\n\nMD Garris and RA Wilkinson. Handwritten segmented characters database. technical report special database 3.\nTechnical report, National Institute of Standards and Technology, 1992.\n\nMichael Garris. Design, collection, and analysis of handwriting sample image databases. (31), 1994-08-10 1994.\nURL https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=906483.\n\n## criticality of changes\nid like to give authors opportunity to respond to my questions to clear up misunderstandings. overall, the visual abstract would be strongest addition in my point of view. discussing the conceptual concerns and related work would further round out the manuscript.", "broader_impact_concerns": "none as of this time", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jQazqdhXVG", "reviewer": "Reviewer_hH9B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\n**Comprehensive Study**: The study conducted a large number of experiments across a variety of tasks and models, leading to robust and reliable insights about the impact of data augmentation on self-supervised anomaly detection (SSAD).\n\n**In-Depth Analysis**: The study goes beyond simply reporting performance measures. It provides an in-depth analysis of the reasons behind the success or failure of SSAD, especially highlighting the critical role of data augmentation.\n\n**Essential Insights**: The study underlines the critical importance of alignment between pseudo and true anomalies and shows how poor alignment can harm performance or lead to biased error distribution.\n\n## Weaknesses:\n\n**Hyperparameter Selection**: While the study highlights the critical role of data augmentation in SSL for AD, it doesn't provide clear guidance on how to select these hyperparameters effectively, which could be a challenging task, especially in unsupervised learning settings.\n\n**Limited to Image Data**: The study primarily focuses on image-based SSAD. While the findings might be generalizable to other types of data, explicit verification is necessary. It might not provide much guidance for researchers working with non-image data.\n\n**No New Model**: The research doesn't propose any new model or technique to improve upon existing SSAD methods. Instead, it emphasizes the importance of considering data augmentation as a hyperparameter, which might already be a known factor for researchers in the field.\n\n**No Clear Solution**: The paper highlights issues with hyperparameter selection in unsupervised anomaly detection but doesn't provide a clear solution for resolving them. This might leave readers unsure about how to apply the insights from the study in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper conducts a comprehensive examination of the impact of data augmentation in self-supervised learning (SSL)-based anomaly detection (SSAD) on image data. It found that the alignment between the data augmentation mechanism and the anomaly-generating mechanism is critical for the success of SSAD. With poor alignment, SSL may even degrade accuracy. The paper highlights that data augmentation remains a vital hyperparameter for SSAD, necessitating careful selection to unlock its full potential. It presents a meticulous study, employing 3 different types of SSAD models across 420 different anomaly detection tasks. It also cautions that unthoughtful implementation of SSL might lead to biased detection outcomes across anomaly types. This research marks the first meta-analysis study on SSAD and emphasizes the necessity for mindful implementation of SSL for accurate and fair anomaly detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper could be improved in a couple of ways:\n\n1. **Dataset Size**: All the datasets used in the study are relatively small. To draw more general conclusions, it would be beneficial to include a larger dataset, such as ImageNet.\n\n2. **Problem Solving**: The observations made in the paper are sound. However, it would enhance the value of the study if the authors could discuss possible solutions to the identified problems and propose clear, actionable strategies.\n", "broader_impact_concerns": "The paper presents potential broader impact concerns:\n\n1. **Misuse**: The developed anomaly detection technique could be used maliciously, such as in invading privacy or discrimination.\n   \n2. **Bias**: If the training data is biased, the anomalies detected could lead to discriminatory outcomes.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XaWI8IqimU", "reviewer": "Reviewer_aT5C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The research problem is well motivated with a significant literature background, and empirically supported by those clear figures and clarifications.\n2. The empirical study with 3 different types of SSAD models across 420 different AD tasks provides an extensive analysis of the role of data augmentations.\n3. The details of the experimental setup and evaluations are clearly presented.\n\nWeakness:\n1. Although there are some interesting observations from those carefully designed experiments across so many experimental trials, would it be possible that we can conduct some theoretical analysis on the role of data augmentations to explain the phenomenon and guide future research in this direction? or provide some discussion on the potential theoretical tools for studying the role of augmentation on the SSL AD?\n2. The observations may lack a clear outline and summarization to draw an in-depth conclusion for a better understanding of the major significance of the comprehensive study for SSL AD. Maybe it can be reorganized to provide a single section to clearly present the overall analytical results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the role of data augmentations in self-supervised learning-based (SSL) anomaly detection (AD) through extensive experiments and analysis. The major contributions of this research work, in my view, can be summarized as follows: 1. the comprehensive results of image-based SSL AD that can draw insights for understanding the effect of data augmentations; 2. bring clarity to the field of SSL AD using the alignment between pseudo and true anomalies. Overall this work presents the empirical exploration under a large lens.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall I think this research work provides a comprehensive study for data augmentations in SSL AD, and I think the authors may consider adjusting the submission with the following points:\n1. summarize all the observations for a clear presentation and provide the corresponding discussion if there is some relationship among all those empirical results. \n2. discuss more about future suggestions after presenting the major insights of the comprehensive study, which may help to enhance the significance of this research study.", "broader_impact_concerns": "I think there is no related ethical concerns about this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper offers a comprehensive study on data augmentation in anomaly detection, and the authors have addressed various suggestions put forth by the reviewers. These include the inclusion of a visual abstract, expansion of the related work section, and the provision of additional results. While one reviewer expressed concern about the lack of a definitive resolution to the observed distribution shift, it is important to note that this paper's primary contribution lies in its systematic empirical study. The formulation of a concrete algorithmic solution can be considered as potential future work in this area.
Decision: Accept (poster)