Rotation Has Two Sides: Evaluating Data Augmentation for Deep One-class Classification
Authors: Guodong Wang
Yunhong Wang
Xiuguo Bao
Di Huang
Venue: ICLR 2024 spotlight (2024)
Keywords: self-supervised learning
deep one-class cilassification
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We observe a robust relationship between the accuracy of rotation prediction and one-class classification (OCC), with a novel method proposed for estimating the transformation distribution within the dataset, thus enhancing OCC.

Abstract: One-class classification (OCC) involves predicting whether a new data is normal or anomalous based solely on the data from a single class during training. Various attempts have been made to learn suitable representations for OCC within a self-supervised framework. Notably, discriminative methods that use geometric visual transformations, such as rotation, to generate pseudo-anomaly samples have exhibited impressive detection performance. Although rotation is commonly viewed as a distribution-shifting transformation and is widely used in the literature, the cause of its effectiveness remains a mystery. In this study, we are the first to make a surprising observation: there exists a strong linear relationship (Pearson's Correlation, $r > 0.9$) between the accuracy of rotation prediction and the performance of OCC. This suggests that a classifier that effectively distinguishes different rotations is more likely to excel in OCC, and vice versa. The root cause of this phenomenon can be attributed to the transformation bias in the dataset, where representations learned from transformations already present in the dataset tend to be less effective, making it essential to accurately estimate the transformation distribution before utilizing pretext tasks involving these transformations for reliable self-supervised representation learning. To the end, we propose a novel two-stage method to estimate the transformation distribution within the dataset. In the first stage, we learn general representations through standard contrastive pre-training. In the second stage, we select potentially semantics-preserving samples from the entire augmented dataset, which includes all rotations, by employing density matching with the provided reference distribution. By sorting samples based on semantics-preserving versus shifting transformations, we achieve improved performance on OCC benchmarks.

Published as a conference paper at ICLR 2024
ROTATION HAS TWO SIDES: EVALUATING DATA AUG- MENTATION FOR DEEP ONE-CLASS CLASSIFICATION
Guodong Wang .2, Yunhong , Xiuguo Bao Di Huang!.2-
State Key Laboratory of Software Development Environment, Beihang University, Beijing, China Schooo of Computer Science and Engineering, Beihang University, Beijing. China Natl.. Comp. Net. Emer. Resp. Tech. Team/Coord. Ctr. of China, Beijing, China (wanggd, yhwang, dhuang) 0buaa. edu. cn, {baoxiuguo 81388113000 com com
ABSTRACT
One-class classification (OCC) involves predicting whether a new data is normal or anomalous based solely on the data from single class during training. Var- ious attempts have been made to learn suitable representations for OCC within self-supervised framework. Notably, discriminative methods that use geomet- ric visual transformations, such as rotation, to generate -anumll samples have exhibited impressive detection performance. Although rotation is commonly viewed as distribution-shifttng transformation and is widely used in the litera- ture, the cause of its effectiveness remains mystery. In this study, we are the first to make surprising observation: there exists strong linear relationship (Pear- son's ' Correlation, r > 0.9) between the accuracy of rotation prediction and the performance of OCC. This suggests that classifier that effectively distinguishes different rotations is more likely to excel in OCC, and vice versa. The root cause of this phenomenon can be attributed to the transformation bias in the dataset, where representations learned from transformations already present in the dataset tend be less effective, making it essential accurately estimate the transforma- tion distribution before utilizing pretext tasks involving these transformations for reliable self-supervised representation learning. To the end, we propose a novel two-stage method to estimate the transformation distribution within the dataset. In the first stage, we learn general representations through standard contrastive pre- training. In the second stage, select potentially semantics- prreeeriss samples from the entire augmented dataset, which includes all rotations, by employing density matching with the provided reference distribution. By sorting samples based on semantics -rentticc--reeriiing versus shifting transformations, we achieve im- proved performance OCC benchmarks.
INTRODUCTION
One-class classification (OCC) involves determining whether a test sample adheres to the same distribution the training set, with access only to in-domain data. It represents classic and funda- mental challenge essential for ensuring the reliable deployment of machine learning systems across a broad spectrum of real-world applications. particularly in safety-critical domains like manufacturing
defect detection (Bergmann et al. 2020 2019) and medical diagnosis (Schlegl et al. 2017).
Generative models (Sabokrou
2018 Zaheer al.) 2020 Perera al. 2019) strive to model the training distribution by assigning high probabilities to training data. However, they often assign unexpectedly high likelihoods to samples that deviate significantly from the training distribution. In contrast, discriminative models encode normality by defining the boundary between the training distribution and out-of-distribution (OOD) samples. In cases where real-world outliers are lack- ing, one typical solution is to generate negative samples by applying geometric transformations, such as rotation, the training samples. Contrasting with these transformed samples helps with distinguishing in-domain data from real-world outliers, with these transformations considered as distribution-ssiittng augmentations (Tack et al. 2020 Sohn et al. 202T) to simulate outliers.
"Corresponding author.

---

Published as a conference paper at ICLR 2024
horse c2r
OCC performance (AUROC
OCC perfom performence (AUROC %0
ship
ruuk
dog
bird
door
13311
10:30
01.21
plane
do3 12-100
Bog 24.181
cat
13.04)
teack (1/10)
80
90
100
05
70
75
80
05
90
95
200
Rotation prodiction accuracy (%)
Rotation prediction accuracy 196)
Figure I: OCC (AUROC) vs. accuracy of rotation prediction on one-class CIFAR-10. Figure (a) shows intra-class correlation evaluated every epochs until convergence and (b) displays inter-class correlation for the converged models. Numbers in the bracket denote the Pearson's 's Correlation r). In both intra-class and inter-class experiment, a strong linear correlation (r > 0.9) is evident.
Based on this assumption, a plethora of literature has leveraged simulated outliers, alheit with varia- tions in how they are utilized in either pretext tasks or contrastive learning. For instance, (Hendrycks) al. 2019 Golan & EI- Yaniv 2018: Bergmann & Hoshen 2020) train a classifier to identify the spe- cific transformations applied to the original samples. On the other hand, (Sohn et al., 202 Tack jtt al. 2020) employ instance discrimination, a special case of contrastive learning, between the orig- inal samples and those with distribution-shiftin augmentations. This approach is believed to reduce the uniformity of learned representations (Sohn et al. 202I) and promote a more compact inlier distribution (Wang al. 2023).
For a better understanding of relationship between rotation prediction and OCC, we first set up a toy experiment by training classifier predict the degree of rotation applied to input images (ranging from 0%, 90", 180", to 270"). Figure [ visually represents the connection between the accuracy of rotation prediction and the performance of OCC from both intra-class and inter-class perspectives. All experiments are conducted on the one-class CIFAR- 10 benchmark, where images from one class were designated as inliers and those from the remaining classes as outliers. Interestingly, in the intra- class experiment, is observed that both OCC and rotation prediction accuracies are increasing over the training process, exhibiting a strong linear relationship for each class. This same trend holds true in the inter-class experiment. A similar phenomenon is also noted in unsupervised accuracy estimation (Deng et al.] 202I) which utilizes linear regression to estimate classifier performance from the accuracy of rotation prediction. Compared to the work training semantic classification and rotation prediction in a multi-task way, the role of rotation prediction is even prominent in OCC. OCC relies solely the design of the pretext task to capture normal patterns, with rotation prediction proving to be the most effective. The significant dependence on rotation prediction in OCC motivates to delve into investigating how rotation impacts OCC.
While rotation has been a widely used technique in the literature for OCC (Wang et al.) 2023 Tack et al.) 2020 Sohn al. 2022 Hendrycks al., 2019 Golan El- El-yaniv. 2018), there has been limited discussion regarding the circumstances under which rotation benefits OCC due to the lack of detailed analysis on the relationship between rotation and input images. Previous toy experiment (Figure T has demonstrated that different classes derive varying benefits from using rotation prediction as pretext task for OCC. We posit that rotation may not necessarily induce the semantic shift in the original training distribution to the extent previously assumed for different classes. other words, some transformed samples may still pertain the same underlying semantic concept. For instance, certain images may lack deterministic orientation may deviate from the representative orientations found in the dataset, termed aggnoin- images (RAI). Oppositely, images with a deterministic orientation division, rotation generally produces two different results, i.e., rotated RAI are semantics- preserving images while rotated non-RAIs belong to semantics-shifting
non-rotation-agnosii images (non-RAD). Based on the
In the case of RAI, it becomes practically impossible for a classifier to distinguish the original sample from its rotated versions. When such ambiguous images with incorrect labels are incorporated
2

---

Published as a conference paper at ICLR 2024
into training, they inevitably misguide the training process. Similar findings have been reported in prior research (Pal et al. 2020), which introduces the Visual Transformation for Self Supervisevii (VTSS) hypothesis. This hypothesis suggests that if the pretext task involves transformations already present in the dataset, the learned representations will be of reduced utility. This naturally raises the question: How
identify the transformations that are already inherent in the dataset?
Unfortunately, there has been a scarcity of research endeavors aiming at addressing this question. While previous study (Mohseni et al., 2022) can identify set of shifting transformations from pool of available options at dataset level, advanced methods provide instance-wise predictions by either leveraging a human-inspecced reference subset (Yang et al., 202T) or adopting the early-stop strategy (Miyai et al.) 2023). However, both methods depend on human to carefully tune the hyper- parameters, limiting their wide applications.
In this work, we present a novel two-stage framework that addresses the problem from the perspec- tive of distribution matching. In the first stage, we train a model to learn general representations via contrastive pre-training (Chen et al.) (2200) and extract representations from all the training images as reference. the second stage, we learn to sample semantics-preserving images from the entire augmented dataset, which include all transformations, guided by learnable probability estimator. The goal is to match the distributions of the sampled set and the reference set. During testing, the learned probability estimator assigns high confidence logits to semantics- prreeerriigg images and vice versa. This framework allows us to estimate the transformation distribution and effectively distinguish between semantics-preserving and semantics-shifting images.
Our contributions are summarized as follows:
We establish a connection between rotation prediction and one-class classification, reveal- ing a surprisingly strong linear relationship.
We introduce novel two-stage unsupervised framework that estimates the transformation distribution within a dataset and assesses semantic shifting at the instance level.
By sorting samples based semantis--reesrringg versus shifting transformations, hance performance on visual one-class classification benchmarks.
2 RELATED WORK
One-class classification (OCC) entails training classifier differentiate anomalous data, which is not accessible during training, from the normal data available in the training set. Unlike anomaly detection (AD), which considers both normal and (optionally) anomalous instances during training, OCC is restricted to only normal instances in its training phase, making it special case of AD. Generative models (Sabokrou et al., 2018: Zaheer al., 2020 Perera al.. 2019). trained to model the given distribution, identify test samples located in low-density regions outliers. However, they often struggle with the curse dimensionality (Kirichenko et al. 2020). We recommend readers refer to a comprehensive survey OCC for further details (Perera al. 2021). In the wake of the success of self- supervisedd representation learning achieved through methods such as creating pretext tasks (Noroozi & Favaro 2016 Doersch et al. 2015 /Zhng al. 2016 Gidaris et al. 2018) or employing contrastive learning (He et al.) 2020) Chen et al. (020). discriminative models offer an alternative approach that circumvents the complex process of density estimation and leads to the improved OCC performance. For instance, (Hendrycks et al. 2019 Golan & El-Yaniv 2018 Bergman & Hoshen| 2020) train classifier recognize the transtormations applied the original samples. They rely on this surrogate classifier (e.g., maximum softmax probability as the scoring function) for OCC, under the assumption that outliers cannot be perfectly predicted on rotation angles, as they are not encountered during training. (Tack et al.] 2020; Sohn et al. 2021 Wang jet al.] (202)) have gone further, achieving enhanced results by conducting instance discrimination with advanced anomaly scoring functions. Despite the favorable performance gained with the help of self-labeled generated outliers, these methods manually design discriminative training objectives involving specific transformations.
Analysis on visual transformations has primarily focused on the multi-class supervised sce- nario (Benton et al. 2020; Cubuk et al. 2019 Lim al. 2019; Mahan et al.) 202 Chatzipantazis ctt al. (202)) where class boundaries are explicitly defined by manually annotated labels, with limited attention in the challenging one-class classification. The Visual Transformation for Self-Supervision
3

---

Published as a conference paper at ICLR 2024
xy-shullle
Fixed
Leamable
Ä
A
C
B
B
B
D
C
D
C
C
b
goel
training data
C
C
B
B
B
pooks
fo
Ä
"B
gool
Stage I: Contrastive pre-training
candidate pool
sample
training data
Gumbel softinas
KMD
rotate
fo
1:5
ridatinn regle:
sampled reference
features
(1) Data preparation
(2) Sampling
(3) D Density matching
Stage II: Instance-aware transformation distribution estimation
Figure 2: Overview of two-stage transformation distribution estimation framework. The first stage aims at learning general image representations through contrastive pre-training and the sec- ond stage identifies potentially semantics- preetiiriigee images by aligning the reference set and the sampled set from from a universal set that encompasses all possible transformations. rry-shuffle is proposed for unbiased sampling by reorganizing candidate pools.
(VTSS) (Pal et al. 2020) is the first to raise concerns about the use of potentially confticting trans- formations pretext tasks. In the self-supervised zone, methods addressing transformation conflict relatively scarce. (Tack et al.) (222)) select shifting transformations by measuring the out-of- distribution (OOD) quality. as assessed by area under the receiver operating characteristic curve (AUROC), between in-distribution data and transformed samples using vanilla SimCLR (Chen et al. 2020). Meanwhile, (Mohseni et al. 2022) employ Bayesian optimization (Akiba et al.) 2019) to search for effective shifting transformations from pool of available options through trial-and- process, aiming to minimize
елrог process. Each trial entails a time-consuming training-validatin classification loss on the validation set. However, both of two approaches cannot identify trans- formation conflicts for each instance within the dataset. A subsequent effort (Yang et al. 202T) introduces instance- wise predictions by utilizing generative adversarial network (GAN) auto- matically discover the transformations present in the input dataset. While this method proves effec- tive for straightforward datasets like MNIST, it heavily relies on a human-inspected reference subset, which is assumed to include the most representative and frequently occurring data guide the learn- ing of transformation parameter distribution. Consequently. does not scale well to more complex scenarios due to the unavailability of such perfect reference subset. An alternative straightforward approach, as proposed by (Miyai et al. 2023 Feng et al. 2019). is to train a rotation predictor be- fore it reaches the point of over-fitting and rely the classifier distinguish between RAIs and non-RAIs. However, this early-stop strategy relies on an additional validation set to approximate the epoch just before over-fitting. It necessitates splitting the dataset into multiple consecutive folds and following the standard train/validation split, which can be time-consuming. In contrast, this work focuses on estimating the transformation distribution in an unsupervised manner and determining the likelihood of transformed sample belonging to the original distribution, addressing these chal- lenges from the perspective of distribution matching.
TRANSFORMATION DISTRIBUTION ESTIMATION
Existing OCC methods often rely on manually designed transformations to create pretext tasks or perform contrastive learning. These methods assume that these transformations, which are not present the dataset, suitably shift the original data distribution. Distinguishing between in-domain samples and transformed ones assigns a model with the ability to detect real outliers. However, empirical observations suggest that the choice of effective data transformations depends on the spe- cific in-domain datasets themselves. Moreover, effective the transformations are already present in the dataset, a phenomenon line with prior work (Pal al. 2020). To address these issues and discover the transformations that already
find that the learned representations become less
4

---

Published as a conference paper at ICLR 2024
present in the dataset, we propose a staged learning- based framework. This framework enables us to estimate the distribution of transformations and effectively differentiate between semantics- preserving and semanties- shittitis images. Without loss of generality, we focus examining the role of rotation in our main experimental results.
3.1 STAGE I: SELF-SUPERVISED REPRESENTATION LEARNING
In line with the recent successes of self supervisd learning for one-class classification (OCC) Tack et al. 20200; Sohn et al. (202T); Wang & Isola (2020), we initially train a model denoted as Iø o fo. which consists of a feature extractor Ja and a projection head Iú. where & and 0 denote the corre- sponding sets of network parameters, using a contrastive pre-training approach (Chen et al. 2020). During the pre-training, positive pairs are defined as two augmented views of the same original image, while negative pairs consist of the augmented versions of all other images. The most- used augmentations to generate positive views for visual representation learning involve random resized cropping, random horizontal flipping, and color jittering, all of which are assumed to be semantics-prestt presservg (Chen al. 2020, He al., 2020). Once the model converges, we use the fixed feature extractor fe to extract features from all training images, creating a reference set Dref == {ff(x:)) ) (N is the dataset size) to supervise the subsequent density matching stage. Notably, different from the existing methods (Tack el al.) 20200, Sohn et al., 2021 Wang et al. 2023) that consider rotations as the distribution-shine shifting transformation and push away the original images from their rotated versions, we exclude rotation from our pre-traing vee-training process to prevent the rotation itself from becoming point of examination as positive or negative augmentation. This strategy ensures distinct representations for images and their rotated counterparts.
3.2 STAGE II: TRANNSFORAATION DISTRIBUTION ESTIMATION
The second stage comprises three steps: data preparation, sampling, and density matching.
Data preparation. The core idea behind estimating the transformation distribution lies in matching density between the original training set and the set of transformed images encompassing all rotation angles, Following the common practice of rotation prediction (Hendrycks et al., 2019 Tack et al.) 2020 we consider four-way classification of rotations, namely {0", 90"; 180", 270% J. and it is easily extended to more classes or regression task. Formally, given batch of M images C := {z; (e for each F,, we rotate it by four angles to create set of rotated images that covers all rotations, denoted as 'D == [r)] E CAA E {0", (0"900"; 180%, ,,20")} where R2 represents rotation transformation with angle r.
Sampling. The next question is how to accurately identify images within D that preserve semantics. On top of the feature extractor fe. we introduce a learnable linear layer h. to estimate the confidence score h, fo(x), sxx) for short, guiding the sampling process. A higher confidence score indicates a higher likelihood of a sample belonging to the input distribution. We first partition D into a set of P : } M where each p, { {Rg (I:)). Ryo (z.,) Ris0 (I;). R270 (æx)) acts candidate pool and contains four rotated images with different rotation angles. We use a "competing- to-win" strategy to select potentially semantics-pren preserving images from each pt- Specifically, we employ a differentiable sampler that selects images based on their confidence scores 8, using a reparameterization technique known as Gumbel softmax (Jang al., 2016). Note that we choose only one sample from each p;, resulting in a total of M samples constituting the sampled test set Dap that will be matched with the reference set Dref- Since both stages use the same set of training images, there is a risk that the original sample, i.e., 2; rotated by 0", always receives the highest confidence score. To avoid this trivial solution, we introduce the ryy-shuflle strategy to ensure that each pool y contains different images with varying rotation angles and that the sampling is position- irrelevant (Figure Specifically, based on Dapp we first shuffle the images within the same rotation angles and then shuffle the images across different angles. Standard augmentations are also applied to generate diverse positive views, similar to those used in the first stage.
Density matching, Density matching is performed in the representation space extracted by the pre-trained model fo in the first stage. For training h.,, we utilize an unbiased approximation to the maximum mean discrepancy (MMD) (Gretton al. 2012) with multiple Gaussian kernels to measure the distribution discrepancy between the sampled set D and the reference set Dref in the ez-normalized representation space. The MMD loss is formally defined Equation (T)
5

---

Published as a conference paper at ICLR 2024
n
T
1
TR
MMD (Dap (MMD2(Dpp Dres) n12
k(
k k yp)
m
j=
y'=1
(1)
1
m
k(z,, 24),
71772
717
where k(r, y/ = (d(x), ((y)) kernel function and (-)) denotes the explicit mapping function. Through the matching of the two distributions, the linear layer hy tends to assign large logits to preeerring samples.
At inference, by normalizing the confidence scores (x)( among four rotations of sample one can easily infer the probability p (x)) of (x) belonging to the input distribution. We use the ensemble strategy with the same augmentations as in training for stable predictions.
(R-(wr))
((R((x))
Ere (00000000000000 180 18000,7000 (=(R,(x)))
Relation to Positive-unlabeled learning (PU-learning). PU-lcarning (Mordelet & Vert 2014 Bekker & Davis, 2020) is binary classification paradigm where the training data comprises posi- tive examples and unlabeled examples. Our task of identifying transformations present in the dataset can indeed be seen as a specialized instance of PU-learning. In our context, the original set can be considered positive class, while the transformed images belong unlabeled set. However, we di- verge from conventional PU-learning approaches, such as estimating class priors (Garg et al. 2022]) or employing two-step bootstrap method (Mordelet & Veri 2014). Instead, we adopt two-stage pipeline to identify semantics preserviireeeggeeeeee images from the perspective of distribution matching.
4 EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
We employ R8 (He al. 2016) the feature extractor So for both stages, and operate on the representations right after the global averaging layer. We follow the same augmentation configurations as (Tack al. 2020) generate positive views. We train the learnable linear layer hy for 50 epochs and use stochastic gradient descent (SGD) with momentum as the optimizer and set the learning rate to 0. To mitigate biases in the selection of samples, we exclude images from the sampled set when creating the reference subset at each iteration. Our batch size is set to 256, and the temperature 7 the Gumbel softmax is fixed at 1.0.
4.2 VISUALIZATION
We count the number of arg max,efo (00_0009,00000000 11090,220777 (x), for each class and present them as stacked histograms in Figure 3 is evident that 0 rotation has the highest frequency for all classes in CIFAR-10, 10. indicating a predominant right-side-up orientation among most images. This supports the effectiveness of rotation as distribution-shifting transformation to generate virtual outliers for OCC (Tack et al. 2020 Sohn al. 202T Wang et al. 2023). However, there are exceptions where rotation does not alter the semantics. leading to the uncertain predictions for different rotations. as these images lack a deterministic orientation. With more RAI images involved in training, we observe a linear decrease in OCC performance in Figure 3c Figure g visually demonstrates that our method accurately identifies RAIs and non-RAIs across different classes in both the training and testing sets. It illustrates our method's ability to avoid over-fitting on the training set (Miyai cet al., 2023) and to generalize well with unseen test samples. Notably, non-RAIs generally feature centered objects and zoomed-in appearance, while RAIs capture objects from distance in peculiar postures. For instance, RAIs in the plane and bird are primarily captured in low-angle shots with the sky as the background, and rotating these images does not alter their semanties. Conversely, non-Ro -AI- originally oriented hight-side-up on the ground, exhibit unusual postures when rotated, with the ground appearing upside down in the sky.

---

Published as a conference paper at ICLR 2024
2012
Nurbeer Fmagge
Mire
300
200
0.00
3.20
0:25
0.90
(a) Rot. Distribution training set. (b) Rot.
Distribution in testing (c) RAI ratios the OCC results.
Figure 3: Stacked histograms for rotation distribution in the training set (a) and testing set (b). The relationship between RAI ratios the testing and OCC performance is shown (c).
nom: RAI
RAI
Ione RAI
RAI
plane
plans
bird
bird
cal
cat
dler
deer
dog
dog
frog
frog
honse
horse
ship
ship
bruck
Figure 4: Examples of RAI and non-RAI predicted by our method. Images on the left and right part from the training and testing sets from CIFAR-10, respectively.
Furthermore, we are interested in how rotation classifier performs on RAIs/non-RAIs Intuitively, a well-trained classifier can distinguish between different rotations non-RAIs but struggles with those of RAIs. For evaluation, its accuracy on the testing set in CIFAR- 10. Figure 5 displays the rotation accuracies and OCC per- formance on RAIs and non-RAIs actoss various classes. This figure highlights two key observations: 1) the classifier achieves significantly lower accuracy on RAIS compared to non-RAIs in nearly all CIFAR-10 10 classes, indicating accurate classification of RAIs and non-RAIs by framework; 2) the performance bottleneck in OCC lies in the incorrect rotation prediction for RAIs. Therefore, re- lying solely on the classification accuracy of the classifier (Hendrycks al. 2019) for OCC cannot reliably distinguish RAIS from outliers, as both are incorrectly predicted. In such cases, using these images with the misleading labels of rotation angles will misguide the training process.
first train 4-way rotation classifier until convergence and report
4.3 OTHER TRANSFORMATIONS AND PRE- TRAINING METHODS
While our previous analysis primarily focuses on examining the rotation transformation, it is worth noting that our method is not restricted to rotation alone: it can be applied to various other types of transformations such as vertical/horizontal flip, translation, and composited transformations. Fur- thermore, our method allows for the utilization of supervised feature extractors, not limited to self- supervised ones. In this section, we evaluate these transformations on the Street View House Num- bers (SVHN) dataset (Netzer et al.) 2011), where the central digit (0 through 9) in each image is the label. This evaluation provides a better understanding how transformations shift or preserve semantics, given the clear definition of numbers. We first train the ResNet-19 18 from scratch in pervised way on the training set SVHN, excluding horizontal flip as it is examined the following stage of distribution estimation. Figure 6 visualizes the estimated distribution of the test set for four

---

Published as a conference paper at ICLR 2024
100
Rotation prediction accuracy
100
RAI
RAI
80
All
80
OCC performancce (AUROC
60
40
20
0
plane
car
cat
deer
dog
frog
horse
ship
Figure 5: We individually calculate the accuracy of rotation prediction and the OCC performance
for RAIs and non-RAIs. Both show a strong link between rotation prediction and OCC.
Vertical flip
Horizontal flip
Translatoon
Rotation +vertical flip
Figure 6: Our method is also applicable other transformations. Below each sub-figure, the orig- inal images in the test set are displayed, with x-axis tick values corresponding to the ground truth specified by the original dataset.
transformations, with stacked histograms below displaying images which the model makes in- correct predictions. It is evident that different classes exhibit distinct transformation distributions. For flip transformation, nearly half of the images are predicted as vertically flipped for number O, indicating the involvement of vertical flip, which aligns with common sense. Similarly, numbers 1, 3, and 8 involve vertical flip to some extent, while number lacks invariance to horizontal flip. For translation, we consider five directions (original, up, down, left, right) with an 8-pixel shift, demon- strating the model's s ability correctly identify off-centered digits. Our method is also applicable to composited transformations, where images are rotated followed by vertical flip. Intriguingly, our method can even spot semantic-shifting images induced by incorrect labeling!
4.4 ONE-CLASS CLASSIHICATION
Following (Golan & Yl-ianiy 2018 Sohn et al. 2021), conduct evaluations popu- lar OCC benchmarks, including CIFAR-10 (Krizhevsky et al. 2009). CIFAR-100 (20 super- classes) (Krizhevsky et al., 22009), and Cat-vs-Dog (Elson al. (007). In these benchmarks, images from one class are treated as inliers, while those from the remaining classes are considered outliers. We utilize AUROC threshold-free metric for OCC evaluation.
The preceding analysis highlights that relying solely on a classifier for rotation prediction (Gidaris) al. 2018)) sub-optimal for OCC due to the presence of noisy labels. One straightforward ap- proach is to discard these noisy samples; however, this strategy fails to take full advantage of all the training data and does not capture the comprehensive training distribution. We suggest incor- porating additional pretext tasks for representation learning in multi-task manner beyond rotation prediction. For instance, combining contrastive learning with rotation significantly enhances OCC performance (Tack et al. 2020; Wang et al. 2023). Here, we employ the two one-class classifiers. Before introducing proper modifications, we establish a simple criterion for determining whether transformed sample is semantically shifted not, based the predicted
p( (x)).. Since the original sample, i.e.. I; without any rotations, serves as trustworthy in-domain data, we take all transformed samples satisfying p( R2 (xx)) 2 p(xx) as in-domain (semantics- prentticipp)) data.
8

---

Published as a conference paper at ICLR 2024
(a) One-class CIFAR- 10.
Method
Plane
Car
Bird
Cat
Deer
Dog
Frug
Horse
Ship
Truck
Mean
Goom
(Golen EI Yanix
2018
74.7
95.7
78.1
72.4
87.8
$7.8
83.4
95.5
93,3
91.3
86.0
Hendrocks et R0
78.3
94.3
86.2sn4
80.8a0a
89,4
95.1ama
92.3a03
89.7nn
88.4
Ro44
Tans: (Hoii Vala mi 2019
904451
96-4201
85.9
81.1ans
91.3ga1
80.6
89.9
95.9am
95.0ga
92. 62m
89.8
GOAD [Bergnian
0140
75,5
a4.
$1.8
72.0ap3
85.7a0s
84.4103
82.9uss
93.9
92.9.03
89.5
85.1
iDECOD
66.52n0
18.
86.0ans
89.2gm
W8.2.4
97.8
97.2
91.2
CSI [Tack 7020
89.9
19:
13.1
86,4212
93.9gm
93.2an0
=5.1
98. 7am0
97.9
Satu
94.3
CSI mi8
90/6:m
99.1
94,4
87.4gm1
94,5
93.5gm
95.6,83
98.9
98. 98.1ga
96. 1gn0
94.8
UniCoa" [Wang st 2023
$9.4.m2
99,2
93.3
89.2
94.1
94,2
96.4:aa
98. 98.6
m7.7
96.3
94.8
UniCon
90.0cma
93
90.4a02
95.
95.8am
96.8-a1
98.7
97.6
96. 96.8
95.4
(h) CIFAR-100 (20 super-classes).
(c) Cat-vs-Dog.
Method
AUROC
Method
Cat
Dog
Mean
GEOM (Golan & EI- Yaniv
2018
78.7
RotNet (Golan & El- El-Yaniv. 2018
86.1
86.6
86.4
Rot (Hendrycks et al. 2019
79.7
Denoising Sohn et al. X202
41.3
60.6
51.0
Rot+Trans (Hendrycks et al.
2019
79.8
Contrastive Sohn
202
89.7
85.7
87.7
GOAD (Bergman & Hoshen
2020
74.5
DROC Sohhn al. 2021
91.7
87.5
89.6
CSI (Tack et al. 2020
89.6
CSI' TTac et al 2020
88,9
88.6
88.8
CSI + ours
90.5
CSI + ours
89,5
89.3
89.4
UniCon" (Wagg et al. 2023 UniCon ours
92.0
UniCon" (Wang al. 2023
92.1
89.2
90.7
92.6
UniCon
92.5
90.0
91.3
Table 1: AUROC scores on one-class (a) CIFAR-10, 10, (b) CIFAR-100 (20 super-classes) and (c) Cat- vs-Dog. For 10, we report the means and standard deviations of AUROC averaged over three trials. t and $ denote the values from DROC (Sohn et al., 202T) and CSI (Tack et al.) 2020). respectively. denotes the reproduced results by ours.
Powerful anomaly detectors like CSI Tack et al. 2020) and UniCon (Wang et al., 2023) uti- lize rotation transformations to generate pseudo outliers and learn representations in contrastive learning framework. However, our analysis demonstrates that rotation does not necessarily alter semantics. Incorporating these noisy images eeemantics-preservve training disrupts model training, leading to sub-optimal OCC results. By identifying semantic- preserving/shifting images, we modify the training objectives of CSI (Tack et al., 2020) and Uni- Con (Wang et al., 2023) to handle the two types of images separately. For CSI, instead of predict- ing the label out of four rotation angles for each image, we abstain from making predictions for semantic- semantic-preserving images after rotation and only pull close their distances at feature level. For semantic-shiftime images, we still predict their rotation labels and promote their separation from other instances, as done in CSI (Tack et al. 2020). For UniCon, we utilize the basic version (Wang) al., 2023) without soft aggregation and hierarchical augmentation tricks for a fair comparison. Rather than promoting the concentration of all original images and dispersion of their rotated ver- sions, we aggregate all semantic- prenttiippreervingg images and push away semantic-shifting ones. We apply the same criterion as (Tack al. 2020) and (Wang et al., 2023) compute anomaly scores. Table [ presents the results on three benchmarks. Building on powerful one-class classifiers, our approach consistently enhances performance across nearly all classes. Notably, the improvement is particularly significant when a class contains more semantic-preserving images w.r.t rotation.
but considered outliers) into
5 CONCLUSION
In this study, we initially establish connection between the widely used rotation prediction task and one-class classification, exhibiting a strong linear relationship. We emphasize the significance of identifying the transformations present in the original dataset before incorporating them into the pretext task for learning. Subsequently, we introduce a novel two-stage framework designed to automatically estimate the distribution of transformations in unsupervised manner. Leveraging the estimated distribution, we segregate the training data into two distinct sets: one comprising samples without semantics shifting and the other with such shifting. We then address these sets separately for visual representation learning. We believe that our work sheds light on the critical role of understanding the transformations present in the dataset for improving OCC.

---

Published as a conference paper at ICLR 2024
Acknowledgments. This work is partly supported by the National Key R&D Program of China (2022DD0161902), the National Natural Science Foundation of China (U20B2069 and 62022011), the Research Program of State Key Laboratory of Software Development Environment (SKLSDE- 1023 and the Fundamental Research Funds for the Central Universities.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In ACM SIGKDD, 2019.
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109:719-760, 2020.
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Wilson. Learning invariances in neural networks from training data. In NeurIPS, 2020.
Liron Bergman and Yedid Hoshen. Classification-bass anomaly detection for general data. In ICLR, 2020.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehensive real-world dataset unsupervised anomaly detection. In CVPR, 2019.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. CVPR, 2020.
Evangelos Chatzipantazis, Stefanos Pertigkiozoglou, Edgar Dobriban, and Kostas Daniilidis.
Learning augmentation distributions using transformed risk minimization. arXiv preprint arXiv:2111.08190, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment: Learning augmentation strategies from data. In CVPR, 2019.
Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about classi- fier accuracy under varying testing environments? In /CMI, 2021.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Jeremy Elson, John R Douceur, Jon Howell, and Jared Saul. Asirra: a captcha that exploits interest- aligned manual image categorization. In ACM CCS, 2007.
Zeyu Feng, Chang Xu, and Dacheng Tao. Self-supervised representation learning by rotation feature decoupling. In CVPR, 2019.
Saurabh Garg, Yifan Wu, Alexander J Smola, Sivaraman Balakrishnan, and Zachary Lipton. Mixture proportion estimation and pu learning: a modern approach. In NeurlPS, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Izhak Golan and Ran El- El-Yaniv. Deep anomaly detection using geometric transformations. In NeurlPS, 2018.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. JMLR, 13(1):723-773, 2012.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. In NewrlPS, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztioo with gumbel-softmax, arXiv preprint arXiv: 1.0/ 144, 2016.
Ramneet Kaur, Susmit Jha, Anirhan Roy, Sangdon Park, Edgar Dobrihan, Oleg Sokolsky, and Insup Lee. idecode: In-distribution equivariance for conformal out-of-distribution detection. In AAAI, 2022.
Polina Kirichenko, Pavel Izmailov, and Andrew Wilson. Why normalizing flows fail to detect out-of-distriuttion data. In NeurIPS, 2020.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In CVPR, 2021.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In NewrlPS, 2019.
Scott Mahan, Henry Kvinge, and Tim Doster. Rotating spiders and reflecting dogs: class con- ditional approach to learning data augmentation distributions. arXiv preprint arXiv:2106.0400 2021.
Atsuyuki Miyai, Qing Yu, Daiki Ikaii, Go Irie, and Kiyoharu Aizawa. Rethinking rotation in self-supervised contrastive learning: Adaptive positive or negative data augmentation. In WACV, 2023.
Sina Mohseni, Arash Vahdat, and Jay Yadawa. Shifting transformation learning for out-of- distribution detection. BMVC, 2022.
Fantine Mordelet and J-P Vert. A bagging svm to learn from positive and unlabeled examples. Patlern Recognition Letters, 37:201-209, 2014.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop. 2011.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Dipan K Pal, Sreena Nallamothu, and Marios Savvides. Towards a hypothesis visual transforma- tion based self-supervision. BMVC, 2020.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In CVPR, 2019.
Pramuditha Perera, Poojan Oza, and Vishal M Patel. One-class classification: survey. arXiv preprint arXiv:2101.03066 2021.
Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially learned one-class classifier for novelty detection. In CVPR, 2018.
Thomas Schlegl, Philipp Seeböck. Sebastian M Waldstein, Ursula Schmidt-Erfutt, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In IPMI, 2017.
11

---

Published as a conference paper at ICLR 2024
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluating representations for deep one-class classification. In ICLR, 2021.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. In NeurIPS, 2020.
Guodong Wang, Yunhong Wang, Jie Qin, Dongming Zhang, Xiuguo Bao, and Di Huang. Unilat- erally aggregated contrastive learning with hierarchical augmentation for anomaly detection. In ICCV, 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICML,, 2020.
Seunghan Yang, Debasmit Das, Simyung Chang, Sungrack Yun, and Fatih Porikli. Distri- bution estimation to automate transformation policies for self-supervision. arXiv preprint arXiv:2111.12265, 2021.
Muhammad Zaigham Zaheer, Jin-ha Lee, Marcella Astrid, and Seung-lk Lee. Old is gold: Redefin- ing the adversarially learned one-class classifier training paradigm. In CVPR, 2020.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
12

---

Published as a conference paper at ICLR 2024
Supplementary Material
Here, we additionally provide:
Class-wise performance of RAI and non-RAI binary classification and one-class classifica- tion, given manually annotated labels.
Ablation studies on pre-training dataset, pre-training paradigm and network architecture.
Evaluation the multi-class Timy-ImageNet| Le & Yang (2015).
Evaluation on the real industrial MvTec-AD dataset (Bergmann et al. 2019)
A
CLASS-WISE PERFORMANCE
A.1 ACCURACY OF RAI IDENTIFICATION
For quantitative evaluation, binary labels of RAI and non-RAI for the CIFAR-I0 training set are manually annotated. The class-wise prediction recall and precision of RAI are reported in the Ta- ble showing that our method accurately identifies RAIs.
Table A: Class-wise recall and precision on the RAI identification.
Class
Plane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Mean
RAT ratios (%)
15.3
0.9
17.0
18.5
7.8
8.5
14.9
2.5
2.5
2.9
9.1
Recall
91.1
97.7
92.6
89.2
81.3
88.3
77.8
93.6
89.6
87.8
84.7
Precision
88.3
89.6
89.8
84.3
84.8
71.4
88.6
80.7
88.2
81.1
88.9
A.2 ACCURACY OCC ON RAI/NON-RAI
Based on CSI Tack al. 2020). CIFAR-10. 10. Table displays the results for the classes which include relatively high proportion of RAI samples. Our method significantly enhances OCC performance on RAI, especially for classes with larger proportion of RAI samples, such as Plane and Cat class.
present separate results for RAI and non-RAI on one-class
Table B: Class-wise AUROC scores (%) on OCC performance.
Class
Plane
Bird
Cal
Deer
Dog
Frog
Mean
RAI rabis (%)
20.2
18.6
24.4
8.1
9.7
12.9
14.74
CSI
RAI RAI
73.1
90.2
79.4
92.0
91.4
92.4
86.4
CSI Ours
84.2(+11.1)
93.1 93.1(42.9)
85, 85.3(+5.9)
92.8(+0.8)
92.5(+1.1)
92.8(+0.4)
90.1(+3.7)
CSI
non- RAI
93.2
96.5
90.8
95.2
94.9
97.4
94.7
non-RAI
93.4(40.2)
96.5(40.0)
91.9(+1.1)
95.6(10.4)
95.2(+0.3)
97.7(40.3)
95.1(+0.4)
B ABLATION STUDY
Our ablation study considers variations in pre-training datasets 1C0 vs. ImageNet-IK), pre- training paradigms (self-supervised vs. supervised) and architecture types (ViT-B/16 (Dosovitskiy et al. 202T) vs. ResNet-18) in the first stage. dablos[eeessttrrtes the architecture-agnosti nature of our method, with ResNet- 18 leading to larger improvement than ViT-B/16. This is attributed to the lack of inductive bias in VIT, which struggles to learn local relations well with small amount of data, such as CIFAR-10. The results also highlight our method's preference for the supervised training strategy, emphasizing the significance of well-clustered image representations in contrast to uniform distribution (Wang & Isola 2020) induced by self-supervised training. When pre-training the model the large-scale dataset, improved performance is further observed.
13

---

Published as a conference paper at ICLR 2024
Table C: Ablation study on pre-training dataset/paradigm and architecture in the first stage.
Method PNDA
Pre-training dataset
Pre-training paradigm
Architecture
Recall
Precision
CIFAR-10
Supervised
ResNet-18
77.1
62.0
One-class CIFAR-10
Self-supervised
ViT-B/16
82.6
83.6
CIFAR-10
Supervised
ViT-B/16
90.2
85.4
Ours
One-class CIFAR-10
Self-supervised
ResNet-18
84.7
88.9
CIFAR-10
Supervised
ResNet-18
93.6
90.3
ImageNet-IK
Supervised
ResNet-18
95.5
93.5
C MULTI-CLASS CLASSIHICATION
We additionally conduct experiments on the Tiny-ImageNet Le & Yang (2013) which has 100,000 images of 200 classes. As the two-stage pipeline, the first stage pre-trains the encoder by using SimCLR and the second stage learns rotation distribution. Likewise, we use the prediction (Equa- tion (2) to identify RAI and non-RAL. Given the partition of RAI and Non-RAI, we again pre-train the model by using SimCLR, following the practice (Miyai al,, 2023) to deal with RAIs and non- RAIs, ie., RAIs and their rotations are positive while non-RAls and their rotations are negative. For fair comparison, we keep the same hhper-paramene parameters in the pre-training, We adopt linear probing and report the top-1 classification accuracy in Table D We improve the baseline by +2.59%, which attributed to the enhanced distinction between RAI and non-RAL.
Table D: Linear evaluation ImnylImaeNNet.
RAI ratio (%)
Top-1 1 accuracy (%)
PNDA (Miyai et al. 2023)
31.3
37.17
Ours
28.9
39.76
D REAL INDUSTRIAL DATASET
Our experimental evaluation on the MvTec-AD dataset (Bergmann et al. 2019). adopting patch
representations 32x32 built upon UniCon-HA (Wang et al.) (023), demonstrates our method's superiority in Table F| compared to counterparts that incorporate the same rotation augmentation. Though our method lags behind the state-of-the-art anomaly detector, we underline the need for
tailored pretext tasks (Li et al. (2211) leveraging pre-trained models (Bergmann et al. 2020) in the context of industrial anomaly detection.
Table E: Results on the MvTec-AD dataset Bergmann et al. 2019
Method
Image
Pixel
RotNet (Hendryeks al. 2019
71.0
92.6
DROC (Sohn et al. (021))
86.5
90.2
UniCon-HA (Wang
2023
89.8
94.3
Cut-Paste (Li et al. 2021
95.2
96.0
Ours
90.6
95.2
14


Reviews:
[{"review_id": "4RCPhGJonO", "reviewer": "Reviewer_h8uB", "summary": "This paper introduces a novel method to improve performance on one-class classification problem. They observe a strong correlation between the accuracy of rotation prediction and one-class classification. They introduce a two-stage unsupervised framework that differentiates rotations that are semantic preserving (rotation-agnostic images) vs semantic shifting (non-rotation agnostic images) to enhance performance on the one-class classification benchmarks.", "questions": "- My major suggestions are summarized in Weakenesses section\n- In introduction: “While rotation has been a widely used technique in the literature for OCC…” missing citations, please add them here?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method introduced detects rotations that remain unchanged and are semantically similar to original images, ensuring their exclusion as outliers. This method demonstrates sufficient generalizability for other transformation (ex., gaussian noise), as supported by section 4.3. I feel that this contribution is of sufficient interest to the research community.\n- The results on OCC presented in the paper outperform the baselines and can be added on top on existing methods. It emphasizes the significance of identifying the transformations present in the original dataset before incorporating them into the pretext task for learning.", "weaknesses": "- Related works section of the paper is difficult to follow and seems incomplete.\n    - The subheading of **One-Class Classification** is particularly confusing as it lacks comprehensive discussion on the relevant OCC literature. The authors directly diverge to self-supervised learning methods for OCC. It fails to give complete picture of OCC for non-experts. I would suggest the authors to briefly also give an introduction to OCC and existing generative methods or point readers to more detailed survey paper [1].\n    - There exists a confusion regarding anomaly detection and OCC cited in related work, in which cases are they both considered the same?\n- Missing relevant citation: the authors seem to be missing an important citation on rotation estimation [2]. I would suggest the authors to include it for completeness of related work.\n\n[1] One-Class Classification: A Survey, arxiv 2021\n\n[2] Self-Supervised Representation Learning by Rotation Feature Decoupling, CVPR 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2b9R7WYCY", "reviewer": "Reviewer_Soam", "summary": "The study makes a surprising discovery: a strong linear relationship exists between the accuracy of rotation prediction and the performance of OCC and they show that representations learned from transformations already present tend to be less effective. To address this, the paper proposes a staged learning-based framework for one-class classification (OCC) that aims to identify semantics-preserving images. The framework consists of two stages: self-supervised representation learning and transformation distribution estimation.", "questions": "- How does the analysis across other kinds of popular transformations look like and does this approach still hold, I would encourage the authors to include talking about other transforms even if they do not seems to work well.\n- This shouldn't use in-text citations\n\n> facturing defect detection (Bergmann et al., 2020; 2019) and medical diagnosis Schlegl et al. (2017).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- I think the authors do a great job at explaining and motivating the problem they are working on.\n- The paper seems novel in its exploration of the relationship between rotation prediction and one-class classification (OCC). It highlights a surprising observation of a strong linear relationship between the performance of rotation prediction and the performance of OCC.\n- The authors back up their approach on empirical observations, highlighting the importance of effective data transformations and the potential decrease in effectiveness if transformations are already present in the dataset. This empirical foundation strengthens the credibility of their proposed solution. Their approach seems to be very well-motivated\n- The experiments though little are well-designed, valid, and exhaustive, with comparison to a range of baselines as well as some ablation studies.", "weaknesses": "- A big weakness right now is the lack of extensive empirical validation. The authors currently only perform experiments on CIFAR-10 and their experiments on other kinds of transformations are also very limited. Though the authors show interesting results for another transformation, it is immensely difficult with this set of results to comment on how well their approach could work.\n- One of the really interesting findings from this paper is about transformation bias, and representations learned from transformations already present tend to be less effective. I believe this would be well-shown by experiments across multiple datasets and multiple kinds of models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fkhiCDYr1J", "reviewer": "Reviewer_vKuc", "summary": "This paper proposes a technique that can learn the rotation distribution of images within a dataset. The approach is motivated by the authors' study of one-class classification (OCC), which involves predicting whether data belongs to a particular class seen in training or is anomalous. Specifically, the authors are investigating the seeming strong linear relationship between rotation prediction accuracy and OCC. The authors attribute this to transformation bias, where samples that are semantic-preserving vs. semantic-shifting lead to different behavior when training OCC with contrastive learning approaches.\n\nTo this end, the authors propose a two-stage approach for learning the transformation distribution: 1) perform a standard contrastive self-supervised representation learning phase, and 2) transformation distribution estimation. For 2, the authors create a dataset consisting of the original samples in {0, 90, 180, and 270} degrees and learn a differentiable sampler with Gumbel Softmax to predict images that preserve semantics (i.e., rotating it does not necessarily change the orientation at which the picture must have been taken). Finally, MMD is used to perform density matching in the representation space to identify such semantic-preserving samples.\n\nThe authors use their model to 1) visually show a good model in learning RAIs vs. non-RAIs, 2) a correlation between the amount of RAIs in training and OCC performance, and 3) that using their approach consistently adds ~1% gain to OCC.", "questions": "1. In Section 4.2: it seems that images with a prediction of 0 is deemed non-RAI and anything else is deemed to be RAI. Couldn't RAI images still be predicted with an angle of 0? How many of the RAI images are classified as angle 0 and how many images classified as angle 0 are actually RAI?\n\n2. Do you have any examples / analysis of failure modes where images like those in figure 5 are mistakenly predicted to be RAI vs. non-RAI?\n\n3. Can you clarify the rule used in section 4.4 to determine if a sample is semantically-shifted? Which r is used?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper include a clear motivation, strong analysis of the correlation between rotation prediction and OCC, and its unsupervised data-driven approach. The intuition to use a predictor in conjunction with the contrastive pretrained representations and density estimation to align the dataset makes sense for extracting out the transformation distribution exhibited within the training set. Understanding this distribution is an interesting task. The visual results of selecting RAI vs. non-RAI images are compelling. The issues with existing OCC approaches are well-analyzed and the the proposed approach provides modest but consistent improvement to existing OCC approaches.", "weaknesses": "One weakness is that there is not a quantifiable way to measure the accuracy in RAI vs. non-RAI determination. One reason for this is that RAI images may be classified as 0, making it hard to separate these images from the truly non-RAI images. There is also not a discussion / inclusion of failure modes to understand where the model may succeed vs. fail. Another weakness is that the utility of the transformation distribution may be larger but is focused on OCC, and datasets used to evaluate OCC are from CIFAR-10, which may have different characteristics than some of the anomaly detection settings where labeled data of out-of-distribution samples could be limited. I also think it is odd that OCC is a common thread / motivator of the paper but the description of the technique for OCC is not in the approach section. The claim of the paper then is a bit broad, in some parts reading as if it is most concerned about OCC and other parts about the distribution being learned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oj88jn6vBk", "reviewer": "Reviewer_Yjs1", "summary": "Targeting improvement of existing OCC, this paper makes an observation of the strong linear relationship between the rotation prediction and the performance of OCC. To the end, this paper proposes a two-stage framework where in the first stage, standard contrastive learning is used, while in the second stage, semantics-preserving samples are selected from the augmented dataset. Experiments are conducted on several anomaly detection benchmarks.", "questions": "See the weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1.\tThe contribution is clear, and the motivation sounds reasonable.\n2.\tAnalysis on the impact of rotation prediction on OCC is intuitive.\n3.\tExperiments show the effectiveness of the proposed method.", "weaknesses": "1. While the analysis is intriguing, its applicability remains limited to rotation-related datasets and methods. For instance, it may not be suitable for numerous real-world anomaly detection tasks, such as MVTec and VisA.\n\n2. The improvements, as depicted in Table 1, are somewhat modest compared to existing methods.\n\n3. An essential evaluation is missing. This paper identifies RAI images and treats them differently from the original method. However, it is unclear whether the observed improvement stems from these RAI images alone. Assessing the performance of RAI images separately might lead to a more substantial improvement.\n\n4. The paper's structure could be improved. There is significant overlap in the information presented in Figures 1, 2, 4, 5, 6, and 7. It may be advisable to move figures like 4-7 to the supplementary section.\n\n5. The rationale behind the authors' decision to use the version of UniCon without soft aggregation and hierarchical augmentation remains unclear. Since hierarchical augmentation is an integral module within UniCon, it is advisable for the authors to use the full version of UniCon as the baseline for their study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EldXopVNDM", "reviewer": "Reviewer_jwSM", "summary": "This work targets improving OCC performance by discriminating the RAIs in the training dataset, which is inspired by a surprising observation: there exists a strong linear relationship between the accuracy of rotation prediction and the performance of OCC. The proposed distribution matching-based method is interesting and proved to be effective.", "questions": "1. In Page 1 \"In cases where real-world outliers are lacking, one typical solution is to generate negative samples by applying geometric transformations, such as rotation, to the training samples.\", why are the augmented samples negative?\n2. In Sec.3.1, are other pre-trained models suitable for stage 2? Why?\n3. In Sec.3.2, how to ensure there is one sample in p_i belonging to the input domain after xy-shuffling?\n4. How to pick the RAI samples according to Eq. (2）？\n5. There is no definition of p(R_{r}(x)) before it is been used in Eq.(2). And the end of Eq.(2) should be a period.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is novel and interesting.\n2. The proposed method is promising and well-analyzed.\n3. The paper writing is good.", "weaknesses": "1. In the first stage of the proposed method, there is no reason presented for the choice of contrastive pre-training.\n2. There are no large-scale dataset evaluations, such as imagenet.\n3. There is no direct quantitative evaluation of the RAI predictions. For instance, can the authors annotate a test set with binary labels of whether or not a sample is RAI?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)