SPADE: Semi-supervised Anomaly Detection under Distribution Mismatch
Authors: Jinsung Yoon
Kihyuk Sohn
Chun-Liang Li
Sercan O Arik
Tomas Pfister
Venue: Accepted by TMLR ()

Abstract: Semi-supervised anomaly detection is a common problem, as often the datasets containing anomalies are partially labeled. We propose a canonical framework: Semi-supervised Pseudo-labeler Anomaly Detection with Ensembling (SPADE) that isn't limited by the assumption that labeled and unlabeled data come from the same distribution. Indeed, the assumption is often violated in many applications -- for example, the labeled data may contain only anomalies unlike unlabeled data, or unlabeled data may contain different types of anomalies, or labeled data may contain only `easy-to-label' samples. SPADE utilizes an ensemble of one class classifiers as the pseudo-labeler to improve the robustness of pseudo-labeling with distribution mismatch. Partial matching is proposed to automatically select the critical hyper-parameters for pseudo-labeling without validation data, which is crucial with limited labeled data. SPADE shows state-of-the-art semi-supervised anomaly detection performance across a wide range of scenarios with distribution mismatch in both tabular and image domains. In some common real-world settings such as model facing new types of unlabeled anomalies, SPADE outperforms the state-of-the-art alternatives by 5% AUC in average.

Published in Transactions on Machine Learning Research (02/2023)
SPADE: Semi-supervised Anomaly Detection under Distribution Mismatch
Jinsung Yoon, Kihyuk Sohn, Chun-Liang Li, Sercan Ö. Arik, Tomas Pfister finsungyoon, kihyuks, chunliang, soarik, tpfister @google..oooo
Google Cloud Al
Reviewed on OpenReview: https: // openreview. net/ forum? JuDp
Abstract
Semi-supervised anomaly detection is common problem, as often the datasets containing anomalies are partially labeled. We propose canonical framework: Semi-supervised Pseudo- labeler Anomaly Detection with Ensembling (SPADE) that isn't limited by the assumption that labeled and unlabeled data come from the same distribution. Indeed, the assumption often violated in many applications for example, the labeled data may contain only anomalies unlike unlabeled data, or unlabeled data may contain different types of anomalies, or labeled data may contain only 'casy-to-label" samples. SPADE utilizes an ensemble of one class classifiers as the pseudo-labeler to improve the robustness of pseudo-labeling with distribution mismatch. Partial matching is proposed to automatically select the critical prpee-parameters for psoudo-labeling without validation data, which is crucial with limited labeled data. SPADE shows state-of-the-at semi-supervised anomaly detection performance across a wide range of scenarios with distribution mismatch in both tabular and image domains. In some common real-world settings such as model facing new types of unlabeled anomalies, SPADE outperforms the theeaatf- alternatives by 5% AUC in average.
1 Introduction
Anomaly detection has numerous real-world applications, including identification of manufacturing defects, network security threats, and financial fraud (Chalapathy & Chawla, 2019; Ahmed et al., 2016; Vanerio & Casas, 2017). Anomaly detection can be considered in different settings. One is the fully-supervised setting, where the labels for all samples are available, for both normal and anomalous samples (Chawla et al., 2002; Estabrooks et al., 2004; Hwang al., 2011; Barua el al., 2012). This setting is typically addressed with specialized approaches for data imbalance, e.g. weighted loss functions or resampling methods, An important special case of this fully-supervised setting is when only labeled normal samples exist (Schölkopf et al., 1999; Tax & Duin, 2004; Ruff et al., 2018; Golan & El-Yaniv 2018; Sohn et al., 2021; Li et al., 2021), for which one class classifiers (OCCs) (e.g. with SVM (Schölkopf et al., 1999) or auto-encoder (Ruff et al., 2018)) and Isolation Forest (Liu et al., 2008) are popular approaches. Despite being widely-studied, the challenge towards the real-world use for these supervised settings is their tedious labeling requirement. At the other extreme, there is the fully unsupervised anomaly detection setting where no labeled data available (Breunig et al., 2000; Liu al., 2008; Zong al., 2018; Bergman & Hoshen, 2019; Yoon al., 2022). While the labeling costs can be entirely eliminated for this setting, the performance degradation is often significant compared to the supervised setting (Bergman & Hoshen, 2019; Zong et al., 2018), limiting its applicability for deployment.
To achieve the best of both worlds, we focus on the scmi-supervised anomaly detection sctting, aiming to achieve high performance with a limited labeling budget. In previous works on semi-supervised anomaly detection (Zhang & Zuo, 2008; Bekker Davis, 2020; Blanchard et al., 2010; Akcay et al., 2018; Görnitz et al., 2013; Ruff al., 2020), some focus on the positive-unlabelos setting (Zhang Zuo, 2008; Bokker & Davis, 2020), and others utilize one-class classifiers or adversarial training on semi-supervised learning (Görnitz et al., 2013; Akcay al., 2018). Ruff et al. (2020) treats all unlabeled data normal samples to

---

Published in Transactions on Machine Learning Research (02/2023)
Positive-Unlabaled Setting (PU)
New Types of Anomalies
Easy vs Difficult
Labeled data
Unlabeled data
Labeled data
Unlabaled data
Labeled data
Unlabeled data
samples
Known Easy' anomaly types
'New anomaly types
Figure 1: Three common real-world settings with labeled and unlabeled data coming from different distribu- tions. (Teft) Labeled data only include anomalous samples while unlabeled data have both anomalous and normal. (Middle) The anomaly is a new type (yellow boxes) which isn't in labeled data. (Right) Labeled data only have 'casy-to-label" samples while unlabeled data include 'hard-to-label samples (ycllow boxes).
construct an anomaly detector in semi-supervised settings. In addition, any somi-supervised learning method (even when they aren't developed for anomaly detection) can be adapted to the semi-supervised anomaly detection setting (Sohn et al., 2020; Chen et al., 2020a; Grill et al., 2020).
Most sumi-supervised learning methods assume that the labeled and unlabeled data come from the same distributions (Sohn et al., 2020; Chen al., 2020a; Grill et al., 2020). In other words, the subsets of the data are labeled such that sampling from the unlabeled data is randomly uniform. However, in practice, this assumption often does not hold: distribution mismatch commonly occur, with labeled and unlabeled data coming from different distributions. Some works (Kim et al., 2020) tackle this in a limited setting where only the label distributions are different (e-g., the anomalous ratio is 10% for training but 50% for testing), however, there are other more general real-world scenarios, as exemplified and unlabeled (PU) or negative and unlabeled (NU) settings are common, where the distributions between labeled (either positive or negative) and unlabeled (both positive and negative) samples different (see Fig. 1(Tcft)) (Zhang & Zuo; 2008; Bekker & Davis, 2020). Second, additional unlabeled data be gathered after labeling, causing distribution shift. For example, manufacturing processes may keep evolving and thus, the corresponding defects can change and the defect types at labeling differ from the defect types in unlabeled data (see Fig. 1 (Middle)). In addition, for financial fraud detection and anti-money laundering applications, new anomalies can appear after the data labeling process, as the criminals adapt themselves. Lastly, human labelers are more confident on easy samples; thus, easy samples are more likely to be included the labeled data and difficult samples are more likely to be included in the unlabeled data (see Fig. ((Right)). For example, with some crowd-sourcing- based labeling tools, only the samples with some consensus on the labels (as measure of confidence) are included in the labeled set.
Fig. 1. First, positive
As we experimentally demonstrate (in Sec. 5), standard semi-supervised learning methods (Sohn et al., 2020; Chen et al., 2020a; Grill et al., 2020) are sub-optimal for anomaly detection under distribution mismatch, because they are developed with the assumption that labeled and unlabeled data come from the same distribution. Generated pseudo-labels are highly dependent on small set of labeled data; thus, the trained semiisuperrised models would be biased on the labeled data distribution. Transfer learning methods or the frameworks for distribution shifts may constitute alternatives (Pan & Yang, 2009; Yu et al., 2020; Raina et al., 2007) by treating source/targer ttaret data labeled/unlabele /nnlabeledd data. However, these have not been effective with a small mumber of source (labeled) samples (as shown in Sec. 5).
Motivated by the common real-world scenarios, we tackle the distribution mismatch problem for semi- supervised anomaly detection which critical but under-explored. We propose a novel semi-supervised anomaly detection framework, SPADE, that yiclds strong and robust performance even under distribution mismatch. The key aspects of SPADE can be summarized below:
Semi-suporvised learning to efficiently utilize unlabeled data: Carefully-designed compo- nents enable robust semi-supervised learning, by combination of self-supervised and supervised learning stages.
2

---

Published in Transactions on Machine Learning Research (02/2023)
Frameworks
Description
|| Use of data
Examples
Supervised classification
Train supervised model with labeled data
I.
MLP,, RF, XGBoost
Negative supervised classi- fication
Train supervised model while treating unlabeled data as normal data
L+U
MLP, RF, XGBoost
One-class classifier (OCC)
Train OCC only with labeled normal data
L(normal)
OC-SVM, GDE
Negative OCC
Train OCC while treating unlabeled data as nor- mal data
Linorma)-
OC-SVM, GDE
Unsupervised OCC
Train OCC with unlabeled data refinement
L(normal)
SRR (Yoon et al., 2022)
Seml-supervised learning
Train a predictive model via pecudo-labeling and representation learning
L+U
FixMatch (Solin al., 2020),
VIME (Yoon et al,, 2020)
Domain adaptation
Train prodietive model via domsin-invariant
L+U
DANN (Ganin et al., 2016)
representation learning
PU learning
Train predictive model only with L (anomalous)
L(amomstous)
U
(llan & Noto, 2008), Bogging (MMrieeettM & Vert, 2014)
+ U via weighted ensemble learning
Table 1: Conventional approaches to tackle anomaly detection with semi-supervised settings with distribution mismatch. (L: Labeled data, U: Unlabeled data, MLP: Multi-layer Perceptron, RF: Random Forest, GDE: Gaussian Distribution Estimator).
Data efficiency: SPADE introduces a pseudo-labeling mechanism using an ensemble of OCCs and reduces the dependence on the labeled data as the predictors are trained with a small number of labeled and pseudo-labeled samples.
Robustness against distribution mismatch: SPADE constructs more robust decision boundaries that can identify the pseudo-negative and positive samples without overfitting to the labeled data, a critical aspoct when there is distribution mismatch.
Selecting hyperparameters robustly: We propose a novel approach using a partial matching method to pick the important hyperparameters without a validation set. This innovation is critical as conventional hyperparameter selection would rely on a validation set, which is often unavailable in real world with limited labelod data.
Strong results in real-world settings: We show state-of-the-art semi-supervised anomaly de- tection performance of SPADE in multiple settings that represent common real-world sccnarios. AUC improvements of SPADE can be up to 10.6% on tabular data and 3.6% on image data. We additionally focus on an important real-world machine learning challenge: frand detection with distribution shifts over time due to the adversarial nature of the environment. We show that SPADE consistently outperforms alternatives.
2 Related Work
Semi-supervised learning. State-of-the-art methods (Sohn et al., 2020; Chen et al., 2020a; Grill et al., 2020) are developed under the assumption that both labeled and unlabeled samples come from the same distribution. They have pseudo-labeling approaches based on the consistency of label predictions with different. augmentations. Such approaches are highly dependent on the small amount of labeled data. Thus, the bias from the labeled data would propagate to pseudo-labels of the unlabeled data, causing them to construct a biased predictive model if there is distribution mismatch between labeled and unlabeled data. Kim et al. (2020) tackles this in the sctting where only the label priors are different. DeepSAD (Ruff et al., 2020) tackles semi-supervised anomaly detection problem while treating unlabeled samples as normal samples.
As a way of employing OCCs, SPADE differentiates from typical labeling methods used in semi- supervised learning (Lee et al., 2013; Sohn et al., 2020) that require building binary classifiers to assign pseudo-labels. We argue that OCC-based pseudo-labeling is better-suited when there exists distribution mismatch between labeled and unlabeled data, common pitfall for semi-supervised anomaly detection
3

---

Published in Transactions on Machine Learning Research (02/2023)
applications, and more universally applicable (e.g., binary classifier isn't available for PU settings). Yoon et al. (2022) also employs an ensemble of OCCs for fully-unsupervised settings. However, it only identifies pseudo-normal samples from unlabeled data and it needs prior knowledge on label distribution, which may not be available in practice (more details can be found in Appendix. A.4).
Distribution mismatch. Some recent works directly addressed the distribution mismatch between labeled and unlabeled data. (Chen et al., 2020b; Saito et al., 2021) assume that the distribution of labeled data and testing data are the same but the unlabeled data include additional out-of-distribution samples. Both papers focus on filtering out out-of-distribution samples from the unlabeled data to match the distribution between labeled and unlabcled data. On the other hand, in SPADE, the testing distribution is the union of the labeled and unlabeled distributions and the labeled data distribution is different from the testing distribution. Pang et al. (2019) assumes the existence of positively labeled samples which are included in the PU scenarios in SPADE. Pang et al. (2021) further assumes new anomaly types in unlabeled data, which also addressed in this paper (sce Soct. 5.1).
Domain adaptation. Various methods have been proposed to address the issue of the training distribution being different from the testing distribution (Long et al., 2016; Baktashmotlagh et al., 2013; Sun et al., 2019). These often focus on learning domain- invariin representations for better generalization to testing set with different distributions. If we assume that we have access to features of the test data (which is a common assumption in domain adaptation), we can consider the domain adaptation problem as a semi-supervised learning problem where training data are treated as labeled and test data are treated as unlabeled. However, with small amount of labeled data (less common in domain adaptation setting), the performance of the trained model on small source data would be limited.
Positive-U Positive-Unlabeled Unlabeled (PU) learning. An important special scenario is when we only have the positive samples as the labeled data, while unlabeled data include both positive and negative samples (Zhang & Zuo, 2008). In this setting, the labeled data distribution is clearly different from the unlabeled data, as a special case of semi-supervised anomaly detection with distribution mismatch. Related literature on PU learning is summarized in Bekker & Davis (2020). There are two commonly-used approaches: (i) two-stage models (He et al., 2018; Chaudhari & Shevade, 2012), where the first stage is discovering the confident negative labels and the second stage is training the supervised model using positive labels and confident negative labels; (ii) biased learning by treating all the unlabeled data as negalive samples with class label noise (Liu et al., 2003; Sellamanickam et al., 2011). The shortcoming of (i) is excluding the possible positive samples from unlabeled data, whereas the shortcoming of (ii) is contamination of unlabeled data that affects model training. While being relevant, these are limited to the special case of PU setting, and sub-optimal when applied to the general semi-supervised settings.
3 Problem Formulation
We focus on the general semi-supervised anomaly detection problem with distribution mismatch. Consider the given labeled training data D' = [(x]. = and unlabeled training data D" = )- x'
~ Pkx and x" ~ P' are the feature vectors and P\x and PX are corresponding feature distributions of the labeled and unlabeled data, respectively. For anomaly detection, the labels y E y are either normal (0) or anomalous (1) and there are far more normal examples than anomaly, i.e., P(y = 0) > P(y f1y=1), 1). Most semi-supervised methods assume that both labeled and unlabeled data come from the same distribution (i.e., P'x = Pt)). In this work, we aren't limited by this assumption and allow the scenario of the distributions between labeled and unlabeled data to be different (ie., P\ # Pt). We exemplify such scenarios in Fig. 1. For instance, if new anomaly types are only included in the unlabcled data, P would be different from P'x. The labels y are determined by the unknown function f X + y where x', x" E X. Our main objective is to construct an anomaly detection model f X + y that can minimize the test loss L(f(x), y) in the union of Pkx and Px As a way of motivation, the conventional approaches to tackle this problem along with their limitations are summarized in Table. 1. All these are quantitatively compared with SPADE in Sec. 5. Further details can be found in Appendix A.
No.
4

---

Published in Transactions on Machine Learning Research (02/2023)
Normal sample,
Abnormal sample.
Unlabeled sample.
Unlabeled abnornal sample
*
36
36
(a) Original data
(b) Discard unlabeled data (c) Treat unlabeled data as normal data
(d) One-class classifier without using labels
Figure 2: Examples in supervise anomaly detection with distribution mismatch. (a) Original data distribution. Note that the labeled (color) and unlabeled (grey) data distributions are different; (b) Standard supervised learning approach only with labeled data; (c) Standard supervised learning approach after treating all the unlabeled data as normal samples; and (d) OCC without using labels. Purple line represents the decision boundary.
4 Proposed Method SPADE
Sec. 4.1 first explains the design principles of SPADE, and then the implementation details are provided in the subsequent subsections. Sec. 4.2 introduces building blocks of the framework, Sec. 4.3 and 4.4 explain the details of the pseudo-labeler and Sec. 4.5 describes loss functions and optimization.
4.1 Desiderata
The core idea of our framework, Semi-supervised Pseudo-labeler Anomaly Detection with Ensembling (SPADE), is based on sclf-training, following recent advances in semi-supervised learning (Sohn et al., 2020; Chen el al., 2020a). We aim to train a binary classifier for normal and anomalous data by iteratively learning from labeled and pseudo-labes labeled data. As such, the key component is the pseudo-labeld to assign binary labels to unlabeled data. While is common to use a trained binary classifier for pscudo-labeling (Lee et al., 2013; Sohn et al., 2020), we argue that it may be sub-optimal for anomaly detection with distribution shift the decision boundaries of binary classifiers could be highly biased by the small labeled data. As shown in Fig. 2 (b, c), heavily relying the labeled data training with noisy labeled data would have negative impact when labeled and unlabeled data distributions are mismatched. On the other hand, with OCCs (without using the labeled data at all), we can achieve quite reasonable decision boundaries (Fig. 2(d) still not perfect duc to not using labeled information.
In SPADE, we incorporate these motivations and construct the pseudo-labeler in way that it relies less on the labeled data. More specifically, when constructing the OCCs, SPADE excludes the positive labeled data to avoid overfitting to small number of positive labeled data. In addition, SPADE uses the consensus approach on pseudo-labeling to significantly reduce the label noise in pseudo-labeled samples. As such, SPADE can generalize better to when there is distribution mismatch.
4.2 Building blocks
Fig. 3 illustrates the four components of SPADE framework: (i) (data) encoder, (ii) predictor, (iii) pseudo- labeler, and (iv) projection head. First, the encoder: h h: X' - H maps the input features x into latent representations r = (xx). As the encoder, any neural network architecture can be employed - in our experiments, we use multi-layer perceptron (MLP) for tabular data and convolutional neural networks (CNNs) for image data. The predictor q H y utilizes the learned representation r to output the anomaly scores q(r). The anomaly score is determined by the encoder (h) and predictor (q) as follows: q(((x)). Pseudo-labeler and projection head help the encoder and predictor training. lseudo-labeler v H (0, 1, -1) determines
5

---

Published in Transactions on Machine Learning Research (02/2023)
Self supervised loss
Training
Labeled
Partial Distribution Matching
Projection head (g)
One-class classifier
Subset
Pseudo labeler (v)
Subset 2
One-class classifier
Pseudo- labels
Intersect
Labeled data
(Data) Encoder (h)
Predictor (4)
Pseudo labels
One-class classifler
Subset
Unlabsled
Encoded features
Inference
Supervised
loss
Figure 3: (Left) Block diagram of the proposed semi-supervised anomaly detection framework, SPADE. (Right) We zoom the detailed block diagram of the proposed pseudo-labeler which is an ensemble of OCCs. Predictor is a binary classifier. Blue line represents the inference steps.
the pseudo-labels of the unlabeled data x" using ensemble of OCCs. v(h(x")) = 1/0/ - represents pseudo-anomalous/neee posstssoopoooooooaa///aaa unlabeledd The predictor only utilizes the labeled data and unlabeled data with v(h(x")) = 1/0 for training. Lastly, projection head : H - G is the block to help representation learning of the encoder. Any representation learning method can be utilized, such as contrastive learning and pretext task predictions (such as masked autocncoder).
4.3 Pseudo-labeling via consensus
A major novel component of SPADE the design of pseudo-labeler. The pseudo-labeler (v Fig. 3) is Each OCC is trained with the negative labeled data (D)) and one of K disjoint subsets of unlabeled data (DY, D2 -...; Di) ox(x) outputs the anomaly scores of x. We assign the positive pscudo-labels (i.c. anomalous predictions) to unlabeled data samples if all OCCs agree on them: v(h(x")) if I 0ht
composed of an ensemble of K OCCs (01
where
1 if (h(xx)) > th
10
otherwise
Similarly, we assign a nogative pscudo-label (i.c., normal) if all OCCs agrec on negative pscudo-labels: v(h(x*)) = if IT 6t" where
{
1 (h(x))))) < m
9h
(2)
otherwise
Unlabeled data without consensus are annotated as unknown: v(h(x")) = -1 if IT
APu 1%
OF
0.
4.4 Determining r7" using partial matching
In SPADE framework, thresholds 1f" and 17" are critical parameters. One option is considering them uscr-defined soper-parameters and determining them by the hyper-parameter optimization. However, hyper- parameter luning requires extra validation data which should come from labeled training set (same impacts as reducing the number of labeled samples in training data which critical in semi-supervised setting). Instead, we propose learn these parameters without sacrificing the labeled data for validation. We propose adapting the partial matching method (Christoffel et al., 2016), which has been developed to estimate the marginal distribution of unlabeled data by matching the distribution to the known one-class (either positive

---

Published in Transactions on Machine Learning Research (02/2023)
negative) distribution. The underlying intuition is that normal samples are closer to other normal samples, and anomalous samples are closer to other anomalous samples. In our case, we match the distribution of anomaly scores of the positive labeled data to that of unlabeled data to estimate their marginal distribution and determine sp accordingly. The same is applicd to determine 17" using negative labeled data. Formulations for of and 11" are given in Eqs. and below:
ufc - arg min D
(3)
=rg min a fox to
(4)
where Dw is the Wasserstein distance between two distributions. That is, we determine the subsets of the unlabeled data for laeudi- whose Wasserstein distance from labeled data is minimum. More specifically, the outputs of ok are the one-dimensional anomaly scores and we compute the Wasserstein distance between two onc-dimensional anomaly scores. Wasserstein distance between two ono-dimensional vectors can be computed as the integral of the cumulative distribution function differences.
In some semi-supervised settings such PU and NU, only one-class of labeled samples are available. In that case, we employ Otsu's method (Otsu, 1979) to identify the threshold of the class without labeled samples. With Otsu's 's method, we can determine the threshold that minimizes intra-class anomaly score variances in an unsupervised way. More specifically, Otsu's method is applied to one-dimensional anomaly scores. For all unlabeled samples, we extract one-dimensional anomaly scores from the trained OCCs. Then, we find the threshold that minimizes the intra- class variances of two subgroups (splitted by the threshold) of anomaly scores. In PU setting, set 17 using Eq. 3 and 17" using Otsu's method.
4.5 Loss functions and optimization
We train the anomaly detection model q(h())) using three loss functions: (i) binary cross entropy (BCE) labeled and (ii) BCE on pscudo-labeled data, and (iii) self-supervised loss on the entire data. The self-supervised module (e.g., decoder for reconstruction loss, MLP projection head for contrastive loss) is jointly trained with an auxiliary self-supervised loss.
Next, we describe the loss formulations. The BCE loss on the labeled data is proposed as:
Ly = [([((ccc((((M))))))))
and the BCE loss on pscudo-labeled data as:
Ly. E[Lscx(((M(x")) v(h(x"))) 1fv" E (0,1)
Here, instead of subsampling unlabeled data with known pseudo-labels, we assign a binary weight (1("" 133 to each unlabeled sample so that the loss contribution from pseudo-labeled data can be controlled based on the model quality.
To improve the quality of the encoder (h), utilize auxiliary self-supervised losses with various pretext tasks depending on application domain. This may include the reconstruction objective:
LR = [[[MsE(x,g(((())))
more specific objectives to data type, such as contrastive learning (Chen et al., 2020a) and CutPaste (Li et al., 2021) for image.
Overall, the encoder (1), predictor (4), and the self-supervised module (g) are trained by solving the following optimization problem:
h"g"", = arg min [Cy: + aLy- + 3LR],
h.g.4
where a, 3 are hyper-parameter parameters (we set both a and as 1.0 for the experiments). Training loss is used for the convergence critcria if the training loss is convergod (if no improvement is observed in the loss for 5 epochs), we treat that the models are converged as well. Note that the pseudo-labeler also converges during training, often faster. The overall pseudo-code be found in Alg.
7

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm Semi-supervised Pseudo-labeler Anomaly Detection with Ensembling (SPADE).
Input: Labeled / unlabeled training data D' "Du
Output: Trained encoder (h), predictor (g)
Initialize g. h.
Set positively / negatively labeled data Dj, D/
3: while h, q not converged do
4: D=PPECDO-LABBLER(())
Dh)
5: Update 9,h, using Eq. 5.
end while
function PSEUDO-LABELER(D), D/,, D", h)
Divide D into K disjoint subsets (Dk) K
10:
for k=1:K do
11:
Train OCC models o on DE U2
12:
Set / using partial matching with Dj, Di using Eqs. and 4.
13:
end for
14: Build ggeudo-labeles v following Eqs. 1 and 2.
15: Return pseudo-labeler
16:
end function
5
Experiments
We conduct extensive experiments to highlight the benefits of the proposod method, SPADE, in various prac- tical settings of semi-supervised learning with distribution mismatch. We consider multiple anomaly detection datasets for image and tabular data types. As image data, et al., 2019) and Magnetic tile datasets (Huang et al., 2020). As tabular data, we Covertype, Thyroid, and Drug datasets (see Appendix for detailed data description). In Sec. 5.4, we further utilize two real-world fraud detection datasets (Kaggle credit and Xente) to evaluate the performance of SPADE.
MVTec anomaly detection (Bergmann
In all experiments, unless the dataset comes with its own train and test split, we randomly divide the dataset into disjoint train and test data. Then, we further divide the training data into disjoint labeled and unlabeled data. Note that construct labeled and unlabeled data such that they come from different distributions (more details can be found in the following subsections). We run independent experiments and report average values (standard deviations can be found in Appendix C). We use AUC the evaluation metric. More experimental details (on model architectures, training settings, and pseudo-labelers) are provided in Appendix B. Computational complexity analyscs can be found in Appendix B.7.
We compare SPADE baselines from Table Note that not all baselines are applicable every scenario. More specifically, we use Gaussian Distribution Estimator (GDE) for both OCC (only using the negatively labeled data) and Negative OCC (only excluding the positively labeled data). Note that GDE performs the best in comparison to the alternatives in our experiments (including isolation forests, OC-SVM). We use SRR (Yoon et al., 2022) as the unsupervised OCC baseline and Random Forest as the supervised (only using the labeled data) and negative supervised (treat unlabeled data as negative) baselines. For image data, FixMatch is used instead of VIME as the semi-supervised bascline. We use CutPaste (Li et al., 2021) as the baseline architecture for Negative OCC, Unsupervised OCC, and SPADE for MVTec and Magnetic datasets.
5.1 New types of anomalies
Anomalies can evolve over time in many applications. For fraud detection, criminals might invent new fraudulent approaches to trick the existing systems; or for manufacturing, modified process might yield different defects that have been never met before. Therefore, labeled data can get out-dated and newly- gathered unlabeled data can come from different distributions. To mimic such scenarios, we construct datasets with multiple anomaly types. Among multiple anomaly types, we provide subsets of the anomaly types (and
8

---

Published in Transactions on Machine Learning Research (02/2023)
normal samples) the labeled data, It means that other anomaly types only appear in the unlabeled data. Detailed experimental settings can be found in Appendix. B.2.
Datasets
Thyroid
Drug
Covertype
Metrics (AUC)
Overall
Given
Missed
Overall
Given
Missed
Overall
Given
Missed
Supervised
0.815
0.996
0.741
0.818
0.810
0.833
0.858
0.988
0.693
Negative Supervised
0.622
0.837
0.533
0.676
0.670
0.685
0.761
0.881
0.610
OCC
0.711
0.876
0.643
0.741
0.727
0.765
0.897
0.910
0.880
Negative OCC
0.446
0.637
0.367
0.731
0.700
0.780
0.825
0.832
0.815
Unsupervised OCC
0.429
0.612
0.353
0.769
0.747
0,803
0.843
0.853
0.831
VIME
0.592
0.724
0.538
0.792
0.777
0.820
0.837
0.967
0.672
DANN
0.725
0.876
0.662
0.744
0.730
0.768
0.791
0.979
0.552
SPADE (Ours)
||
0.921
0.997
0.891
0.837
0.831
0.849
0.928
0.957
0.892
Table 2: Experimental results with new types of anomalies scenario in terms of Overall Given / Not given
(Missed) AUC, Iverall/Given/// set for evaluation.
Put all/given, htl/givven/missed missed anomaly types and normal samples in the test
Tables 2 and 4 (left) show that SPADE achieves consistently and significantly better performance all 3 metrics (overall, given, and missed AUC), demonstrating its generalizability unseen anomalies. On the other hand, supervised and semi-supervised (VIME and FixcMttc) methods remain highly biased towards given anomalies and generalize poorly to new types of anomalies. Compared to the best baseline, SPADE improves overall AUC by 0.106, 0.015, and 0.031 on the three tabular datasets.
Each baseline has its own limitations. Supervised classifiers camnot utilize unlabeled data at all, and negative supervised classifier suffers from contaminated labeled data for training the predictive model. OCC models are suboptimal as they cannot utilize the anomalous label information. Semi-supervised learning baselines suffer from distribution mismatch between labeled and unlabeled data. For domain adaptation bascline, it shows poor performances with small number of source samples.
5.2 Labeling based on the 'easiness' of samples
The difficulty for human labeling may differ across different samples while some samples are easy label, some samples can be misleadingly difficult to humans because they appear differently from the known cases. To simulate this scenario, we focus an experiment where the labeled data only includes casy-to-label samples while hard-to-label samples are included in the unlabeled dalaset. To this end, we train logistic regression using the entire training data, and gather the labeled samples where confidence of the trained logistic regression outputs are larger than a cortain threshold and the predictions are correct. Details can be found in Appendix. B.3.
Datasets
Thyroid
Drug
Covertype
Supervised
0.805
0.848
0.878
Negative Supervised
0.626
0.701
0.599
OCC
0.787
0.838
0.888
Negative OCC
0.464
0.741
0.826
Unsupervised OCC
0.484
0.786
0.846
VIME
0.728
0.849
0.843
DANN
0.731
0.754
0.835
SPADE (Ours)
0.833
0.846
0.892
Table 3: Experimental results with labeling based the 'easiness' of samples in terms of overall AUC.
9

---

Published in Transactions on Machine Learning Research (02/2023)
Scenarios
New anomalies
Fasiness
Datasots
MVTec
Magnetic
MVTec
Magnetic
Supervised
84.3
82.3
90.9
81.7
Negative Supervised
76.5
63.5
79.2
59.3
Negalive OCC
81.3
69.0
87.6
70.1
Unsupervised OCC
85.4
72.2
88.4
73.1
FixMatch
81.4
69.1
83.5
70.8
SPADE (Ours)
87.9
85.2
92.1
83.9
Table 4: Experimental results on image domain with (left) new types of anomalies, (right) labeling based on easiness scenarios in terms of overall AUC.
Tables 3 and 4 (right) show that SPADE achieves superior or similar anomaly detection performances compared to the best alternative. This constitutes a great potential in reducing human labeling costs by allowing the labelers to skip samples if they would take too long to correctly label. The experimental results with the opposite setting (only labeling the high-risk samples) can also be found in Appendix D.1.
5.3 PU (Positive & Unlabeled) learning
With only positive samples as the labeled data and all other samples being unlabeled, i.c. the positive and unlabeled (PU) settings, the distributions between labeled (only positive samples) and unlabeled (both positive and negative samples) would be different. We use the same experimental settings with the 'new types of anomalies' scenario except additionally excluding normal samples from the labeled data, to represent PU setting. Detailed experimental settings can be found in Appendix. B.4.
Datasets
Thyroid
Drug
||
Covertype
Metrics (AUC)
Overall
Given
Missed
|| Overall
Given
Missed || Overall
Given
Missed
Negative Supervised Negative OCC
0.786
0.997
0.698
0.839
0.839
0.840
0.846
0.996
0.657
0.470
0.695
0.377
0.739
0.709
0.787
0.849
0.864
0.831
Unsupervised OCC
0.519
0.707
0.441
0.771
0.748
0.800
0.863
0.842
Weighted Elkanota (Flkan & Noto, 2008)
0.772
0.934
0.705
0.711
0.714
0.706
0.699
0.917
0.422
BaggingPU (Mordelet & Vert, 2014)
0.787
0.964
0.714
0.734
0.740
0.724
0.726
0.907
0.497
SPADE (Ours)
0.929
0.996
0.901
|| 0.840
0.842
0.837
||
0.896
0.940
0.839
Table 5: Experimental results on PU settings on 3 tabular datasets in AUC of overall /avell/gg/////ige /missed (not given). Due to the absence of negatively-labcked samples, Supervised, OCC, semi-supervised, and domain adaptation baselines are excluded. Instead, two PU baselines are included..
Table 5 compares the performances of the proposed method (SPADE) in PU settings on multiple tabular datasets. SPADE, generalizes much better and outperforms all other alternatives with significantly better AUC in missed (not given) anomaly types. Note that PU baselines severely suffer from distribution mismatches when new types of anomalics are included in the unlabeled data.
5.4 Time-varying distributions: real-world fraud detection
We evaluate the proposed framework with two real-world fraud detection datasets: (1) Kaggle credit card fraud (0.17% anomaly ratio with total 284807 samples), (ii) Xente fraud detection (0.20% anomaly ratio with total 95662 samples). For these tasks, anomalics are evolving (i.c., their distributions are changing over time) (Grover et al., 2022). To catch the evolving anomalies, we need to keep labeling for new anomalies
Inttps://wwe http.///ssssttttttt//////ttttatt 2https://zindi, afwiia//aaaaaaa attrica//commeiiiiions/tttttttte franeeddtetttttiirccccaaaaagoooe
10

---

Published in Transactions on Machine Learning Research (02/2023)
and retrain the anomaly detection model. However, labeling is costly and time consuming. Even without additional labeling, SPADE can improve the anomaly detection performance using both labeled data and newly-gathered unlabeled data.
Datascts
Kaggle Credit Fraud
Xente Fraud
Labeling ratio
5%
10%
10%
20%
Supervised
0.975
0.977
0.906
0.925
Negative Supervised
0.971
0.976
0.909
0.918
OCC
0.717
0.803
0.891
0.920
Negative OCC
0.838
0.835
0.608
0.630
Unsupervised OCC
0.897
0.897
0.806
0.912
VIME
0.941
0.943
0.859
0.893
DANN
0.921
0.922
0.798
0.822
SPADE (Ours) 0.982
0.983
0.920
0.931
Table Experimental results on two real-world fraud delection datasets in lerms of overall AUC.
In our experiments, we split the train and test data based the measurement time. Latest samples are included in the testing data (50%) and early acquired data is included in the training data (50%). We further divide the training data as labeled and unlabeled data. Early acquired data are included in the labelod data (5%-20%), while later acquired data are included in the unlabeled data (80%-95%). We use AUC as the anomaly detection metric. As shown in Table. 6, SPADE consistently outperforms alternatives for different. labeling ratio values both datasets, taking advantage of the unlabeled data and showing robustness to evolving distributions.
6 Discussions
Accuracy of the lseudo-labels. SPADE is based on the proposed pseudo-labeling mechanism. The accuracy the pscudo-labeler highly related the robustness of scmi-supervised anomaly detection, We analyze the accuracy (in precision) of the pseudo-labels vs. anomaly score percentiles for both normal and anomalous samples.
Precsion (Namal)
Precilon (Norma))
0.98
2.95
0.55
0.90
0.98
Precsion
0.50
0.94
2.75
(Abnrrml)
Precision (Abrormal)
0.50
Proctaion (Abnormal)
0.3
Proposed Pseudo-labeler
Random Pseudo-labeler
0.45
0.5
Random Pauudollabeler
Proposed
Threshold (ff
0.4
Threshold
0.2
Proposed Pseudo labeler
0.3
Random Pseudo labeler
Threshold (n'.
50
100
20
20
100
20
80
100
Anomaly score percentiies
Anomaly score percantiles
Anomaly score percentiles
(a) Thyroid
(b) Drug
(c) Covertype
Figure& 4: Precision for pseudo-labelers across anomaly percentiles on 3 tabular datasets with new types of anomalies. 17" 17 represents the discovered threshold for normal and anomalous pseudo-labels by partial matching in percentile.
Fig. shows that the proposed pseudo-labeler achieves fairly robust pseudo-labeling for normal samples. On the other hand, for anomalous samples, the precision of pseudo-labeling gets high typically when the
11

---

Published in Transactions on Machine Learning Research (02/2023)
anomaly scores are higher than 80%, however we observe drop in precision in some cases, which we attribute to imperfect OCC fitting. While this underlines the room for improvement for pseudo-labeling, due to the robustness of partial matching, the impact of imperfect precision on anomaly detection performance is low. Note that our partial matching algorithm catches this threshold fairly accurately to make pscudo-labels robust without any threshold parameter luning-
Ablation studies. SPADE consists of multiple components and understanding the impact of each component is of high importance. In Table. 7, demonstrate the performance impacts of 5 different components in SPADE on the Thyroid data with the setting of new anomaly types. All components of the SPADE are observed to contribute to the robust anomaly detection performance considerably. Self-supervised learning component contributes to 0.018 AUC improvements of SPADE framework. In addition, with majority votes instead of unanimous votes for pseudo-labeling, the performance of SPADE is degraded by 0.024 AUC. Additional ablation studies on other datasets can be found in Appendix D.2 and D.3.
a is a critical hyper-parameles of SPADE determining the importance of pseudo-label loss in comparison to given labeled data. We analyze the impact of this rrper-parameter in Fig. 5. With 0, the performance is much worse than > 0 on Thyroid (0.08 lower AUC) and Covertype (0.06 lower AUC) datasets. This underlines the impact of utilizing the unlabeled data. In addition, the performances are similar across different a >> O, remonstrating that SPADE isn't sensitive to the hyper-parameter a. Note that, all the experiments in Sec. 5 use a = as the default value.
Variants
Overall AUC
No partial matching
0.898
0.92
(ii) No ensemble
0.894
Overall AUC
(iii) B = 0 (No self-supervised)
0.903
Thyroid
0.88
(iv) No normal samples
0.901
Drug
(v) Majority vote
0.897
0.86
Covertype
0.84
SPADE
0.921
0.82
Table 7: Ablation studies on Thyroid dataset in new anomaly settings: (i) without partial matching, (ii) without an ensemble of OCC, (iii) with , = 0 (No self-supervised learning), (iv) without normal samples for pseudo-labeler training, (v) majority vote instead of unanimous votes for pseudo-labeling.
0.00
0.25
0.75
1.00
1.50
1.75
2.00
a
Figure Overall AUC across different values of a using three tabular datasets. (a= represents SPADE without utilizing pseudo-labels.)
7 Conclusions
Semi-supervised anomaly detection is highly-important challenge in practice in many scenarios, we camnot assume that the distributions of labeled and unlabeled samples the same. In this paper, we focus this and demonstrate the underperformance of standard frameworks in this setting. We propose novel framework, SPADE, which introduces novel pseudo-labeling mechanism using an ensemble of OCCs and a judicious way of combining supervised and self-supervised learning. In addition, our framework involves a novel approach to pick hyperparameters without a validation set, a crucial component for data-efficient anomaly detection. Overall, we show that SPADE consistently outperforms the alternatives in various scenarios AUC improvements with SPADE can be up to 10.6% on tabular data and 3.6% on image data. In addition to anomaly detection, future work extend this semi-supervised framework to multi-class classification or regression with distribution mismatch.
Acknowledgement
We gratefully acknowledge the contributions of Kyle Ziegler and Nate Yoder.
12

---

Published in Transactions on Machine Learning Research (02/2023)
References
Mohiuddin Ahmed, Abdun Naser Mahmood, and Md Rafiqui Islam. A survey of anomaly detection techniques in financial domain. Future Generation Computer Systems, 55:278-288 288, 2016.
Samet Akcay, Amir Atapour-Abarghoeei, and Toby P Breckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In Asian conference on compuler vision, pp. 622 637. Springer, 2018.
Mahsa Baktashmotlagh, Mehrtash T Harandi, Brian C Lovell, and Mathieu Salzmann. Unsupervised domain adaptation by domain invariant projection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 769-776, 2013. 4
Sukarna Barua, Md Monirul Islam, Xin Yao, and Kazuyuki Murase. Mwmote-majorin weighted minority oversampling technique for imbalanced data set learning, IEEE Trons on knowledge and data engineering, 26(2):405-425, 2012. 1
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109 (4):719-704 760, 2020. 1, 2,
Liron Bergman and Yedid Hoshen. Classificationn-are anomaly detection for general data. In International Conference on Learning Representations, 2019. 1
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. MVTec AD-a comprehensive real-world dataset for unsupervised anomaly detection. In CVPR, 2019. 8
Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. The mvtec anomaly
detection dataset: comprehensive real-world dataset for unsupervised anomaly detection. International Journal of Computer Vision, 129(4)) (1)):0008--109,, 2021. 17
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. JMLR, 11:2973-3009, 2010.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93-104, 2000.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. arXiv preprint 1rX.0:1901.03407, 2019. 1
Sncha Chaudhari and Shirish Shevade. Learning from positive and unlabelled examples using maximum margin clustering. In International Conference on Neural Information Processing, pp. 465-473. Springer, 2012. 4
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321 357, 2002. 1
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive lcarning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR,
2020a. 3, 3, 7
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2020b. 4
Marthinus Christoffel, Gang Niu, and Masashi Sugiyama. Class-prior estimation for learning from positive and unlabeled data. In Asian Conference on Machine Learning, pp. 221 236. PMLR, 2016. 6
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 213 213-220, 2008. 3, 10, 20
13

---

Published in Transactions on Machine Learning Research (02/2023)
Andrew Estabrooks, Tacho Jo, and Nathalie Japkowicz. A multiple resampling method for learning from imbalanced data sets. Compulational intelligence, (0(1))18-300 2004. 1
Yaroslav Ganin, Evgeniya. Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. conain-adversaaial training of neural networks. The journal of machine learning research, 17(1):2096-2030 2016.
Izhak Golan and Ran El-Yaniv. Deep anomaly dctection using geometric transformations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 9781 9791, 2018.
Nico Görnitz, Marius Kloft, Konrad Rieck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 46:235- 262, 2013.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised lcarning. arXiv preprint arXiv:2006.077 07733, 2020. 2, 3
Prince Grover, Zheng Li, Jianbo Liu, Jakub Zablocki, Hao Zhou, Julia Xu, and Anqi Cheng. Fdb: Fraud dataset benchnark. Xiv preprint or 2rv: 14417, 2022.
Fengxiang He, Tongliang Liu, Geoffrey I Webb, and Dacheng Tao. Instance-dependent pu learning by bayesian optimal relabeling. arXiv preprinl Xiv: 10:1800 02180, 2018. 4
Yibin Huang, Congying Qiu, and Kui Yuan. Surface defect saliency of magnetic tile. The Visual Computer, 36(1):85 96, 2020. 8, 17
Jae Pil Hwang, Seongkeun Park, and Euntai Kim. A new weighted approach to imbalanced data classification problem via support vector machine with quadratic cost function. Expert Systems with Applications, 38(7): 8580 8585, 2011. 1
Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. orXiv preprint ar arXiv:2077.080 08844.. 2020. 2, 3
Dong-Hyun Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop challenges in representation learning, ICML, volume 3, pp. 896, 2013. 3,
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for
anomaly detection and localization. In CVPR, 2021. 1, 7, 8, 18
Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S Yu. Building text classifiers using positive and unlabeled examples. In Third IEEE International Conference on Data Mining, pp. 179-186. IEEE, 2003. 4
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In ICDM, 2008.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Unsupervised domain adaptation with residual transfer networks. arXiv preprint arXiv:1602, 04433, 2016. 4
Fantine Mordelet and J-P Vert. A bagging svm to learn from positive and unlabeled examples. Pattern Recognition Letters, 37:201-200 209, 2014. 3, 10, 20
Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Iranscctions on systems, man, and cybernetics, (21):62 66, 1979. 7
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions knowledge and data engineering, 13410)11345-1359 2009. 2
Guansong Pang, Chunhua Shen, and Anton van den Hengel. Deep anomaly detection with deviation networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 353-303 2019.
14

---

Published in Transactions on Machine Learning Research (02/2023)
Guansong Pang, Anton van den Hengel, Chunhua Shen, and Longbing Cao. Toward deep supervised anomaly detection: Reinforcement learning from partially labeled anomaly data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1298 1308, 2021. 4
Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. Self-taught learning: transfer learning from unlabeled data. In Proceedings of the 24th international conference Machine learning, pp. 759-766, 2007. 2
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deccke, Shoaīb Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML, 2018.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In ICLR, 2020. 1, 3
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Opemmatch: Open-set semi-supervised learning with open-set consistency regularization. Advances in Newral Information Processing Systems, 34, 2021. 4
Bernhard Schölkopf, Robert C Williamson, Alexander J Smola, John Shawe-Taylor, John C Platt, al.
Support vector method for novelty detection. In NIPS, 1999.
Sundararajan Sellamanickam, Priyanka Garg, and Sathiya Keerthi Selvaraj. A pairwise ranking based approach to learning with positive and unlabeled examples. In Proceedings of the 20th ACM international conference on Information and knowledge management, pp. 663-672, 2011.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NewrIPS, 2020. 2, 3,
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluating representations for deep one-class classification. In ICLR, 2021. 1
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexci A Efros. Unsupervised domain adaptation through self- supervision. arxiv preprint orXiw:1909.110 11825, 2019. 4
David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54(1):45 66, 2004.
Juan Vanerio and Pedro Casas. Ensemble-learning approaches for network security and anomaly detection. Tn Proceedings of the Workshop on Big Data Analylics and Machine Learning for Data Communication Networks, pp. 1 6, 2017.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33. 2020. 3
Jinsung Yoon, Kihyuk Sohn, Chun-Liang Li, Sercan O Arik, Chen- Yu Lee, and Tomas Pfister. Self-supervise, refine, repeat: Improving unsupervised anomaly detection. Transaclions on Machine Learning Research, 2022. URL https:/ http:::://pprreeeeeeee//ooooo// http://wwwe//oooom//////////e
1, 3, 4, 8, 16
Zhongjie Yu, Lin Chen, Zhongwei Cheng, and Jiebo Luo. Transmatch: A transfer-learning scheme for semi-supervised few-shot lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion, pp. 12856 12864, 2020.
Bangzuo Zhang and Wanli Zuo. Learning from positive and unlabeled examples: survey. In 2008 International Symposiums on Information Processing, 2008. 1, 2,
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018
15

---

Published in Transactions on Machine Learning Research (02/2023)
Appendix
A Details of the conventional solutions
A.1 Standard supervised learning
The most straightforward approach is applying the standard supervised learning framework. We can construct the supervised model Ssup only with the labeled data D' as follows.
Nr
Σ ((g(x)).u))
goup = arg m1xx
However, this case, we cammot benefit from the unlabeled data D which can be beneficial for further boosting the performance with various semi-supervised learning framework. Also, the training data distribution X is different from the testing distribution X' which can negatively impact on the test performance. We may treat all the unlabeled data normal samples and apply the supervised learning framework (gsup+) as follows:
N.
Nº
9sup+ = arg max
Σ c(o(x)). +
Σ ,=1
Clg(x3),0))
N
Nu
However, in this case, labeled normal samples contaminated.
A.2 Standard cle-class classifiers (OCCs)
OCCss of the most promising methods to tackle the anomaly detection problem. Instead of using incomplete anomaly labels, can only utilize the labeled normal samples D = {(xx, M;) E D' 0} to construct the OCC (gme). However, in this case, we need to drop all labeled abnormal samples and unlabeled samples which may include quite critical information. We can include the unlabeled cata (D") to construct another OCC (gone+) such as SRR (Yoon et al., 2022). However, it still cannot utilize the labeled abnormal samples.
A.3 suneviuper learning
With both labeled and unlabeled data, we usually prioritize to apply semi-supervised learning approaches. We can utilize the semi-supervised learning framework to construct the anomaly detection model as follows.
N
No.
Σ clg(a)). + (o(a)))
Isemi = arg max
Most semi-supervised learning frameworks assume that the labeled data D' and unlabeled data D come from the same distribution. However, this assumption does not hold in our problem formulation. Thus, possibly-biasod labeled data distribution can negatively affect on the trained semi-supervised model.
A.4 Detailed comparison with SRR (Yoon et al., 2022)
SPADE has some resemblance with the SRR paper (Yoon al., 2022). However, there are clear differences between SPADE and SRR. First, the problem setting is different. One of the biggest novelties of SPADE is tackling important but under-explored problem: semi-supervised learning with distribution mismatch (e.g., common labeling bias). This has not been discussed in SRR which focused setting. Extension from fully unsupervised to general semi-supervised setting is not straightforward. Second, the approach to utilize the positive and negative samples not discussed in SRR, which is critical in SPADE. We should consider how we utilize the normal samples for improving the pseudo-labeler training (please see the ablation studies Table 6) and how we utilize the labeled samples for determining the thresholds Line and Algorithm Third, SPADE can automatically determine the thresholds parameters without true anomaly ratios validation set by the proposed partial matching.
only the fully unsupervised
16

---

Published in Transactions on Machine Learning Research (02/2023)
B Detailed experimental settings
B.1 Convert multi-class datasets into anomaly detection datasets
For Thyroid data², there are 3 classes. The class distributions are (1: 2.47%, 2: 5.06%, 3: 92.47%). We treat label 3 the normal samples and label 1 and 2 as the abnormal samples. We use the pre-defined training and testing dataset division.
For Drug data", there are 7 classes. The class distributions are (1: 75.27%, 2: 2.02%, 3: 4.56%, 4: 8.28%, 3.29%, 6: 2.12%, 7: 4.46%). We treat label as the normal samples and all the other labels as the abnormal samples. We divide the entire dataset into training (50%) and testing (50%).
For Covertype data", there are 7 classes. The class distributions are (1: 36.55%, 2: 48.75%, 3: 6.14%, 4: 0.47%, 5: 1.64%, 6: 2.94%, 7: 3.50)). We treat label and as the normal samples and all the other labels as the abnormal samples. We divide the entire dataset into training (50%) and testing (50%).
For MVTec data (Bergmann et al., 2021); different categorics (15 categories) have different number of anomaly lypes. We set lype 0 as the normal samples and all the other lypes as abnormal samples. Note that we first mix given training and testing data and divide them into training (80%) and testing (20%) to make the same abnormal ratio between training and testing sets.
For Magnetic Tile datasel (Huang el. al., 2020)?, there are 6 types of samples: free, blowhole, crack, break, fray, and uneven. We set the free type as the normal, and the other 5 types as anomalies. We mix given training and testing data and divide them into training (80%) and testing (20%) to have the same abnormal ratio between training and testing sets.
B.2 Detailed experimental settings in Scenario 1: New types of anomalies
On each of the 5 datasets that used in this paper, there multiple types of anomalies. In such scenarios, we only provide subset of anomaly types as the labeled data and the rest of the anomaly types as the unlabeled data. The below explains which types of anomalies are provided as the labeled data for each datasct:
For Thyroid data, we provide label anomaly type to the labeled data (label anomaly type is only the unlabeled data).
For Drug data, we provide label 2,3,4 anomaly lypes to the labeled data (label 5, 6, 7 anomaly types are only in the unlabeled data).
For Covertype data, we provide label 3, 4, 5 as anomaly types to the labeled data (label 6, anomaly lypes are only in the unlabeled data).
For MVTec and Magnetic tile datasets, different categories have different number of anomaly types. We provide the odd types as anomaly types to the labeled data. All the even types of anomalics are included in the unlabeled data.
Note that we only provide 5% of the data as labeled data for tabular datascts and 20% for image datascts, for the scenario of new lypes of anomalies.
hnttpss//archive. ics. uci otg.a///aa......//////////a/arnn
www.e/a.ci..cuuuaeeea.ee htttps:/aaccive..ccc....ee 9https://archive..cca http:///////////aa///a/a/aaaaaaa
7 http://www.....eeeea//aaaaaattta
17

---

Published in Transactions on Machine Learning Research (02/2023)
B.3 Detailed experimental settings in Scenario 2: Labeling based on the easiness of samples
To identify the easiness of the samples, we train logistic regression model using the entire training data, and we gather the labeled samples where confidence of the trained logistic regression model outputs are larger than a certain threshold and the predictions are correct. To balance the labeled data in both normal and abnormal samples, we select top 10% confidences (from trained logistic regression) each class as the labeled data for tabular datasets (20% confidence for image datasets). The rest of the samples are treated as the unlabeled samples.
B.4 Detailed experimental settings in Scenario 3: PU learning
The experimental settings in PU settings are the same with scenario (new types of anomaly) except the following points:
We exclude all the normal samples from the labeled data to make the experiments in PU setting.
We provide 50% of the given anomaly types as the labeled data. However, the number of labeled data is less than Scenario 1 because we exclude all the normal samples from the labeled data.
B.5 Details on model architecture and training
For image data, we use ResNet- 18 as the base network architecture. For representation learning, we incorporate Catta (Li et al., 2021) for MV'Te and Magnetic Tile datasets. We follow all the training details in (Li et al., 2021) (including all the syper-parameters))
For tabular data, we use two-layer perceptron as the base network architecture where the hidden dimensions is the half of the original feature dimensions. Pseudo-labelers consist of 5 Gaussian Distribution Estimator (GDE) based OCCs. For each epoch, update the ensemble of OCCs for the pseudo-labels and further training the data encoder, projection head, and predictor. We set except the ablation studies. We use the training loss as the convergence criteria. More specifically, if the training loss does not improve for 5 epochs, we stop model training.
and , = 1 for all the experiments
train OCCs, we only need data from a single class we do not need label information. For pseudo- labeler of SPADE, we treat the negative labeled data and one of K disjoint subsets of unlabeled data as the one-class data to train the OCCs. We use Gaussian Distribution Estimator (GDE) which utilizes one-class training data (negative labeled data and subsets of unlabeled data) to estimate the density function with maximum likelihood objective for the distribution assumption as the loss function. At inference, the likelihood outputs of GDE for each sample are used as the anomaly scores.
B.6 Baselines
We compare SPADE with various baselines in different settings. Below describes the details of the baselines used in this paper:
Gaussian Distribution Estimator (GDE) for both OCC (only using the negatively labeled data) and Negative OCC (only excluding the positively labeled data)..
Random Forests for the supervised (only using the labeled data) and negative supervised (treat all the unlabeled data as negative)
VIME for the tabular semi-supervised learning baseline and FixMatch" for the image semi- supervised learning baseline.
Domain Adversarial Neural Network (DANN) for the domain adaptation baseline
http://www....a.....//////////aa www.turtaaaaaee.eeeeeeee.. mixture. GaussianNixture html
http:///mmmmeeeeege/earr///////a seeneratted/sklean..eee wwwerrrd///eeeeeeeeeeeeeeeaeeeee aasemblee_anddoooreesttteee RandomPorestCCassiiierr html
Phtte::/ggithaa.cmm/mmppaaaaaaa
18

---

Published in Transactions on Machine Learning Research (02/2023)
Weighted Elkanoto 13 and BaggingPU14 for PU learning baselines.
CatPaste for the base architecture of image domain anomaly detection
B.7 Computational complexity analyses
All the experiments are done on a single V100 GPU. For tabular datasets, training and inference need at most 1 hour per each experiment (with the largest dataset, Covertype). For image dataset, training and inference need at most 4 hours per each experiment, mostly due to the representation learning with CutPaste. Note that the pscudo-labeler (an ensemble of OCCs) is re-trained per an epoch (not per an itcration) and we use shallow OCCs (GDE) for the ensemble to further reduce the computational complexity.
C Standard deviations of the experiment results
In this section, we report the standard deviations of the experimental results described in the main manuscript across 5 independent runs.
Datascts
Thyroid
Drug
Covertype
Metrics (AUC)
Overall
Given
Missed
Overall
Given
Missed
Overall
Given
Missed
Supervised
0.051
0.003
0.076
0.028
0.031
0.031
0.003
0.001
0.008
Negative Supervised
0.037
0.094
0.025
0.058
0.062
0.055
0.003
0.004
0.004
OCC
0.094
0.074
0.108
0.062
0.071
0.052
0.001
0.001
0.001
Negative OCC
0.002
0.006
0.001
0.020
0.022
0.021
0.001
0.002
0.001
Unsupervised OCC
0.017
0.034
0.010
0.013
0.016
0.018
0.001
0.002
0.001
VIME
0.068
0.064
0.072
0.075
0.080
0.067
0.014
0.001
0.032
DANN
0.063
0.075
0.061
0.084
0.083
0.088
0.010
0.001
0.022
SPADE (Ours)
0.029
0.001
0.041
0.024
0.026
0.026
0.001
0.001
0.002
Table 8: Standard deviations of experiments with new types of anomalies scenario in terms of Overall / Given Not given (Missed) AUC, Overall/ Given Put ht////vven/mmeeeee anomaly types and normal samples in the test set for evaluation.
Datasets
Thyroid
Drug
Covertype
Supervised
0.013
0.009
0.002
Negative Supervised
0.010
0.033
0.002
OCC
0.031
0.016
0.001
Negative OCC
0.004
0.020
0.002
Unsupervised OCC
0.015
0.016
0.001
VIME
0.033
0.015
0.017
DANN
0.045
0.037
0.018
SPADE (Ours)
0.021
0.014
0.001
Table 9: Standard deviations of experiments with labeling based on the 'easiness' of samples in terms of overall AUC.
18nttps://pulearn. http:///www..ggggit.ooooo//oo/op http:////ww....aaaaaa/a/////////
Mhttpe::/pplearnngge tit.... www.hhb..oopueeeeeeeeeeooeee ht/ppplearm/ttcppplear/ll/ htw.e////ddeeeettaaaaainnng based- based-pu-learnnn learniin
19

---

Published in Transactions on Machine Learning Research (02/2023)
Scenarios
New anomalies
Fasiness
Datasots
MVTec
Magnetic
MVTec
Magnetic
Supervised
0.048
0.034
0.035
0.025
Negative Supervised
0.074
0.025
0.049
0.034
Negalive OCC
0.034
0.025
0.028
0.026
Unsupervised OCC
0.038
0.024
0.034
0.029
FixMatch
0.033
0.025
0.037
0.034
SPADE (Ours)
0.041
0.032
0.032
0.025
Table 10: Standard deviations of experiments image domain with (left) new types of anomalies, (right) labeling based on easiness scenarios in terms of overall AUC.
Datasets
Thyroid
Drug
Covertype
Metrics (AUC)
Overall
Civen
Missod
Overall
Given
Missed
Overall
Given
Missod
Negative Supervised Negative OCC
0.028
0.001
0.040
0.011
0.013
0.014
0.001
0.000
0.002
0.007
0.018
0.003
0.020
0,021
0.020
0.001
0.001
0.001
Unsupervised OCC
0.016
0.016
0.017
0.016
0.016
0.020
0.001
0.001
0.001
Weighted Elkanoto (Elkan & Noto, 2008)
0.022
0.035
0.026
0.018
0.022
0.021
0.006
0.006
0.010
BaggingPU (Mardelet & Vert, 2014)
0.029
0.019
0.036
0.019
0.020
0.020
0.021
0.016
0.027
SPADE (Ours)
0.042
0.001
0.060
0.008
0.008
0.016
0.002
0.001
0.002
Table 11: Standard deviations of the experiments on PU settings on tabular datasets AUC of over-
all/g ////gvve//nee /missed (not given).
Datasets
Kaggle Credit Frand
Xente Fraud
Labeling ratio
5%
10%
10%
20%
Supervised
0.002
0.001
0.024
0.009
Negalive Supervised
0.002
0.002
0.022
0.012
OCC
0.021
0.043
0.064
0.010
Negative OCC
0.011
0.007
0.005
0.010
Unsupervised OCC
0.004
0.004
0.090
0.011
VIME
0.012
0.013
0.023
0.019
DANN
0.033
0.027
0.013
0.021
SPADE (Ours)
0.001
0.001
||
0.001
0.009
Table 12: Standard deviations of the experiments with two real-world fraud detection datasets in terms of
overall AUC.
Additional Experiments
D.1 Labeling high-risk samples
In this subsoction, we evaluate the performance of SPADE in PNU settings only with the labeled high-risk samples which is a common practical setting in fraud delection applications (including anti-money laundering). More specifically, the predictive model first estimates the anomaly scores of the unlabeled data. Then, the users manually check the samples only with high anomaly scores, and label them as either positive or negative. Thus, most labeled samples are actually high-risk samples and most unlabeled samples are low-risk samples which make the distribution differences between labeled and unlabeled data.
20

---

Published in Transactions on Machine Learning Research (02/2023)
Similar with casiness scenario, we first train simple logistic regression model (with the full label) and compute the anomaly scores of the unlabeled data. Then, we only extract the high risk samples (e-g., with top 2% highest anomaly scores). Then, we provide true labels for 50% (selected by uniformly random) of those high risk samples. It means that we have 1% of labeled data (either positive or negative) and 99% of unlabeled data. We exclude original OCC as the baseline because in some cases, there are too small numbers of negatively labeled samples which make OCC hard converge.
Datasets
Thyroid
Drug
Covertype
Labeling ratio
1%
1.5%
2.5%
1%
1.5%
2.5%
1%
1.5%
2.5%
Supervised
0.758
0.984
0.984
0.578
0.655
0.615
0.619
0.602
0.669
Negative Supervised
0.726
0.814
0.905
0.697
0.727
0.778
0.635
0.667
0.734
Negative OCC
0.466
0.468
0.469
0.725
0.729
0.734
0.829
0.836
0.848
Unsupervised OCC
0.502
0.526
0.519
0.763
0.766
0.769
0.846
0.851
0.865
VIME
0.677
0.703
0.717
0.669
0.681
0.690
0.841
0.843
0.847
DANN
0.735
0.744
0.749
0.724
0.747
0.761
0.749
0.762
0.769
SPADE (Ours)
0.924
0.983
0.981
0.828
0.835
0.838
0.871
0.867
0.865
Table 13: Experimental results with labeling only high-risk samples in terms of overall AUC.
Table 13 shows that SPADE achieves superior or similar anomaly detection performance compared to the best alternative.
D.2 Additional ablation studies
Scenarios
New anomaly types
Easiness
Variants
Drug
Covertype
Thyroid
Drug
Covertype
No partial matching
0.827
0.916
0.811
0.830
0.869
No ensemble
0.830
0.915
0.786
0.830
0.876
(ii) B = 0 (No self-supervised)
0.829
0.919
0.818
0.827
0.877
(iv) No normal samples
0.835
0.922
0.822
0.841
0.887
(v) Majority vote
0.835
0.918
0.807
0.839
0.890
SPADE
0.837
0.928
0.833
0.846
0.892
Table 14:
Ablation studies multiple tabular datasets with new anomaly and easiness settings: (i) without partial matching, (ii) without an ensemble of OCC, (iii) with 3 = (No self-supervised learning), (iv) without normal samples for pseudo-labeler training, (v) majority vote instead of unanimous votes for pseudo-labeling.
21

---

Published in Transactions on Machine Learning Research (02/2023)
D.3 Additional sensitive analyses on 3
In this subsection, we provided additional sensilive analyses on the important hyper-parameter (3) using three tabular datasets with new anomaly settings.
0.92
Overall AUC
0.90
Thyroid
0.88
Drug
Covertype
0.86
0.84
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
B
Figure 6: Overall AUC across different values of using three tabular datasets. (B 0 represents SPADE without self-supe learning.)
22


Reviews:
[{"review_id": "ztExoQ9H06", "reviewer": "Reviewer_Jqdn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#Strengths\n\n1. This paper focuses on the problem that distribution mismatch commonly in semi-supervised learning with labeled and unlabeled data coming from different distributions. It has great practical significance.\n2. This paper proposes a new approach to select the hyper-parameters of thresholds in the design of the pseudo-labeler.\n3. SPADE shows a significant improvement compared with other methods in multiple settings that represent common real-world scenarios.\n4. The methods can be flexibly applied with several self-supervised learning methods to improve performance. \n\n#Weaknesses\n\n1. This paper claim that using partial matching to determine the thresholds $\\eta^p$ and $\\eta^n$ is free of extra validation datasets and it can reduce the number of labeled samples in training data which is critical in semi-supervised settings. But the selection of other hyper-parameters(such as the trade-off parameters $\\alpha,\\beta$  and the selection of models in the training phase) still needs a validation dataset. Further explanation about the validation dataset should be made.\n2. In subsection 4.3, \"Each OCC is trained with the negative labeled data and one of K disjoint subsets of unlabeled data\", how to train the OCCs with negative labeled data and unlabeled data should be described minutely. What loss functions or methods are employed to train the OCCs?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of semi-supervised anomaly detection and proposes a novel framework SPADE which is not limited by the assumption that labeled and unlabeled data come from the same distribution. They introduce a pseudo-labeling mechanism using an ensemble of OCCs and combining supervised and self-supervised learning. In addition, their framework involves an approach to pick hyper-parameters without a validation set. Experiments show that SPADE outperforms the alternatives in various scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The reason that using partial matching to determine the thresholds $\\eta^p$ and $\\eta^n$ is free of extra validation datasets should be further explained.\n2. How to train the OCCs with negative labeled data and unlabeled data should be more described. What loss functions or methods are employed to train the OCCs?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hXgA9WwRDg", "reviewer": "Reviewer_LKzW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- Overall, this paper is well-written, and easy to follow, even though I am not extremely familiar with the anomaly detection task.\n\n- The idea is simple. Ensembling multiple one-class classifiers for anomaly detection improves the robustness of the method and sounds rather natural to me.  The final detector is an integration of the K detectors so that the final performance is improved.\n\n- The effectiveness of the proposed method is validated on different datasets, from synthetic to real-world data, and across tabular to image datasets. Ablation studies also isolate the effects of each component of the final SPADE method. \n\n**Weakness**:\n\n- The major concern stems from the weak link between the distribution mismatch and the proposed method. The whole paper only states what the authors have done but does not mention why doing so can address the issues. Specifically, It remains very unclear to me why the distribution mismatch can be addressed by the proposed method, and which component of the method is the main block design for the distribution mismatch. \n\n- Following the previous concern, it is widely known that consensus can improve robustness, and it is unclear whether this can be claimed as a novelty to the paper, also the author did not illustrate clearly why consensus achieves better robustness specifically for distribution mismatch. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a semi-supervised ensemble method for anomaly detection. In contrast to the previous methods that often assume that the labeled and unlabeled data come from the same distribution, this paper proposes a method that shows empirical robustness even under distribution mismatch. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The author should explain more on why the proposed method can address or alleviate the mismatch issues, and which part of the method is mainly designed to do so, according to the ablation study, each component boosts the performance, and it remains unclear which components can address mismatch, and if all of them are able to do so, please provide an explanation on why they are able to do so. \n\n- In section 4.4, can you describe in detail how did you use Otsu's method to identify the threshold, and how did you calculate the Wasserstein distance, I feel there are many details that are skipped by the authors. \n\n- I realized that in Algorithm 1, g, h, and q are initialized, is there a specific reason for doing this, wondering what would happen if they were continuously trained. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cQLiflbJNz", "reviewer": "Reviewer_52Xa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The motivation is strong and clear. The research problem is realistic and important.\n- Experiments are overall convincing. In partial cases, the performance improvement is clear. \n\n**Weaknesses**\n\n- The proposed method is complicated. There are multiple modules in the proposed method of this work. It needs to balance carefully different modules. \n- Writing needs to be improved. Many key parts lack discussion and description. More details are provided below. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents work on semi-supervised anomaly detection, where labeled and unlabeled data are allowed to come from different distributions. A framework SPADE is proposed accordingly. SPADE shows state-of-the-art performance across multiple scenarios with distribution mismatch in various domains, e.g., the tabular and image. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Is this work the first to target the problem, i.e., semi-supervised anomaly detection under distribution mismatch? There are some discussions on related work. However, there is no discussion about this. \n- The contributions can be reorganized. Some contributions are only the procedures of the proposed method. \n- The paper claims that OCC-based pseudo-labeling is better suited when a distribution mismatch exists between labeled and unlabeled data. Is there some evidence to support this claim?\n- The paper argues that, with a small amount of labeled data (less common in domain adaptation settings), the performance of the trained model on a small source data would be limited. For the proposed method of this paper, how to address the issue of insufficient labeled data at a high level? It is not clear. Besides, could the paper provide intuitions about why the proposed method outperforms previous works?\n- To determine the parameters $\\eta^p$ and $\\eta^n$, the partial matching method is employed. It seems that the method only can help to identify easy outliers. How to address the problem? Additionally, how to determine $\\alpha$ and $\\beta$ in practice? Are there ablation studies for their sensitivity?\n- In partial cases, some baselines outperform the proposed method of this paper. The baselines are not designed for the specific problem in this paper. Could the paper provide more explanations about why they can achieve better performance?", "broader_impact_concerns": "This paper did not provide discussions on broader impacts. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on semi-supervised anomaly detection, where a semi-supervised ensemble method is proposed. The proposed method successfully avoids the assumption that the labeled and unlabeled data come from the same distribution. It has excellent potential to be applied in realistic scenarios. Extensive experiments also confirm the effectiveness of the method. 

Before responses from authors, three reviewers pointed out several concerns including (1) the link between distribution mismatch and the proposed method; (2) the settings of hyper-parameters; (3) unclear descriptions. The authors provide detailed responses to address the concerns. All reviewers are satisfied with the responses and appreciate the contributions of this paper. We, therefore, recommend an acceptance.

This paper is high quality and then selected for the Featured Certification, which is recommended by most reviewers. After the responses from the authors, the claims of this paper are clearly justified. AE and reviewers consider that this paper makes sufficient contributions to the research area of semi-supervised anomaly detection, mainly including its strong motivation, practical problem setting, proposed novel framework, convincing experiments, and detailed analysis/discussions of experimental results. Therefore, it deserves the Featured Certification and should be highlighted. 
Decision: Accept (poster)