Uncertainty-aware Evaluation of Auxiliary Anomalies with the Expected Anomaly Posterior
Authors: Lorenzo Perini
Maja Rudolph
Sabrina Schmedding
Chen Qiu
Venue: Accepted by TMLR ()

Abstract: Anomaly detection is the task of identifying examples that do not behave as expected. Because anomalies are rare and unexpected events, collecting real anomalous examples is often challenging in several applications. In addition, learning an anomaly detector with limited (or no) anomalies often yields poor prediction performance. One option is to employ auxiliary synthetic anomalies to improve the model training. However, synthetic anomalies may be of poor quality: anomalies that are unrealistic or indistinguishable from normal samples may deteriorate the detector's performance. Unfortunately, no existing methods quantify the quality of auxiliary anomalies. We fill in this gap and propose the expected anomaly posterior (EAP), an uncertainty-based score function that measures the quality of auxiliary anomalies by quantifying the total uncertainty of an anomaly detector. Experimentally on 40 benchmark datasets of images and tabular data, we show that EAP outperforms 12 adapted data quality estimators in the majority of cases. Code of EAP is available at: https://github.com/Lorenzo-Perini/ExpectedAnomalyPosterior.

Published in Transactions on Machine Learning Research (12/2024)
Uncertainty-aware Evaluation of Auxiliary Anomalies with the Expected Anomaly Posterior
Lorenzo Perini
jorenzo.perinl@kaulueenn.d
Bosch Center for AT, Germany DTAI lab & Aluven.Al, KU Leuven, Belgium
Maja Rudolph
majja.rudolphhhhss.uoku www.h com
Bosch Center for AI, USA University of Wisconsin- Madonsii-MMadioo, USA
Sabrina Schmedding
cabrina.schmmeddiingide.
Bosch Center for AT, Germany
Chen Qiu
chen. Cicer.qiu@uss.oo www.gggwwwwwsso...oooooo bosch. com
Bosch Center for AI, USA
Reviewed on OpenReview: https: // openreview. net/ forum? id= Qd=
Abstract
Anomaly detection is the task of identifying examples that do not behave expected. Because anomalies are rare and unexpected events, collecting real anomalous examples is often challenging in several applications. In addition, lcarning an anomaly detector with limited (or no) anomalies often yields poor prediction performance. One option is to employ auxiliary synthetic anomalies to improve the model training. However, settc anomalies may be of poor quality: anomalies that are unrealistic or indistinguishable from normal samples may deteriorate the detector's performance. Unfortunately, no existing methods quantify the quality of auxiliary anomalies. We fill in this gap and propose the expected anomaly posterior (EAP), uncertainty- based score function that measures the quality of auxiliary anomalies by quantifying the total uncertainty of an anomaly detector. Experimentally on 40 benchmark datasets of images and tabular data, we show that EAP outperforms 12 adapted data quality estimators in the majority of cases. Code of EAP available at: https://githir
1 Introduction
Anomaly detoction aims identifying the examples that do conform the normal behaviour (Chandola et al. 2009 Anomalies often connected to adverse events, such as defects in production lines [Wang
et al. 2024 excess water usage (Perini et al. 2023), failures in petroleum extraction (Martí et al. 2015),
breakdowns in wind turbines (Perini et al. 2022). Detecting anomalics in time can reduce monetary costs and protect resources from harm Bergmann et al. 2022). For this reason, there has been significant effort to develop data-driven hods for anomaly detection.
Unfortunately, anomalies are inherently rare and sparse, which makes collecting them hard. As a result, the data used to train data-driven methods for anomaly detection only contains limited number of anomalies. In autonomous driving, for instance, sensor malfunctions or unexpected redestrian movements are infrequent.

---

Published in Transactions on Machine Learning Research (12/2024)
but critical to address. As an additional challenge, available anomalies rarely represent all potential cases due to the unpredictable nature of these events. In financial fraud detection, new techniques and schemes are constantly emerging, meaning previously identified anomalies do not cover future fraudulent methods. These factors highlight the difficulty of obtaining comprehensive anomaly datascts, as new and umpredictable anomaly types are inherent to the very nature of these events.
With recent improvements in generative modeling (e.g. (Ho al., 2020 Dhariwal & Nichol 2021 it seems natural to introduce auxiliary anomalies, e.g. for training an anomaly detector (Murase & Fukumizu 2022];
or for model selection (Fung et al., 2023). However, there are several failure cases for generated anomalies that should not be neglected. For one, auxiliary anomalios might be too similar to normal examples. For instance, the defects introduced into images of products might be imperceptible, making the image indistinguishall from a normal counterpart. On the other hand, the quality of a generated anomaly also deteriorates as it becomes too unrealistic, e.g. when the generated defect of a product is too severe. Including poor-quality anomalies for training a model is likely to harm its performance (Hendrycks et al. 2019 Qiu al.) 2022b Li
et al. 2023a b). Although Chen et al.) (2021): ;; Ming et. al. (2022) proposed sampling methods for selecting informative anomalies during training, there is approach for quantifying the quality of auxiliary anomalies.
In this paper, we close this critical gap by introducing the Expected Anomaly Posterior (EAP), the first example-wise score function that measures the quality of auxiliary anomalies. Our approach relies on a fundamental insight: high-quality auxiliary anomalies must fulfill two criteria - they must be (1) distinguishable from normal examples in the training data and (2) realistic, iee. similar to the training examples (e.g., scratches only affect few pixels, leaving an anomalous image relatively similar to a normal one). Finding balance between these two characteristics poses an inherent challenge. On one hand, auxiliary anomalies risk deteriorating anomaly detector's 's performance if they closely resemble normal examples. On the other hand, they become less useful the more dissimilar from the training data.
Building upon this insight, we adopt a Bayesian framework to model the uncertainty of a detector's prediction. This framework accounts for both an example's dissimilarity from the normal class (via class-conditional probability) and its realism (via example density). More specifically, the class-conditional probability reflects how good detector is at discriminating the synthetic anomaly from the normal class, while the example density reflects how well the example fits within the overall data distribution. The expectation of the posterior probability that an example is anomalous rellects our concept of the quality of auxiliary anomaly: the approximation we derive in Section 3 to compute the EAP will give lower scores to indistinguishable and unrealistic anomalies.
In sunmary, we make three following contributions.
In Section we compute the expected anomaly posterior (EAP), which measures the quality of anomaly by accounting both for aleatoric and epistemic uncertainty.
In Section 4 we provide theoretical analysis of EAP, including its properties and guarantees.
In Section 5 we run an extensive experimental analysis and show that EAP enhances detector's
performance when using high-quality anomalies to enrich training or perform model selection.
2 Related Work
Anomaly detection. Designing an anomaly detector requires developing a way to assign real-valued anomaly scores to the examples (Chandola et al. 2009 Han et al. 2022b), where the higher the score, the more anomalous the example. Existing approaches often rely heuristic intuitions about expected anomalous behavior (Pang et al. 2021 Qiu 2023). Propagation-basce detectors, such as those using proximity to training examples, assume similar instances share the same label (e.g., SSDO) /Vercruyssen et al. 2018 Loss-based detectors, on the other hand, learn a decision boundary (e.g., a hypersphere over normals and
assign scores based on the distance to this boundary (Ruff et al. 2020 Zhou al.) 2021 Gao et al. 2021
Qiu et al. (002a)). Self-supervised detectors learn models through solving auxiliary tasks and score anomall lies according model performance on self-supervised tasks (Golan & Yaniv
2018 Bergman & Hoshen
2020
2

---

Published in Transactions on Machine Learning Research (12/2024)
Qiu et al. 2021p. Recently, foundation models have enabled zero-shot anomaly detection Jeonn al. 2023 which overcomes the need to collect anomalies for training but still requires their use in model selection (Fung et al. 2023
Uncertainty quantification. Uncertainty quantification is the task of measuring the reliability of pre- dictions in machine learning models (Hüllrmeeerr & Waegeman 2021) It typically considers two types of uncertainty: the aleatoric uncertainty, which indicates randomness in the data such as overlapping classes, and the epistemic uncertainty, which reflects the model's lack of knowledge due to limited or biased training data. Bayesian frameworks are commonly used to capture both uncertaintics by leveraging their estimated posterior distributions (Bengs et al.] 2022). Recent works have introduced advanced methods, such as posterior networks (Charpentier et al. 2020) and second-order scoring rules (Bengs al. 2023). to refine uncortainty estimates and mitigate limitations in traditional approaches. Such rescarch has shown good
results in applied tasks, such as out-of-distribution detection (Ming et al. 2022) and model calibration (Deng
ct al. 2022 leading to robust tools for tasks that require reliable prediction under uncertainty.
Data quality. Traditional quality score functions evaluate training examples by (1) defining a utility function that takes input subset of the training set and measures the performance of the model, and (2) finding function that assigns score to an example by quantifying its impact the model's performance when included/ included/excluuddd excluded for training (Yoon et al. 2020 Sim et al. 2022 Jiang et al. 2023). Methods like
leave-one-out (Loo) iteratively remove one example at time to observe how test performance varies. Various techniques, such as DATASHAP (Ghorbani & Zou 2019), BETASHAP (Kwon & Zou 2021). KNNSHAP (Jia)
et al. 2019), DATABANZHAP (Wang Jia 2023), and AME (Lin et al. 2022), compute the marginal contribution of an example by bootstrapping the training set and assessing its impact on model training. DATAOOB is an out-of-bag (OOB) evaluator that measures out-of-bag accuracy variation. Other methods like LAVA and influence functions (INF) quantify how the utility changes when specific example is more weighted. The Appendix A provides a detailed overview. Unfortunately, all existing data quality estimators focus on evaluating the impact of training examples and are impractical for evaluating auxiliary anomalies for two reasons. First, retraining a detector multiple times is computationally prohibitive (e.g., Loo). In contrast, EAP trains the detector only once. Second, they require an i.idd validation set with the same distribution as the test set to mcasure the performance variation with and without the example, which is challenging in anomaly detection due to the common distribution gap between validation and test H In contrast, EAP is flexible and can operate effectively even in the absence of high-quality validation set by directly leveraging the detector's posterior probabilities. This allows it to evaluate auxiliary anomalies without relying on a separate dataset for comparison, which is especially useful in anomaly detection scenarios where a well-matched validation set may not be available. For completeness, we will empirically compare EAP against the main data quality estimators in the experiments (Section
3 Methodology
In this Section, we rntroe the problem setup and notations (Section 3.1 and describe our proposed approach for quantifying the quality of auxiliary anomalies (Section 3.2
3.1 Problem setup
Let PP) be a probability space, and X: Ω Rd, S - 10, two random variables representing, respectively, feature vectors and class labels (0 for normals, 1 for anomalies) P A training dataset is i..d. sample of pairs D = {x1, ((1i,,))..... (In, }n)} ~ P(X,Y) drawn from the joint distribution. We denote by X(Y the class conditional probability, and by p(x) the class densily. Because of the rarity of anomalies, we assume to have only m << n (labeled) examples from the anomaly class, in addition to n - m (labeled) normal examples.
Anomalics may manifest noveltics out-of-distribution examples, creating distribution shifts the positive class. 2 Anomaly detection often treated binary claselfication task, where anomalies belong to the positive class. See, for example, Chandola al. (2009) for further information.
3

---

Published in Transactions on Machine Learning Research (12/2024)
Since anomalies provide valuable training signals, but are rare to acquire, Henddyykks al. (019 Murase
& Fukumizui (2022) propose using auxiliary anomalies during training. With significant. improvements in generative modeling, there are many candidate methods for generating synthetic anomalies to complement the training data. Our goal is to evaluate candidate synthotic anomalics with a quality score function & such that higher scores indicate that the synthetic anomaly useful for training a detector. Before formalizing our research task, we introduce the following definition.
Definition 3.1 (Categorization of Anomalies). Given the examples an Zn 3y E Rd we define that
ZR. realistic anomaly if it has high conditional probability and non-zero density
P(Y = 1|X = xR) E [0.5, 1] and p(æx) > 0;
Ty is an unrealistic anomaly if it has high conditional probability and null density
P(Y = 1|x z.) E [0.5, 1] and p(xx)) 0; 3y is indistinguishable anomaly if it has mull conditional probability and non-zero density
P(Y = 1|X = xx) = 0 and p(x;) > O.
Strictly speaking, unrealistic anomalies refer lo examples falling outside the support of the real data generation model, i.e. with density equal to zero. Note that these are not merely low-density examples, which can be realistic, as anomalics are naturally rare and often occur in low-density regions. On the other hand, indistinguishable anomalies refer to examples that are confidently classified as normals by the anomaly detector, i.e. with class conditional probability of being anomalous equal to zero.
An anomaly detector is function f: Rd - E that assigns real-valued anomaly score fxx to any a E Rd. The delector f is learned using the training set D. and can be used for estimating the class conditional probability X) by mapping the scores to [0. 1] KKiegel et al. 2011).
Given: with m < n anomalies, a set of auxiliary anomalies {a E Rd). and a detector f;
Challenge: Design quality score function o: Rd - R for the auxiliary anomalics, such that for any realistic anomaly 2n I and any unrealistic indistinguishable anomaly zu. 24 1, the realistic anomaly has a higher quality score ((FR) > ((x:).
With this categorization, estimators for PX) alone cannot differentiate between va and Zu, while estimators for p(X) alone cannot diffcrentiate between Ig and 2, thus not qualifying as good quality estimators. Intuitively, a score must. quantify the conditional probability to distinguish between 2i and x, but the estimate nceds to account for an example's density. Roughly speaking, the lower an example's density the more uncertain the estimate, because the lack of similar training data prevents model from learning the correct probability. This introduces additional level of uncertainty (namely, epistemic), which requires Bayesian perspective to be properly measured (Hillermeier & Waegeman 2021 Bengs et al. 2022
3.2 The Expected Anomaly Posterior
Capturing detector's uncertainty challenging bocause one ncods to account for (1) the example's proximity to the normal class (i.e., the aleatoric uncertainty) and (2) the lack of training data in the region where the example falls (i.e., the epistemic uncertainty). This is particularly complicated in anomaly detection because the epistemic uncertainty tends to be high for most anomalies, as they often fall in low-density regions (Bengs et al. 2023
We propose EAP, a novel approach that estimates the quality of auxiliary anomalies by capturing an anomaly detector's uncortainty. The key idca to model cach example's probability of being an anomaly ag. This captures both types of uncertainty: the one on the class prediction Y e (0, (i.e. alcatoric), and the one on the probability na (Le. epistemic). The quality score we propose is the expected posterior of this parameter.
4

---

Published in Transactions on Machine Learning Research (12/2024)
Assumption. For any r the class conditional distribution Y|X x is a Bernoulli
P(Y x) = (ERNOULLI(TT)), TT BETA (co. So)-
The parameter T can be interpreted as the probability of the example being an anomaly. With the Beta prior we can incorporate prior knowledge such as the expected ratio of anomalies in the data (Perini et al., 2023). Since we have at most one observation of Y for each I, we follow Charpentier et al. (2020) and model the posterior over Ty by conditioning on pseudo observations. Given N pseudo observations "/....,, UN hypothetically drawn from P(Y ø), the posterior,
Tx 01 UN BETA + a1, 30 + N -a1),
(1)
is conjugate to the Beta prior, where an = Ei= ( = 1) is the number of anomalies in the pseudo observations. That is, if we could sample N labels for the same example x, i.e. (z, yr)...., (2. gx), we would derive the posterior distribution of Y|X = x by using simple Bayes update (Eq. 1 However, sampling N labels for the same example is practically impossible. Thus, need to parametrize a1. Roughly speaking, if we drew n training examples from p(X, Y) we would expect to draw N = n p(x)) examples with features x,
among which P(Y = X|X 2) are anomalies:
@1 n P, nx P(Y P\= X
conditional probability
data deusity
where : indicates that the quantity is estimated. We describe how we compute both terms below expectation of the posterior in Equation | reflects the quality of an auxiliary anomaly x: if the evidence N is high, we have enough samples to rely on the expected conditional probability for evaluating the auxiliary anomaly (EAP x P(Y=1[X==)) Note that, while n represents the training set size, it can also be treated as hyperparameter. In this context, it acts like certainty budget distributed over the two classes.
M The
while if N is low, the quality reflects our prior belief (EAP x acth )
Estimating the data density. Computing p(x) has two main challenges. First, most traditional density estimators suffer the well-known curse of dimensionality (Verleysen & François 2005 Bengio et al. 2005
Second, deep estimators (e.g., Normalizing Flows (Kobyzev et al., (2020)) are prohibitively time-consuming to be employed for data quality scores. Thus, EAP relies on the rarity score (Han et al. 2022a),, which is fast to compute and weakly affected by the curse of dimensionality. The rarity score (1) creates k-NN spheres centered on each training example, and (2) assigns the smallest radius of the sphere that contains the given synthetic example. If the synthetic example falls outside of all spheres, it considered too uncommon and gels null rarity.
We use the rarity score T with estimated k to estimate the data density Intuitively, the density behaves as the inverse of the rarity score: highly uncommon examples should have low density. Thus, we take the reciprocal value of the rarily score and normalize it using the training rarity scores:
1/rg(z) p(x) 1xrg(0) + E. 1/r
(3)
Finally, want to highlight that the weak estimator in Eq 3 is enough for our task for two reasons. First, assigning mull density to synthetic anomalies falling outside of all training spheres allows unrealistic anomalies to be detected. Second, the provided implementation is fast, as one can compute the training spheres only once for the whole set of synthetic examples.
Estimating the conditional probability. Computing P(Y 1|X x) in anomaly detection is hard task because (1) class probabilities are generally unreliable for imbalanced classification tasks (Wallace &
"Note that the principle itself is not restricted to this particular choice of cstimators. One could apply other method
without altering the validity of the framework.
We explain how we compute in Appendix A.2

---

Published in Transactions on Machine Learning Research (12/2024)
Dahabreh 2012 Tian al. 2020; and (2) the available anomalics might be non-representative of the whole anomaly class (i.e., we have access to biased set). This makes traditional calibration techniques often impractical (Silva Filho et al. 2023 Deng et al. 2022). However, we mainly care about having probabilities that satisfy two properties. First, they must be coherent with the detector's prediction, namely a predicted anomaly (normal) needs probability greater (lower) than 0.5. Second, we want the proportion of predicted anomalies match the expected proportion of true anomalics. This guarantecs that, if the detector's ranking is accurate, the class predictions are optimally computed.
For this task, we employ squashing scaler /Vereruyssen et al. 2018 to map the anomaly scores f(z) to [0, 1] probability values
P(Y = 1|X = x) - 2 (2))
(4)
where À is the detector's decision threshold, which we set such that the number of training examples with probability > 0.5 (after mapping) is equal to the number of training anomalies m (Perini et al. 2023
Roughly speaking, Eq. uses a monotonic function to map À 0.5, all scores to the interval (0.5, 1 and all the scores < À to 0.5). This guarantees that, when transforming the estimated P(Y = 1|x = x) to predictions by thresholding at 0.5, the predicted classes are the same as when thresholding scores with the decision threshold X. This transformation satisfies both properties listed above.
Computing the quality scores. Using our point estimates for the data density and the conditional probability, the parametrized posterior distribution Ta ..r...... UN can be computed by substituting to Eq.
01 P(Y 1|X 2) p(z) and = n p(z).
Finally, we compute the quality of an auxiliary anomaly æ by taking its expectation
<(z) TT Dn UNl + n. p(x) P(Y = 1|X =x)
an Bo + p(x)
4 Theoretical Analysis of EAP
We theoretically investigate two tasks. First, we illustrate the main properties of EAP, namely how it behaves when subject to (1) large training sets (n + +00), (2) small training sets or zero-density examples, (3) high-class conditional probabilities. Second, we answer the following question: Given a realistic anomaly
Yas on indistinguishable anomaly Vs and an unrealistic anomaly By as in Definition 3.1 does EAP runk ((xx) > ((z,), ?
i) EAP has three relevant properties:
P1) Converyence class conditional probabilities. The number of training examples indicates how strong the empirical evidence is. That is, the detector f has enough evidence to estimate properly the class conditional probability. Thus, for high n, EAP converges the class conditional probability
xxx)= + P(Y = =x for - +00;
P2) Conwergence to the prior's mean. No empirical evidence implies that the posterior remains equal to the prior. Thus, EAP assigns the prior's mean for relatively small n or null-density region,
On
((x)
for n + 0 or p(x) - 0.
ao + 90
P3) The quality of distinguishable anomalies increases with their densily. Given 1[y=1[==) & for an example x, its quality depends only on its density: the dloser/mors similar the training examples, the higher the density, the higher its quality:
8
6(z) 11
On + 30 + n p(x)

---

Published in Transactions on Machine Learning Research (12/2024)
Roughly speaking, assuming that the syuthetic anomaly is distinguishable for the anomaly detector from normal counterpart, the closer it is to the training examples (normal anomalous) the more likely it will help to improve the decision boundary when used for training.
ii) EAP" ranking guarantee.
We show that EAP ranks the anomalies as (1st) realistic, (2nd) unrealistic, and (3rd) indistinguushabll. Theorem 4.1. Let Ba Do, R be, respectively, a realistic, unrealistic, and indistinguushable anomaly. 1f the estimators in Eq. 3 and Eq. 4 satisfy the properties of Definition 3.1 then
0g0
0.5 <(xx) Φ(x,) 6(x,).
(5)
ag + 9g
Proof. Using the definition of indistinguishable and unrealistic anomaly, we immediately conclude
o(xu)
0k0
30 p(.pp())) ((x))
ong 3o
0g
because p(z;) 0. As a second step, we assume that
oa tot
< 0.5 and show algebraically that
au + n P(Y 1 = ØR) p(zR)
C >
Φ(xu) > 6(zu) > 6(œn) - o >
ag + 8o + p(5n))
080 + 80
= = n ((FR) I(00 + 30) P(Y = = mR) - aul >0 0 = P(Y = 1|X = 2x)
00
+ 3g
which holds as P(Y= 1|X xR) > 0.5 +80
5 Experiments
We empirically evaluate two aspects of our method EAP: (a) whether it measures properly the quality of auxiliary anomalies, and (b) its impact on selecting auxiliary anomalies for learning a model or for model selection. For the first aspect, we compare EAP's ability to rank high-quality examples above low-quality ones. For the second aspect, we evaluate the downstream impact including high-quality samples in training or model selection, measuring how these examples improve the performance of the learned model. We argue that, together, these evaluations illustrate both the standalone utility of EAP a quality metric and its practical relevance for improving anomaly detection. To this end, we address the following five experimental questions:
Q1. How does EAP compare to existing methods at assigning quality scores?
Q2. How does a model's performance vary when including high-quality anomalies for training?
Q3. How does a model's performance vary when including low-qualitty anomalies for training?
Q4. How does the performance of a CLIP-based zero-shot anomaly detection method vary when using
the selected auxiliary anomalies for prompt tuning?
Q5. How do EAP scores vary for different priors?
5.1 Experimental Setup
Baselines. We compare EAP with 12 adapted baselines: Loo, KNNSHAP Jia et al. 2019). DATA-
BANZIAF (Wang & Jial 2023), A.E Lin et al. 2022). LAVAEV (Just et al. 2023 INF (Feldnan & Zhang 2020), and DATAOOB (woon & Zou 2023) are existing data quality evaluators that measure the impact of a training example on the model performance. We adapt these methods by including each auxiliary anomaly (individually) in the training set and evaluating its contribution. RANDOMEV assigns uniform random scores to each auxiliary anomaly. RARITY (Han et al. 2022a) computes the rarity score of each auxiliary anomaly. Finally, we include the estimators for the data density P,, the class conditional probability Pyzz and linear combination of them Pyle NP,,
Code available https://githhe wwtp://wwaa..eeee//oooccccaaaaaa

---

Published in Transactions on Machine Learning Research (12/2024)
Data. We carry out our study on 40 datasets, including 15 widely used benchmark image datasets (MvTEa EC) Bergmann et al. 2019), 3 industrial image datasets for Surface Defect Inspection (SDT) (Wang) et al. 2022). and an additional 22 benchmark tabular datasets for anomaly detection with semantically useful anomalics, commonly referenced in the literature (Han et al.) 2022b). These datascts vary in size, feature count, and anomaly proportion.
For each dataset, we construct an auxiliary set of anomalies by combining realistic, indistinguishable, and unrealistic anomalies (3 each). Realistic anomalies are labeled anomalies provided with the dataset, indistinguishable anomalies are labeled normal examples with swapped labels, and unrealistic anomalies come from other datascts. Specifically, to collect unrcalistic anomalies we randomly select 5 datasots out of 40, subsample them to specific count, and fix their dimensionality to d via random projections (either extending or reducing it). sreudo-uuaiity labels "good" and "poor" are assigned to real anomalies and the other two groups, respectively, reflecting the ground truth where real anomalies should have high-quality scores.
Sctup. For each dataset, we procood as follows: (i) We create a balanced test set by adding random normal examples and 50% of available anomalies; (ii) We generate a set of l auxiliary anomalies as described above with 5 = 40% of available anomalies; (iii) We create a training set by adding 10% of available anomalies and all remaining normal examples to the training set. (iv) We apply all methods to evaluate the external set of anomalies, using the training set for validation when required (as m < n, we avoid partitioning the training set); To mitigate noise, steps (i)-(iv) are repeated 10 times with different seeds, resulting in a total of 4000 experiments (datasets, methods, seeds). While computing EAP is fast, the baselines have high computational costs because they train a model several times. To run all experiments, we use an internal cluster of six 24- or 32-thread machines (128 GB of memory). The experiments finish in 72 hours.
Models and Hyperparamotors. For all basclines, we use SSDO (Vercruysson et al. 2018 the underlying anomaly detector ʃ with k= 10 and Isolation Forest (Liu et al. 2008) prior. This choice is motivated as follows. First,, such a combination has been analyzed and used often by researchers (Drogkoula et al. 2023 Stradiotti et al. 2024a [erban et al. 2024 Pang et al. 2023 Stradiotti et al. 2024b). Second, because some of our datasets include tabular data, we need to employ a fast yet accurate detector for such a data modality. Recent papers such as (Stradiotti et al.] 2024bb highlight that SSDO + Isolation Forest is one of the best-performing detectors. When exposed selected auxiliary anomalies, we employ an SVM with RBF kernel (for images) and RANDOM FOREST (for tabular data) to make the normal vs. abnormal classification. For images, we use the pre-trained VrT-B-16-SIcH the features from images and use them as inputs to FAP and all baselines. Our method EAP has one hyperparameter, namely the prior ao, So, which we set to m2 (the proportion of anomalies in the training set) and 1 m, Intuitively, this corresponds to the expected proportion of (real) anomalies an external datasct was sampled from P(X,Y). The baselines have the following hyperparameters: KNNSHAP and RARITY have l = 10, DATABANZHAF, AME, INF and DATAOOB use 50 models. All other hyperparamoters are set as default (Soenen et al. 2021
(Zhai et al 2023) to extract
Evaluation Metrics. We employ four evaluation metrics. First, we use the Area Under the Receiving Operator Curve (AUCqtt) to evaluate the methods' ability to rank good-quality examples higher than poor-quality ones based on quality scores compared to the pseudo-quality labels. Second, we qualitatively analyze the impact of using auxiliary anomalies in training a model, showing the learning curves (LCg) with the number of added anomalies following the ranking of quality scores on the x-axis and the test accuracy on the y-axis. We compute the area under the learning curve up to Y of ranked anomalies (AULC,) and the test accuracy after including top 1/2 of ranked anomalies (ACCg). Similarly, we compute the LCp following the methods' inverse ordering and measure the AULC, of including up to 3/2 of inversely-ranked anomalies, where lower values are desirable. Finally, separately for each metric, we rank all methods from the best (position 1) to the worst (position 13) in each experiment. By monitoring the rankings, we get insights on "how often" our method outperforms the competitors, as opposed to the traditional "by how much". We denote the ranking- based motrics by adding a r in front in Table 1
6Code: http://tthbbbccmm/opeenaaaaal
8

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Summary of the results obtained by the methods over image (above) and 22 tabular (below) datascts. Columns 6 10 show the ranking values for the metrics employed (columns 2 5) and their average. For metries desiring lower values, we mark with a 4. Overall, EAP achieves the best performance and ranking position for most evaluation metrics as well the best avg. ranking.
Image Datasets
EVALUATOR
qUCqbe
2 g
ACC₂
(ULC,(4)
qu
rAULC
rACC
rAULC,
Avg. Rank
EAP
0.803
0.717
0.833
0.688
1.99
3.94
3.12
3,66
3.18
RARITY
0.681
0.698
0.786
0.738
4.21
5.07
5.34
7.96
5.65
LAVA
0.742
0.665
0.755
0.709
2.70
8.17
7.44
4.71
5.75
Pule NP.
0.700
0.795
0.729
4.80
5.61
5.79
LOO
0.537
0.693
0.777
0.694
7.22
6.56
6.14
6.44
RANDOMEV
0.491
0.696
0.793
0.756
8.92
5.93
9.90
7.38
DATAOOD
0.505
0.685
0.739
0.668
8.66
7.53
9.55
4.35
7.52
RNNSHAP
0.509
0.695
0.794
0.754
8.58
6.36
5.95
9.44
7.58
AME
0.512
0,693
0.793
0.756
8,56
6.42
5.85
9.94
7.69
INF
0.488
0,681
0.778
0.744
8.87
7.01
6.73
8.55
7.79
PaIn
0.494
0.671
0.710
0.644
9.30
8.55
10.62
3.04
7.88
DATABANZHAP
0.500
0.668
0.775
0.748
8.71
6.68
8.54
7.98
0.535
0.600
0.733
12.37
11.72
7.72
22 Tabular Datasets
EAP
0.821
0.779
0.839
0.717
1.91
4.27
4.28
2.11
3.14
RARITY
0.724
0.782
0.839
0.753
3.59
4.42
4.91
4.92
4.46
LAVA
0.723
0.744
0.794
0.747
3.62
7.48
7.17
4.08
5.59
Pate + N.,
0.676
0.758
0.815
0.771
4.93
6.16
6.23
6.16
5.87
KNNSHAP
0.541
0.770
0.825
0,805
7.61
5.24
5.29
9.15
6.82
RANDOMEY
0.502
0.773
0.827
0.809
8.25
4.92
5.33
9.72
7.06
AME
0.498
0.772
0.827
0.809
8.53
5.05
5.37
9.75
7.17
Loo
0.504
0.750
0.798
0.792
7,90
6.87
7.10
7.77
7.41
P.
0.554
0.703
0.768
0.752
6.77
9.29
7.78
Pole
0.533
0.712
0.748
0.753
8.20
10.56
10.66
4.30
8.43
DATAOoB
0.513
0.729
0.768
0.785
8,48
9.39
9.84
7.43
8,78
INF
0.434
0.742
0.795
0.816
10.29
7.78
7.37
10.40
8.96
DATABANZHAF
0.415
0.736
0.789
0.817
10.92
8.47
8.16
10.53
9.52
5.2
Experimental Results
Q1. EAP vs basclines assigning quality scores. Figure shows the methods' mean AUC on both image (Ieft) and tabular data (right). On images, EAP outperforms all baselines on 13 out of 18 datasets, achieving average AUCqur significantly higher than LAVa and RARITY by 6 and 12 percentage points, shown in Table [ Also, EAP consistently obtains the lowest average ranking positions (1.99 for rAUCqur). On tabular data, EAP obtains an average AUCqı 0.821, which is around 10 percentage points higher than the runner-ups. Also, EAP outperforms LAVA and RARTTY on 18 and 17 datasets and obtains the best. average ranking (1.91).
Interestingly, only EAP, LAVA, RARITY, and Pyla NP, achieve performance better than random, while other methods get average AUCar around 0.5, which highlights their inability to distinguish good and poor auxiliary anomalies consistently. As second remark, EAP performs lower than random for the dataset TILE. This happens because the defective images extremely different than the normal images, thus resulting in real anomalies falling in zero-density regions, which our method would categorize as unrealistic.
Q2. Including high-quality anomalies in training. We measure how the performance of model varies when introducing the top & of auxiliary anomalies into the training following the methods' rankings. Figure 2 (top) shows the learning curves (LCa) ) for four representative image datascts. Overall, including high-ranked anomalies first. has the claimed impact on the model's performance: the learning curve grows sooner for EAP compared to all baselines. Consequently, EAP obtains an AULC, that is, on average, higher than all the basclines by between (vs. RARITY and Pylz + NP.) and (vs. LAvA) percentage points. After including thir of the auxiliary anomalies for training, EAP shows an average improvement on the test performance (ACCa) of 4 to 10 points all baselines, standing out the only method significantly better

---

Published in Transactions on Machine Learning Research (12/2024)
EAP
Raifty
X
Lava
KNNShhap
Loo
AME
DataBanzhar
DataOob
Randomēv
Px
Pyjk
Pype+
image Data
Tabuler
Data
0.7
0.6
0.5
0.4
0.3
SDLB
grid
A
screw
transis
metairu
zipper
ccntttirr
SDI
hazeiru
botitle
canle
palr
leatrier
word
cazoute
carpat
nile
cover
penelgg
optcly)
doners
celeti
canto
shurtle
breadt
wareeer
Cardlet
annttor
campaig
tondsgeh
SpanBar
cenous
sutell!
froud
fuut
rima
yeast
lansiat
Datasets
Figure 1: The plot illustrates the average AUC our obtained by each method on per-datasct basis (left for image data, right for tabular data). EAP achieves the highest performance for most datasets, beating the runner-ups RARITY and LAVA respectively, 30 and 31 datasets out of 40.
EAP
Parity
KNNShap
Datallanchal
DataDob
Inf
Random
Py|x+NPx
1.0
High- Quality First
Test Accuracy
0.9
XCCC
JACC
metalnut
PiI
screw
Gle
Quality Frrt
Test Accuracy
0.7
0.6
30
6.0
90
60 90 120
150
30
90 120
30 60
90
# Training Auxiliary Anomalles
Figure 2: Learning curve (LC) obtained by following the method's ordering (top) and inverse ordering (bottom) for four representative image datasets. Top: EAP' LCg grows sooner (i.c., better) than the other methods', which confirms that including high-quality anomalies in the training set has a larger impact on the test performance. Bottom: EAP LCp rises later (i.e., better) than most baselines', showing that low-quality anomalies have a comparatively modest impact on the test performance.
than RANDOMEV. Moreover, Table shows that EAP achieves the best average ranking for both AUTCa (i.e., 3.94) and ACC, (i.e., 3.12).
On tabular data, Table [ shows that EAP achieves an average AULC, slightly lower than RARITY (0.779 0.782) and similar ACC, (both 0.839). This occurs because, for most tabular datasets, RARITY assigns high-quality scores to the unrealistic anomalies which are obviously different than normal data. When including them for training, it yields an improvement of the RANDOM FOREST'S test accuracy surprisingly From a ranking perspective, EAP obtains the best average with rAULC, 4.27 (vs. RARITY'S 4.42), and rACC, = 4.28 (vs. RARITY'S 4.91).
Q3. Including low-quality anomalies in training. Figure 2 (bottom) shows the LCp obtained by following an inverse ordering the methods, ie., lower ranked anomalies are included first. Using this inverse ordering should result in much slower growth of the LCC,s: in some cases, the test accuracy using EAP remains stable (METALNUT, SCREw), while in others shows strong fluctuations going up and down quickly (Pm..). Interestingly, every baseline's performance goes up for TILE: this is due to their poor ability to assign
10

---

Published in Transactions on Machine Learning Research (12/2024)
Table 2: Test AUCs (%) of prompt tuning with auxiliary anomalies selected by each baseline on MvTEC.
EAP performs the best on 12 of 15 classes and achieves the highest average AUC.
Evaluator
bottle cable capsule carpet grid
hazel leather
metal
pill
screw
tile
tooth
transis
wood
zipper
avg
KNNSHAP
82.4
56.7
51.4
58.8
72.7
92.1
51.7
49.1
60.2
77.0
77.1
72.7
44.7
66.5
AME
82.4
56.7
51.4
82.1
71.2
72.7
98.4
51.7
49.1
60.2
69.2
75.4
68.3
72.7
44.7
67.1
LOO
82.4
60.7
64.4
85.8
58.8
68.2
98.8
51.7
49.1
60.4
69.2
77.0
62.1
72.7
44.7
67.1
DATAOOB
82.4
60.7
63.7
85.8
71.2
72.7
98.8
51.7
49.1
60.4
69.2
75.4
62.1
72.7
44.7
68.0
Pyle
82.4
60.6
51.2
85.8
71.2
68.2
98.4
56.2
65.6
60.4
75.4
62.1
72.7
44.7
68.3
INF
95.3
56.7
45.7
96.1
83.6
68.2
98.8
51.7
65.7
60.4
69.2
75.4
62.1
93.0
75.9
73.2
DATABANZ
95.3
41.1
64.4
96.1
83.6
72.7
98.8
51.7
65.7
60.4
84.9
75.4
62.1
93.0
75.9
74.7
RANDOMEV
95.3
69.2
63.7
96.1
83.1
68.2
98.8
56.2
65.7
60.4
87.8
75.4
62.1
96.4
75.9
76.9
RARITY
94.3
69.2
64.4
96.1
83.6
83.8
98.8
45.1
63.5
60.4
91.8
75.4
77.1
96.4
75.9
78.4
P.
94.3
70.2
64.6
83.6
76.7
86.9
90.8
63.5
60.4
84.9
75.4
77.1
93.0
75.9
79.6
LAVA
94.3
70.3
64.4
96.1
83.6
83.8
98.4
90.9
63.5
60.0
87.8
75.4
77.1
93.0
75.9
80.9
Pale + N..
94.3
70.2
64.6
96.1
92.6
83.8
94.4
90.8
63.5
60.4
87.5
77.0
77.1
93.0
75.9
81.4
EAP
70.3
64.4
92.6
83.8
98.8
90.9
63.5
64.8
77.0
77.1
93.0
75.9
82.4
Table Comparison between EAP with default og and six variants EAP, that set og = w. 3g Rankings show low sensitivity such a choice, long ag < 0.4.
w0
EVALUATOR
AUCqut
rACC,
Avg. Rank
EAP
0.811
2.92
2.73
2.83
EAP
0.812
2.78
3.25
3.02
EAPo.3
0.810
3.39
3.40
3.40
EAPo.1
0.810
3.01
3.46
3.24
EAPo.05
0.808
3.63
3.78
3.71
EAP0.01
0.805
4.10
3.54
3.82
EAP0.4
0.800
4.71
4.79
4.75
scores for this dataset, as described in Q1. Surprisingly. DATAOOB obtains the lowest AUTCr, while FAP has the second best AULC, with just two percentage points as gap. However, when ranking the experiments, EAP achieves the best average ranking (3.66 of rAULC,), thus being the preferred method for most of the experiments.
On tabular data, the results confirm the previous analysis: EAP has the lowest AULC, on 137 experiments out of 220, while RARITY and LAVa achieve so only on, respectively, 29 and 31 experiments. This motivates that EAP obtains average AULC, that is better than all baselines by between 3 and 10 percentage points, as shown in Table 1
Q4. Prompt tuning for zero-shot anomaly detection. CLIP-based anomaly detection methods save the effort of collecting training examples and enable a zero-shot anomaly detection (Jeong et al. 2023 However, their detection performance depends on the choice of prompts, which is usually tuned by using labeled real-world anomalies, We study the impact selected auxiliary anomalies on prompt tuning for the MyTEC datasets. Specifically, we search a prompt for each object class achieving the best performance on the selected auxiliary anomalies from pool of 27 candidate prompts (see details in Appendix A.3 and apply the best-performing prompt to CLIP at test time.
Table 2 reports the test AUCs of CLIP with the best-performing prompts selected by each data valuation method We can see that EAP performs the best on 12 of 15 classes and achieves the highest AUC averaged over all classes. Thanks to accounting for both the class conditional probability and the data density, FAP clearly outperforms RARITY and Pa, which only consider the data density, Pyirs which only considers the class conditional probability, and their naive linear combination Pyzz + N The results confirm that EAP selects high-quality auxiliary anomalies for the model selection purpose. We list the prompts selected by EAP in Appendix A.3
Most values identical hecause the haselines often select the same prompt from our discrete of options.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Q5. EAP sensitivity to +o%% EAP requires two hyperparameters: du, Sn of the prior Beta distribution. To remove one degree of freedom, we set ag + P 1 such that the dataset size n is much stronger (n times) than our initial belief. Then, we investigate how varying the parameter on E [0, 0.5) impacts EAP's overall performance (Bu = 1 ag). We compare seven versions of our method by setting ao E {0.01, 0.05, 0.1, 0.2, 0.3, 0.4), in addition to the original EAP that leverages the contamination level og m/n. We call EAP.' the variant that uses ao = w. Table 3 shows the rankings of these 7 variants for AUCqur and ACC, and their average Overall, both parameters have a low impact on our method: while a higher value for ag improves the AUCqur in some experiments this improvement does not yield better performance at test time terms of ACCa. Morcover, sctting on too high or too low has inherent risks: EAP and EAP are the worst variants by lar, with siguificant drops in performance compared to the other variants.
6 Conclusion
This paper addressed the problem of evaluating the quality of an auxiliary set of synthetic anomalies. With this quality score, one can enrich an anomaly detection dataset to learn more accurate anomaly detector. We proposed the expected anomaly posterior (EAP), the first quality score function for auxiliary anomalics derived from approximation for the posterior over the probability that a given input is an anomaly. We showed that our approach theoretically assigns higher scores to the realistic anomalies, compared to unrealistic and indistinguishable anomalies. Empirically, we investigated how EAP compares to adapted data quality estimators at (1) assigning quality scores, (2) using such scores to enrich the data for training, and (3) model selection. On 40 datasets, we show that EAP outperforms all 12 baselines in the majority of the cases.
References
Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of dimensionality for local kernel machines. Techn. Rep, pp. 1, 2005.
Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. Advances in Neural Information Processing Systems, pp. 29205 29216, 2022.
Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. On second-order scoring rules for epistemic uncertainty quantification. In International Conference on Machine Learning. PMLR, 2023.
Liron Bergman and Yedid Hoshen. Classification-based anomaly detection for general data. In International Conference on Learning Representations, 2020.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the Conference on Computer Vision and Pottern Recognition (CVPR), pp. 9592-9600, 2019.
Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlogger, and Carsten Steger. Beyond dents and scratches: Logical constraints in unsupervised anomaly detection and localization. International Journal of Computer Vision, pp.
2022.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly Detection: A survey. ACM computing surveys (CSUR), pp. 1 1-58, 2009.
Bertrand Charpentier, Daniel Züggerr, and Stephan Günnemann. Posterior network: Uncertainty estimation without ood samples via density-bace based pseudo-counts. Advances in Neural Information Processing Systems, pp. 1356 1367, 2020.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of-distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Balbao, Spain, September 13-17, 2021, Proceedings, Part III 21, pp. 430 445. Springer, 2021.
We include AUCaur as the first column for the sake completeness.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Ailin Deng, Adam Goodge, YA Lang, and Bryan Hooi. Cadet: calibrated anomaly detection for mitigating hardness bias. In International Joinl. Conference on Artificial Intelligence, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, pp. 8780 0094,,94,44 2021.
Maria Drogkoula, Konstantinos Kokkinos, and Nicholas Samaras. A comprehensive survey of machine learning methodologics with emphasis in water resources management. Applied Sciences, pp. 12147, 2023.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estination. Advances in Newral Information Processing Systems, pp. 2881 2891, 2020.
Clement Fung, Chon Qiu, Aodong Li, and Maja Rudolph. Model selection of anomaly detectors in the absence of labeled validation data. arXiv preprint arxiv:2310. 10460,, 2023.
Feng Gao, Jing Li, Ruiying Cheng, Yi Zhou, and Ying Ye. Connet: Deep semi-supervised anomaly detection based on sparse positive samples. IEEE Access, pp. 67249 67249-67258, 2021.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pp.
PMLR, 2019.
Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. In Aduances in Neural Information Processing Systems, pp. 9758 9769, 2018
Jiyeon Han, Hwanil Choi, Yunjey Choi, Junho Kim, Jung- Woo Ha, and Jaesik Choi. Rarity score: A new metric to evaluate the uncommonness of synthesized images. Proceedings of the International Conference on Learning Representations, 2022a.
Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. In Advances in Neural Information Processing Systems, pp. 22142-32159) 2022b.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Representations, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Newral Information Processing Systems, pp. 6840 6851, 2020.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, pp. 457-506, 2021.
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winelip: Zero-/few-shot anomaly classification and segmentation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19606 19616, 2023.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. Efficientt task-specific data valuation for nearest neighbor algorithms. Proc. VLDB Endow., pp. 1610-1623, 2019.
Kevin Fu Jiang, Weixin Liang. James Zou, and Yongchan Kwon. Opendataval: unified benchmark for data valuation. Advances in Newral Information Processing Systems Datasels and Benchmarks Track, 2023.
Hoang Anh Just, Feiyang Kang, Jiachen Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. Proceedings of the International Conference on Learning Representations, 2023.
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, pp. 3964-3979, 2020.
Hans-Peter Kriegel, Peer Kroger, Erich Schubert, and Arthur Zimek. Interpreting and unifying outlier scores. Tn International Conference on Data Mining, pp. 13 24, 2011.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Yongchan Kwon and James Zou. Beta shapley: unified and noise-reduced data valuation framework for machine learning. International Conference on Arlificial Intelligence and Stalisties, pp. 8780-8802. 2021.
Yongchan Kwon and James Zou. Data-oob: Out-of-bag estimate simple and efficient data value. In International Conference on Machine Learning. PMLR, 2023.
Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Stephan Mandt, and Maja Rudolph. Deep anomaly detection under labeling budget constraints. In International Conference on Machine Learning, pp. 19882-19910. PMLR, 2023a.
Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt. Zero-shot anomaly detection via batch normalization. In Advances in Newral Information Processing Systems, 2023b.
Jinkun Lin, Anqi Zhang, Mathias Lécuyer, Jinyang Ti, Aurojit Panda, and Siddhartha Sen. Measuring the effect of training data deep learning predictions via randomized experiments. In International Conference on Machine Learning, pp. 13468 13504. PMLR, 2022.
Fci Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth icee international conference data mining, pp. 413 422. IEEE, 2008.
Luis Marti, Nayat Sanchez-Pi, José Manuel Molina, and Ana Cristina Bicharra Garcia. Anomaly Detection based sensor data in petroleum industry applications. Sensors, pp. 2774-2797, 2015.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: distrr-dit detection with posterior sampling. In International Conference on Machine Learning, pp. 15650 15665. PMLR, 2022.
Hironori Murase and Kenji Fukumizu. Algan: Anomaly detection by generating pscudo anomalous data via latent variables. IEEE Access, pp. 14259-44277 2022.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: review. ACM computing surveys (CSUR), pp. 1-38, 2021.
Guansong Pang, Chunhua Shen, Huidong Jin, and Anton van den Hengel. Deep detection. Tn Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1795 1807, 2023.
anomaly
Lorenzo Perini, Vincent Vercruyssen, and Jesse Davis. Transferring the contamination factor between anomaly detection domains by shape similarity. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4128 4136, 2022.
Lorenzo Perini, Paul-Christian Bürkner, and Arto Klami. Estimating the contamination factor's distribution in unsupervised anomaly detection. In International Conference on Machine Learning, pp. 27668-27677 PMLR, 2023.
Chen Qiu. Self-Superuised Anomaly Detection with Neural Transformatioss. PhD thesis, Technische Universität AisserslauternLanddae 2023.
Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. Neural transformation learning for deep anomaly detection beyond images. In International Conference on Machine Learning, pp. 8703 8714. PMLR, 2021.
Chen Qiu, M. Kloft, Stephan Mandt, and Maja Rudolph. Raising the bar in graph-level anomaly detection. In International Joint Conference on Artificial Intelligence, 2022a.
Chen Qiu, Aodong Li, Marius Kloft, Maja Rudolph, and Stephan Mandt, Latent outlier exposure for anomaly detection with contaminated data. In International Conference on Machine Learning, pp. 18153-18167. PMLR, 2022b.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. Proceedings of the International Conference Learning Representations, 2020.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Codruta Maria Serban, Gheorghe Sebestyen, and Anca Hangan. Anomaly detection in water consumption patterns using prediction and clustering approaches. In 2024 IEEE International Conference on Aulomation, Quality and Testing, Robotics (AQTR), pp. 1 6. IEEE, 2024.
Telmo Silva Filho, Hao Song, Miquel Perello-Nicto, Raul Mantos-Rodrigueet Meelis Kull, and Peter Flach. Classifier calibration: survey on how to assess and improve predicted class probabilities. Machine Learning, pp. 1 50, 2023.
Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine learning: "ingredients", strategies, and open challenges. In International Joint Conference Artificial Intelligence, pp. 5607-5614, 2022.
Jonas Soenen, Elia Van Wolputte, Lorenzo Perini, Vincent Vercruyssen, Wannes Meert, Jesse Davis, and Hendrík Blockeel. The effect of hyperparameter tuning on the comparative evaluation of unsupervised anomaly detection methods. In Proceedings of the KDD 21 Workshop on Oullier Delection and Descriplion, pp. 1-9, 2021.
Luca Stradiotti, Lorenzo Perini, and Jesse Davis. Combining active learning and learning to reject for anomaly detection. In 27th European Conference on Artificial Intelligence, 19-24 October 2024, Santiago de Compostela, Spain Including 13th Conference on Prestigious Applications of Intelligent Systems (PAIS 2024), pp. 22662226622222 2273, 2024a.
Luca Stradiotti, Lorenzo Perini, and Jesse Davis. Semi-supervised isolation forest for anomaly detection. In Proceedings of the 2024 STAM International Conference on Data Mining (SDM), pp. 770-678. STAM, 2024b.
Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re-calibration for imbalanced datascts. Advances in Newral Information Processing Systems, pp. 8101-8113, 2020.
Vincent Vercruyssen, Wannes Meert, Gust Verbruggen, Koen Maes, Ruben Baumer, and Jesse Davis. Semi-supervieed anomaly detection with an application to water analytics. In ICDM, pp. 527-536, 2018.
Michel Verleysen and Damicn François. The curse of dimensionality data mining and time series prodiction. In International Work-Con Conference on Arlificial Neural Nelworks, pp. 758 770. Springer, 2005.
Byron C. Wallace and Issa J Dahabreh. Class probability estimates are unreliable for imbalanced data (and how to fix them). In 2012 TEEE 12th international conference on data mining, pp. 695-704. IEEE, 2012.
Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 6388 6421. PMLR, 2023.
Ruyu Wang, Sabrina Hoppe, Eduardo Monari, and Marco Huber. Defect transfer gan: Diverse defect senthesis for data augmentation. In 33rd British Machine Vision Conference 2022, 2022.
Ruyu Wang, Sabrina Schmedding, and Marco F Huber. Improving the effectiveness deep generative data. In Proceedings of the Winter Conference on Applications of Computer Vision, pp. 1922-4933 2024.
Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pp. 10842 10851. PMLR, 2020.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, In Proceedings of the International Conference Compuler Vision (ICCV), 2023.
Yingjie Zhou, Xucheng Song, Yauru Zhang, Fanxing Liu, Ce Zhu, and Lingqiao Liu. Feature encoding with autoencoders for weakly supervised anomaly detection. IEEE Transactions on Neural Networks and Learning Systems, pp. 2454-2465, 2021.
15

---

Published in Transactions on Machine Learning Research (12/2024)
A Appendix
A.1 Data quality estimators.
Any data quality estimator can be seen as a mapping that assigns scalar score to any example (2, y). Such a score quantifies the impact of (z, y) on the model's performance when trained including the example in the training set. For this task, they introduce a utility function (D) = PERF(f, D) that takes as input subset. D of D and measures the performance of f when trained on it. Next, we briefly describe the existing data quality estimators employed in the experiments and refer to (Jiang al. 2023) for additional details.
LEAVE ONE OUT (T.00) is defined as (2 y) = U(D) - ((x,))())))))))))) where U is commonly chosen as the accuracy:
DATASHAP generalizes Loo's approach to the concept of marginal contributions, which measures the average change in utility when (z, y) is removed from any training set. Given a training set cardinality I N, the marginal contribution is defined
M,(x,y)
N (N
1
E ((.))) U(D,)
DjCD.1D,1=j-1
where D, is random subsct D cardinality j --1 that does not contain (x, v). Then, DATASHAP Ghor
bani & Zou 2019 computes the score as @D y) N E My(z,
BETASHAP Kwon & Zou 2021 generalizes DATASHAP by considering a weighted average of marginal -= (x, v), for some weights wI....,WN.
contributions @BeTASHAP (z.
DATABANZHAF (Wang & Jia 2023
exploits the same formulation as BETASHAP but sets the weights to
2- (N-1)
AME Lin et al. 2022 shows that the average marginal contribution taken over random subsets of D can be efficiently estimated by prodicting the modcl's prediction. They employ LASSO regression model that minimizes
N
arg min (U(D) ((())) + TERN
Σ hil.
where 1(D) is the multi-dimensional characteristic function, D is a random subset draw the data distribution, X is the regularization parameter, and g: 1) + T is a predefined transformation. The values % represent the quality of (zi, 9:).
kNNSHAP (Jia et al. 2019) differs from DATASHAP on the choice of the utility function:
Noat
U(D) Neath (3,9))EN(,,
Σ
1 =)),
where A is the number of neighbors, Neat is the size of the validation set, and M(x; D) indicates the set. of nearest neighbors for the validation example I; over the subset D. Roughly speaking, it measures the proportion of examples in D that are neighbors of 2; and share the same label y4-
INFLUENCE FUNCTIONS (INF) (Feldman & Zhang 2020) approximate the difference of utility functions in Loo by splitting into two subsots of equal cardinalities and randomly drawing subsots from each of them:
Φise (a, y) Ep, (U( -D,,)..)
where all the subsets from D2 contain (x, v), while none the subsets from D2 contain (z, y). Lava (Just et al. 2023) measures the quality (z, y) by quantifying how fast the optimal transport cost between the training and validation sets changes when increasing more weight (ax y). That is,
éLav (L. v) = N
Σ NJ. 3
where hy is part of the optimal solution of the transport problem.
16

---

Published in Transactions on Machine Learning Research (12/2024)
DATAOOB Kwon & Zou 2023 relies on the concept of out-of- uut-of-bag estimate to capture the data quality. Given B weak learners fb. each trained on bootstrap sample of D, the quality score is
ODTAAOOS v) Ex=1 1 (wh = 0yT (mt))
Ex=1 [ww 0)
where w/ is the number of times (x, is selected in the b-th bootstrap, and T is evaluation metric (e.g,, correctness (y fo(x)))
A.2 Rarity score
Formally, given synthetic image with extracted feature æ, the rarity score is function rą: R + R such that
if & Ba(ri) RED otherwise
rx(x)=
(6)
min r,ED: (x:)
where (z,) is the distance between and its k-th nearest neighbor in D, and Ba(a;) = (ald((;;,,) s N (x;)) is the k-NN sphere with z; as center and N (x;)); as radius. The rarity score strongly depends on could map far unrealistic examples to a positive high score, namely they would be considered authentic, while low values of & could map real examples slightly different than the training data null score, namely they would be considered artifacts.
the choice the hyperparameter ke high values
Because the rarity score strictly depends the hyperparameter k {1...., 1}, need to estimate proper value k. Let's assume the existence of an optimal k and use the small of anomalies to estimate it. Idcally, & should be: (1) as low as possible to assign mull scores to unrealistic anomalies, and (2) high enough to assign positive scores to the real training anomalies.
Following this insight, we assume Bayesian perspective and set a normalized variable's K prior to be uniform
1 4-1 K = n n-1 BETA((, UNIF(O, 1).
Roughly speaking, min-max normalize K to [0, 1] to exploit that Beta prior with Bernoulli likelihood results in Beta posterior distribution. Because we want the minimum & that assigns positive scores to the training anomalies (ommamm we compute for each &m the minimum kn such that Tk,, (#m) > 0. The set of n-I Imsm is the empirical evidence for the Bayesian update, which
normalized { Am-1)
K
Kii -1 n-1
BETA 1+ Σ km
Ki 1
n-1
n n-1
mmm
m.m
Finally, we estimate & as the 95th percentile of the posterior distribution of K
arg min P =E(0,1]
K
51 2 0.95
(7)
which guarantees that at least 95% of real anomalies get positive rarity score.
17

---

Published in Transactions on Machine Learning Research (12/2024)
A.3 Prompt tuning for CLIP
XCandidate proupt templates for NvIec:
['('', 'damaged (P),
['rlaslean ()', <() with flav'l,
['perfect "0 with defect'), ['unblenished 0',
damage'),
['O without flav', '0 with flav'].
I'O without defect', 'O with defect'),
['a photo of normal O','a photo of monalous O'],
['a cropped phato of nornal ('', 'a cropped photo of an anonalous ()),,
photo of ['s photo of normal () for inepection', 'a photo of an anonalous ( for inspection'), viewing',
['a dark photo of
()1,,
I's photo
normal
photo
viewing').
['a bright photo of nornal {)', 'a bright photo of anemalous {}'),
I'a close-up photo of normal O',
close-up photo nornal ()' fa blurry photo of anenalous {}*),
anonaloua 0?),
('a blurry photo
I'a photo of a amall nornal D'' 'a photo of small anaraloua ()', large
()', 'a photo ['s photo of normal () for vieual inspection', 'a photo of an anonslous O for vieual inspection") anonaly detection', photo an [ photo of (H''a photo of something'],
a'a photo
large
(+),
I's photo
anonaly detection'),
a'a cropped photo
0', 'a cropped photo of something'), ['a dark photo of (F', fa dark photo of something'],
I'a photo of a O for inspection', 'a photo of sonething for inspection'],
['s bright photo
()', bright photo eonething'],
['s close-up photo of a O)' close-up photo of sonething'),
('s blurry photo
O', blurry photo sonething'],
['a photo of { for visual inspection', 'a photo of somathing for visual inspaction'],
(' photo of () for anonaly detection', 'a photo of something for anonaly detection']
XEAP selected prompts for WvTec:
['bottle', "danaged bottle'],
['cable without defect', 'cable with defect'), l'unblanished capsule', 'capsule with damage'), ['carpet', 'danaged carpet'],
I's bright photo of normal grid', 's bright photo of an anonalous grid"), ['hazalnut eithout defast', 'hazalmut defact"], ['s photo of normal leather for inspection', 'a photo of anonalous leather for inspection') ) ['metalcut', 'danaged netalnut'], ['pill', 'danagod pill'],
['a close-up photo of a screx', 'a clase-up photo of somathing'], ['tile', 'danaged tile'], ['toothbrusb without flav', 'toothbrush with flav?), I'a blurry photo ['vood', "danaged wood'], ('zipper without defect', 'zipper with defect']
nornal transistor', blurry photo an anonalous transiator'!
Table 4: Summary of the results obtained by the 13 methods over all 40 datasots. We report the mean + std, computed over all experiments. Overall, EAP achieves the best performance and ranking position for all evaluation metrics as well the best average ranking (last column).
EVALUATOR
AUCai
AULC,
ACC,
AULC,(L)
rAUCat
FAULC2
rACC,
rAULC,
Avg.
Rank
EAP
+
0.15
0.84
3.76
RARITY
0.70
0.15
3.87
3,51
3.49
4.99
LAVA
0.73 1 0.13
0.71
0.16
0.78
0.17
0.73
1 0.14
3.21
1: 2.03
7.79
3.63
7.29
3.62
4.36
1 2.54
5.66
1: 2.19
Paiz + NP
0.67 +: 0.18
0.73
+: 0.16
0.81
+ 0.16
0.75
+ 0.14
4.82
+: 3.22
5.87
3.40
5.90
+ 3.37
6.55
+ 3.22
5.79
+ 2.53
LOO
0.52 + 0.17
0.72
1 0.16
0.79
4 0.16
0.73
- 0.14
7.59
1 3.48
6.42
1
3.57
6,86
4 3,31
7.04
+ 3.42
6.98
1 2.90
KNNSHAP
0.53 1 0.08
0.74
0.16
0.81
0.15
0.78
0.15
8.05
1: 2.02
2.84 5.74
5.59
2.60
9.28
1 2.36
7.16
1.40
RANDOMEV
0.50 + 0.06
0.74
0.15
0.79
0.15
2.43 8.55
5.50
+ 2.45
9.80 + 2.21
7.36
1.66
AME
7.41
Palx
0.52 1 0.14
0.69 0.16
0.73
0.16
0.13 0.70
8.69
3.10
9.66
3.07
10.64 - 2.46
3.73
2.67
8.18
2.13
DATAOOB
0.51 + 0.15
0.71 0.15
0.75
0.16
0.73
0.14
8.56
3.38
8.55 +: 3.15
9.71
2.56
6.04
3.71
8.22
2.41
INF
0.46 + 0.14
0.71
0.79
0.17
0.14 0.78
9.65
2.61
7.13
3.17
7.08 0208:200 2.95
9.56
2.79
8.43
2.26
P.
0.53 + 0.11
0.63
4 0.15
0.69
+ 0.16
0.74
7.59
+ 3.59
11.28
4 2.68
10.38 - 3.32
6.06 4 3.27
8.83 + 2.51
DATABANZUAF
0.45 + 0.14
0.71
0.16
0.17
0.79
0.14 0.14
9.92 +
8.26
+: 3.24
7.49 3:24
9.63 - 2.72
8.83
2.39
18


Reviews:
[{"review_id": "atqXa59G4A", "reviewer": "Reviewer_s2gF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Measuring the quality of auxiliary anomalies is an important task, considering that labeled anomalies are hard to acquire.\n2. The proposed method shows strong empirical performance on various datasets, compared to many competitors.\n3. The authors have performed various experiments, including zero-shot anomaly detection, to evaluate their approach.\n\nWeaknesses\n1. In Definition 3.1, it seems incorrect to define unrealistic anomalies as having null density, since every sample in the data space can have non-zero density even though it is highly unrealistic. Can you provide any concrete examples or logic to support your choice of Definition 3.1, based on real-world scenarios?\n2. The problem setup is confusing. Anomalies are usually defined as data samples that are different from the observed training samples. Then, “$p(X)$ is low” is the same as “$P(Y=1|X)$ is high.” as the authors also mention as “epistemic uncertainty tends to be high for most anomalies.” Is it correct to consider $p(X)$ and $P(Y=1|X)$ separately, treating anomalies like a different label?\n3. The conditional probability $\\hat{P(Y=1|X=x)}$ that the authors aim to estimate in Eq. (4) is the same as $P(Y|x)$, which is the goal of anomaly detection. If one can successfully estimate the probability, why not directly solving the problem?\n4. I can’t understand the intuition behind Eq. (4), which is an essential part of the proposed method. A more detailed explanation would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This work proposes expected anomaly posterior (EAP), an uncertainty-based score function that measures the quality of auxiliary anomalies.\n2. The authors analyze theoretical properties of EAP.\n3. The authors show that EAP outperforms 12 adapted data quality estimators on 40 benchmark datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses above.", "broader_impact_concerns": "I don't have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YPHFTD2rXK", "reviewer": "Reviewer_FR6C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I am positive for this paper being accepted in TMLR (but some changes should be made). The authors did a great job in formulating the method and the experiments are extensive and well supported. Their contribution in the formulation of a quality score is relevant and their use of the expected anomaly posterior is novel and interesting. The results compare the method to a variety of baselines and show a consistent improvement. \n\nBelow a list of strengths:\n- the results show evidence of the method's improvement\n- the quality scores are well formulated and cheap to compute\n-  the impact of adding the auxiliary samples during training has been investigated and documented\n- sensitivity to hyperparameter tuning has been discussed\n\nMy main concern while reading this paper has been the motivation. I recognised that this may span from a lack of expertise in the matter from my side, although I have been working on uncertainty estimation for a while. Is it really needed to model anomaly detection as a binary classification task? I see anomalies as marked outliers, something that can be identified as shifting from the original data distribution. I would have liked to see a comparison of uncertainty estimation and anomaly detection on the main task. For instance, one could take the task of quality assessment on a production line by computer vision. The model would output an uncertainty score for each prediction. Predictions with high uncertainty are most likely anomalies. How does this compare to anomaly detection as binary classifiers? I think a discussion of this type could improve the paper as it would clarify this kind of doubt in other people like me reading the paper. \n\nMore weaknesses below in the requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to rank auxiliary anomalies, namely artificially generated anomalies that can be used to train anomaly detectors. An anomaly is defined as a dataset sample that does not conform to the normal behaviour, e.g. defects in production lines. Detecting such anomalies can be essential in numerous industrial applications, and adding auxiliary anomalies to the training set of anomaly detectors could make a difference in improving their representation since they are - by definition - low density samples. \n\nThe key innovation is the use of the expected anomaly posterior as the quality score of auxiliary anomalies.\nThe authors assume the probability of each sample being an anomaly to be modelled as a a Bernoulli distribution over the examples. One sample is either an anomaly or it is not. The authors then model the probability of drawing an anomaly out of $N$ draws $(N = n \\cdot p(x))$ of the same features $x$ as the conditional probability $P(Y=1|X=x)$, for which they can compute the expectation when $N$ is high. If $N$ is low, the value can be then formulated in relationship to a prior representing the expected ratio of anomalies in the data.\nThe expectation of this conditional probability is used as the quality score. \n\nFurther contributions are the estimation of the data density and conditional probability, which are necessary to get to the quality score. The data density is estimated based on the inverse of the rarity score. The conditional probability is then computed by a squashing scaler over the anomaly scores.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to suggest to the authors some improvements:\n- The paper is at times written unclearly or hard to follow. I detail these parts below, where the authors could improve the clarity by rephrasing some sentences or giving explicit examples. \n- In section 3.2, the concept of sampling $N$ labels for the same example $x$ (virtually by assigning a probability distribution to it) could be illustrated with a figure, as this would make the concept clearer for the reader\n- Eq. 3 is based on a strong assumption that the rarity score can be used to estimated data density. I think the authors should add some basic experiments to strengthen this claim where they compare traditional density estimation to this method and assess formally the error. It would make the contribution stronger.\n- In Eq. 4 there is a $f(x)$ that had not been defined elsewhere. For me it was a bit confusing, because I do not think that there should be $f(x)$ there. Maybe the authors could clarify or correct the equation, if needed.\n- The authors should comment on P3 in page 6. Essentially, their method strongly depends on the density of observed anomalies. However, they justified the method by stating that anomalies are -by definition- sparsely distributed and that this is why auxiliary anomalies are needed as a form of data augmentation. I see a trade-off here that could have been discussed further, because this could be a main limitation of the method. It would be nice to see how by varying the density of anomalies the results remain stable. \n- This point links to the one above. The authors assigned $l/3$ for each type of anomalies when building the dataset. This fraction shows only one side of the medal. How would the method perform when, for instance, no realistic anomalies are available (or only an infinitesimal fraction is, e.g. $l/1000$)\n- In Q5 on page 10, the authors evaluate the sensitivity to the hyper parameters. I was disappointed to see the result table in the appendix instead of the main paper, so I would move it, if possible. Moreover, it is not clear to me why only the rAUC was used in the evaluation. Would it be possible to add the AUC values as well?\n\n\nSections where the writing was not clear:\n- On page 7 the authors use a lot of terms in parentheses. I would avoid because it becomes very confusing to follow. For example, when they state \"EAP consistently obtains the lowest (best) average .. \" I would choose only one term between the two. Similarly on the caption of Table 1 (page 8) and on page 9. \n- In \"evaluation metrics\" I really struggled to understand what the rankings score rAUC meant. Since the authors use this metric in several places to describe the improvement of their method over the baselines, the authors should clarify what this means by giving an example or making the description less cryptic.", "broader_impact_concerns": "No ethical statement was required for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hx27jY2frZ", "reviewer": "Reviewer_MpzE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. Anomaly quality evaluation is a previously unexplored problem.\n2. Extensive evaluation validates the robustness and versatility of the proposed method.\n3. EAP improves both training and model selection.\n\nCons:\n1. The problem of anomaly quality evaluation is intriguing and novel, as stated in the paper. However, after analyzing the method and experiments, it seems that the core task is still anomaly detection rather than evaluating anomaly quality. The design and experiments essentially focus on improving the performance of an anomaly detector $f$, which makes the contribution feel somewhat overstated compared to the high-level goal.\n2. In the contribution section, the paper first introduces aleatoric and epistemic uncertainty, which are central to the proposed approach. However, these concepts are not adequately explained, making it hard for readers unfamiliar with the background to follow. Additionally, while the introduction presents the high-level challenges (balancing dissimilarity and realism), the paper lacks sufficient technical details on how these challenges are addressed, making me confuse with the underlying mechanisms.\n3. The algorithm is built on two key aspects: dissimilarity and realism. However, the paper does not clearly explain why conditional probability is an appropriate metric for dissimilarity or why data density represents realism (as suggested in Eqs 2 and 3). Without a rigorous justification, these choices seem arbitrary, reducing the clarity and confidence in the proposed framework.\n4. The estimation of rarity $r_k$ and the selection of the anomaly detector $f$ are not well justified. For instance, the paper uses SSDO and Isolation Forest in experiments but does not explain the rationale behind these choices. If the method is truly generalizable, it should also be tested with more advanced anomaly detection methods. The lack of exploration into whether the proposed method is detector-agnostic leaves room for skepticism about its universality.\n5. Experiments: (1) In Table 2, certain results are identical across methods for some datasets, why? (2)  Including visualizations of normal and synthetic images, particularly for the three types of anomalies (realistic, indistinguishable, and unrealistic), would greatly aid understanding. (3) The spacing between figures in the experimental section is too small, making the paper difficult to read and visually unappealing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an uncertainty-aware scoring function for evaluating the quality of auxiliary anomalies. By integrating class-conditional probability and data density, the proposed method addresses the challenge of assessing synthetic anomaly quality while enhancing anomaly detector performance. Both theoretical analysis and extensive experiments are provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The 3 reviewers recommend accept, leaning accept and leaning reject.  While not all recommendations are "accept," the task of evaluating auxiliary/synthetic anomalies is interesting, which might benefit the community.   Two of the reviewers are satisfied with the revision and the third didn't indicate additional revisions.
Decision: Accept (poster)