Dealing with Uncertainty in Contextual Anomaly Detection
Authors: Luca Bindini
Lorenzo Perini
Stefano Nistri
Jesse Davis
Paolo Frasconi
Venue: Accepted by TMLR ()

Abstract: Contextual anomaly detection (CAD) aims to identify anomalies in a target (behavioral) variable conditioned on a set of contextual variables that influence the normalcy of the target variable but are not themselves indicators of anomaly. In many anomaly detection tasks, there exist contextual variables that influence the normalcy of the target variable but are not themselves indicators of anomaly. In this work, we propose a novel framework for CAD, normalcy score (NS), that explicitly models both the aleatoric and epistemic uncertainties. Built on heteroscedastic Gaussian process regression, our method regards the Z-score as a random variable, providing confidence intervals that reflect the reliability of the anomaly assessment. Through experiments on benchmark datasets and a real-world application in cardiology, we demonstrate that NS outperforms state-of-the-art CAD methods in both detection accuracy and interpretability. Moreover, confidence intervals enable an adaptive, uncertainty-driven decision-making process, which may be very important in domains such as healthcare.

Published in Transactions on Machine Learning Research (01/2026)
Dealing with Uncertainty in Contextual Anomaly Detection
Luca Bindini
luca, www.bbirrddniiauifffte
AI Lab, Department of Information Engineering University of Florence, Italy
Lorenzo Perini
lorenzo. perenzo.prrin@@kuueeeee....
DTAI Research Unit, Department of Computer Science & Le
KU Lewven, Belgium
Stefano Nistri
stefanonistrij) @ggmall com
Cardiology Service
CMSR Veneto Medica, Italy
Jessc Davis
jesse. jesse.davris
DTAI Research Unit, Department of Computer Science & Lewwen.AI KU Leuven, Belgium
Paolo Frasconi
paolo.frasconi@uunn
AL Lab, Department of Information Engineering University of Florence, Italy
Reviewed on OpenReview: https: oppnnreewe........... net/ nor/forum? id yLXXNDouu
Abstract
Contextual anomaly detection (CAD) aims to identify anomalies in a target (behavioral) variable conditioned on a set of contextual variables that influence the normalcy of the target variable but are not themselves indicators of an anomaly. In this work, we propose novel framework for CAD, normalcy score (NS), that explicitly models both the alcatoric and epistemic uncertainties. Built on heteroscedastic Gaussian process regression, our method regards the Z-score a high-density interval that reflect the reliability of the contextual anomaly assessment. Through experiments on benchmark datasets and a real-world application in cardiology, we demonstrate that NS outperforms state-of-the-art CAD methods both detection accuracy and interpretability. Moreover, high-density intervals enable an adaptive, uncertainty-driven decision-making process, which may be very important in domains such as healthcare.
random variable, providing contextual anomaly score but also
1 Introduction
In some applications of anomaly detection, there are variables that only play a contextual role and are not indicators of an object being anomalous. As rumning example for this paper, suppose that we are interested in assessing the normaley of aorta diameters in patients monitored for potential cardiae risk. In need to collect contextual variables such weight, height, sex, and age, since the size of aorta (the variable for which we are genuinely trying to assess normalcy) depends on them. However, in this specific application, we do not want to flag patients as anomalous if they are overweight or they are too short. for their age.
this application,
This particular problem sctting known as conditional (or contextual) anomaly dctoction (CAD) Song
et al., 2007 Valko et al.) 2011]] Tang et al. 2013 Liang & Parthasarathy 2016 Li & Van Leeuwen 2023 The name originates from the simple observation that rather than looking the joint distribution of all

---

Published in Transactions on Machine Learning Research (01/2026)
variables, we split variables in two groups: contextual variables, x, and behavioral variabled] y. We then fit conditional model and say that given z is anomalous if p(ylat) is small according to our model. Note that modeling the conditional rather than the joint distribution is statistically more efficient, allowing us to work in application domains where data is not abundant.
Related techniques started to be developed much earlier in medical statistics for constructing reference intervals (RIs) and centile curves (Goldstein] 1972 Healy] 1978) [Rossoon 1991; Altman 1993 Zierk et al, 2021 Ammer et al.) 2023). One example application is the construction of child grow standards that provide percentiles of height and weight stratified by age and sex (WHO Multicentre Growth Reference Study Group, 2006). In this body of literature (which, interestingly, does not intersect with the more recent machine learning CAD literature), contextual variables x (such as age and sex) play the role of covariates while y (e.g., height) plays the role of a response variable and is typically a continuous scalar. A classic approach is to assume that y|ı (perhaps after performing some variable transformations (Box & Cox 1964))
is normally distributed, fit regrossion model to predict y from z, and finally compute Z-score (Roman et al. 1989 Lee el. al., 2010 Colan 2013) the difference between predictions and measurement, scaled by error standard deviation (see also Section 2 A Z-score above 2 is a typical threshold to flag anomaly.
A common trait of both CAD and methods for constructing RIs is that anomalous (out-of-reference) data points are identified by analyzing the conditional distribution p(ylæ). Effectively, these approaches are trying to model the aleatoric uncertainty (AU), which originates from the intrinsic variability of the behavior within the subpopulation that share the same context. The (implicit) underlying assumption is that anomalous data points are exactly those with large AU. From the point of view of the model, there is no way of reducing AU, except maybe by including more contextual variables (which may not even be possible). However, there is another source of uncertainty: we may lack samples in certain regions of the data space. This is known epistemic uncertainty (EU) (Senge et al.) 2014 Depeweg et al. 2018 Hüllermeier & Waegeman 2021), and in the present setting it occurs when certain context is underrepresented in the data. On the one hand EU may arise because certain subpopulations are inhorently (very) small and not included in the data. On the other hand, EU may arise due to various ethical, practical, and logistic challenges, which is the case in medical domains such as pediatrics (Ammer et al.) 20233 Zierk et al.] 2021 Ceriotti 2012), or due to various exclusion criteria, which is the case for imaging studies in cardiology (Ricci et al., 2021 Asch et al.] 2019: Lancellotti et al. 2013). In our running example, overweight children may not appear in the data for these reasons. Accounting for the EU is crucial because normaley assessment may be unreliable if the model's EU is large.
Despite the importance of modeling both AU and EU, to the best of our knowledge neither existing CAD nor normaley approaches in medicine distinguish between them. This paper fills this gap. Specifically, we instantiate this idea as heteroscedastic Gaussian process regression (HGPR) (Goldberg et al., 1997) that uses separate Gaussian processes to model the mean and standard deviation. This approach confers several advantages and overcomes some well-known limitations of classic Z-scores (see, e.g., (Mawad et al 2013 Colan 2013 Curtis et al. 2016)). First, it models the fact that the variability of behavior depends on the context. Consequently, what is considered anomalous behavior is coupled to the specific context. Second, using two Gaussian processes enables disentangling the AU in the data and EU in the model. This idea has been previously explored for active learning (Patel et al.) 2022) but not for anomaly detection. Third, being a Bayesian approach, it regards Z as random variable and thus naturally provides a method for computing high-density intervals the anomaly score. This allows conveying sense of the uncertainly of the model's assigned anomaly score.
Our key contributions are:
We explicitly disentangle AU and EU with two independent Gaussian processes.
We enrich the anomaly score with a 95 % Highest-Density Interval (HDI), providing a calibrated and interpretable measure.
In Song et al. 2007 contextual variables are called environmental attributes and behavioral variables are called indicator attribules.
2

---

Published in Transactions on Machine Learning Research (01/2026)
We validate our method on standard publicly available benchmarks, consistently outperforming established CAD and AD methods in terms of ROC AUC and PR AUC.
We demonstrate the effectiveness of approach on real-world clinical cohort involving aortic measurements, showcasing its potential impact in cardiology.
2
Related Work
Iselation Forest
100
90
Height (cng (am))
80
10
Age (m)
100
90
Height (cm)
80
NS
Holight ccm)
70
Age
Ground truth
100
90
Heighttcon (em)
80
70
Normal
Anometous
Uncertain (gradlent)
10 15
20
25
35
Age [m)
Simulated data from WHO growth curves for girls. Here H ~ N(h(A), (A)), where the age- indexed mean function () and standard deviation (AA) are derived from published data (WHO Multicentre Growth Reference Study Group 2006). the training data, A Exp((0.4), so that young girls have much higher probability of appearing in the training data. Non-contextual anomaly detcction algorithms (such isolation forests) trained find the support of the joint p(H, A) fail to spot children that are too tall too short for their age, and additionally flag outliers data points where the marginal p(A) is small. ROCOD, a state-of-the-art. algorithm for CAD, does not suffer from this problem but fails to correctly estimate conditional outliers in the presence of context-specific AU (beteroscedaasticitt) Our approach (NS) correctly identifies conditional outliers and, in addition, can abstain when EU (estimated as the width of the 95% HDI, i(H, A), see Section 3.2 is high: In the plot, points with i(H, A) > 2 are colored in blue with an intensity proportional to i(H, A)).
Figure 1:
3

---

Published in Transactions on Machine Learning Research (01/2026)
The task of contextual anomaly detection (CAD) can be formalized follows. Let D == (zi, (1), dataset, where 2; E TR are contextual variables and yi E IR are behavioral variables. The goal of CAD is to identify data points that are anomalous with respect to the conditional distribution p(y(z). Formally, CAD involves learning a probabilistic model for the conditional distribution p(y(z) that assigns an anomaly score s(2, y) to each observation (a: y), where lower probabilities correspond to higher anomaly scores. This approach ensures that points in proob-bbbbbiltty regions of p(y[z) are assigned low anomaly scores, aligning with the intuition that less probable points should be considered more anomalous.
N be
The foundational work CAD was introduced in (Son et al. 2007 In that paper, contextual and be- havioral distributions were modeled independently using Gaussian mixture models and their dependencies were learned through mapping functions. Anomalies can be then identified as deviations from the learned dependencies. However, the approach may overstimate outlierness of objects with unusual contexts, as it would happen by modeling the joint y) (see also Figure
ROCOD [Liang & Parthasarathy 2016) aims mitigate the context sparsity problem by iotruigg a dual modeling approach that leverages both local and global perspectives to detect anomalies. The local model in ROCOD estimates the expected behavior of an object based on its contextual neighbors, which are identified as the objects most similar in terms of contextual attributes. In contrast, the global model adopts a holistic view of the dataset, learning the overall dependencies between contextual and behavioral variables through regression models. These two models are integrated via an adaptive weighting mechanism. ROCOD dynamically adjusts the relative importance the local and global models based on the density of contextual neighbors for each object. When an object has a sufficient number of contextual neighbors, the local model is prioritized. Conversely, in sparse contexts, the global model dominates, ensuring robust anomaly detection across diverse scenarios.
QCAD (Li & Van Leeuwen 2023) builds upon CAD and ROCOD by using quantile regression forests (Mein- shausen & Ridgeway 2006) to model p(ylæ)) Unlike ROCOD, which relies on the conditional mcan to infer expected behavior, QCAD estimates conditional quantiles, providing a more detailed representation of the underlying data distribution. The anomaly score is computed by estimating the deviation of the observed y from the predicted p(y]z), looking at the width of the conditional percentile interval in which y falls. Intuitively, if y lics outside the estimated quantile range, the score is adjusted proportionally.
Due to its simplicity and interpretability, the Z-score is widely used in normaley studies (Roman et al. 1989
Colan 2013
Campens et al.] 2014) and in the development of reference intervals (Lee et al. 2010
Zierk
ct al. 2021
In our context, the classic Z-score is defined as follows:
Z(z, yー f() f(x)
(1)
where y and x are the measured values of the behavioral variable and the contextual variables, respectively, f is a model trained to predict behavior from context, and S an estimation of the standard deviation (such as the square root of the square loss on the training set). Often, f is a linear model, perhaps after the application of some ad-hoc Box-Cox transformations (Box & Cox 1964p (for example, one might regress the log of height on age in Figure 1 order to model the saturation of growth).
The above Z-score definition does not model heteroscedasticity in the data, i.e. situations when variabil- ity of the behavioral variable in the populations depends the given context. Since the denominator averages out this variability, Z(x. y) is overestimated in low-variance contexts and underestimated in high- variance contexts (Chubb & Simpson] 2012 Mawad et al., 2013). Early approaches attempled to reduce heteroscedasticity by splitting data into separate groups (Goldstein 1972). ) Later, Allmann (1993) proposed a way to estimate variance by training a socond lincar regression model on the absolute residuals of the first model, a technique that may be seen as a simplified, linear, and non-Bayesian version of the technique proposed in this paper.
4

---

Published in Transactions on Machine Learning Research (01/2026)
3 Methodology
This section describes our proposed normalcy score (NS); which tailors the classic Z-score to the contextual anomaly detection setting. Unlike the Z-score, NS is a random variable, even when the model and the test point (zt y) are fixed. As in Te et al. (2005). we joindly train two Gaussian process regression (GPR) models: one to model the mcan and one to model the variance of the target variable as functions of the contextual Teatures. We use GPR for three reasons. First, it provides a flexible non-paramelric framework for modeling plylz) as Gaussian distribution. Intuitively, GPR defines a prior over functions f(x) with variance (x) using mean function m(x) and covariance (kernel) function k(x,='), which encodes assumptions about smoothness and relationships between data points. Second, it allows disentangling the aleatoric and epistemic uncertainty. Finally, it permits providing both a point estimate and high-densily interval for an object'ss 's anomaly score.
We begin by formalizing the definition of NS and its underlying Bayesian framework; then we describe how high-density intervals are derived for anomaly scores.
3.1 Normalcy Score
NS quantifies the deviation of a target variable from its expected behavior relative to the contextual variance. To define NS, we first introduce two koy components modeled using Gaussian process regression (GPR): f1 and ʃ2.
The first GP models the mean of the target variable y given the contextual features æ:
(1(x) ~ GP(m) ki (xr,x)))
where mi (x) is the mean function, and kn (I., is the covariance function of the GP.
The second GP models the log standard deviation:
[2(x) ~ GP(m2(1), ka(,,
where mz(x) is the mean function, and (2 x²) is the covariance function.
We model the log-standard deviation to ensure that the prodicted standard deviation remains strictly positive following Goldberg et al. (1997).
With these definitions, NS is given by:
NS(x,1) v) y y- (1))
eJ2(1)
where y is the observed value of the target variable. Note that NS(x, u) is defined similarly to Z(x, v) but is random variable.
A key challenge with standard GPs is their running time, which scales as O(N3), where N is the number of points in the dataset. To address this, use sparse approximation based on inducing variables (Snelson & Ghahramani 2005 Titsias 2009). Using set of M inducing points to summarize the GP posterior and approximate the covariance matrix, the running time is reduced to O(M2N), where M < N.
We use NS to disentangle alcatoric uncertainty (AU) and epistemic uncertainty (EU) as follows:
AU: Captured directly by the second GP fe, which models the log standard deviation. This un- certainty reflects the intrinsic variability of y given x, which cannot. be reduced even with more data.
EU: Captured through the posterior of both fi and fz. Sparse regions in the contextual feature space lead to higher posterior variances, as the GPs are less certain about the function values in those areas. By sampling from the posterior distributions of fi and f2, we can quantify this uncertainty.
5

---

Published in Transactions on Machine Learning Research (01/2026)
To summarize:
We place independent GP priors on Ji and f2, modeling respectively the mean and the log standard deviation of y
To make inference scalable, we adopt a sparse variational approximation with inducing points.
Training is performed using natural gradient descent. for the variational parameters and the Adam optimizer for the remaining parameters.
3.2 Characterizing the NS distribution
Our model assumes that both f1 and f2 are drawn from Gaussian processes. Specifically, the residual y fz) follows a Gaussian distribution. Separately, the reciprocal of the standard deviation term -- where f2(x) models the log standard deviation, follows log-normal distribution:
f2(x) (ogNormal(-ma(())00(()
As result, pNNS(z, y)) is not normal. Still, we wish to summarize it in terms of position and dispersion, that quantify AU and EU, respectively.
Computing a point estimate. The score s(x, y) for given object is simply defined as the expected value of NS. The independence of the two processes allows us to compute it as follows:
s(x, v) = E[NS(r, y)1 = Ely - fi()]] Bie B(e-/s
Using properties of the lormnlrmal distribution, this simplifies to: s(x v) (y mi (x:))e maa)))
Computing the high-density interval. One key aspect of the Bayesian approach is that it offers a principled method for quantifying the F.U associated with the predicted mean and scale by looking at the concentration of the density p(NS(a, y)). A simple way of expressing the concentration of p(NS(x, y)) is by computing its highest densily interval (HDI) (Kruschke) 2015), defined as the smallest interval of values of NS such that p(NS(z, y)) integrates to given threshold (usually 95%). There is no general analytic form for the HDI. Hence, we first obtain samples of NS by sampling from fi and f2 and then estimate by kernel density estimation. We finally performn numerical integration. We denote the length of the 95% interval as i(x, v). As the dataset size increases, FU decreases and p(NS(x,y) becomes more concentrated, resulting smaller HDI.
Aleatoric and epistemic uncertainty. As more data accumulate in the neighbourhood of x, the posterior means mi(x) and m2(1)) converge to the underlying regression functions, while the posterior variances of (z: and (2) shrink. This captures the reduction of epistemic uncertainty with increasing sample size. In contrast, the aleatoric component e²*2("), representing the intrinsic variability of x, does not vanish even in the limit of infinite data. The expectation NS(x, y) therefore contains a multiplicative correction factor (0))22 arising from the log-normal distribution of - this term converges to 1 as oz(x) decreases and ensures that the point estimate 8(1, y) is consistent with the exact posterior expectation. The length of the high-density interval i(x, y), on the other hand, depends on both of()) and (xx) and shrinks as these postcrior variances decrease, making faithful indicator of epistemic uncertainty.
4 Experiments
The goal of the empirical evaluation is address the following four questions:
+QQ:: How does NS perform in detecting contextual anomalies compared to other methods?

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Summary of UCI benchmark datasets with injected anomalies.
Anomaly
Behavioral
Dataset
#Samples
Ratio
Variable
Abalone
4177
2.4%
Rings
Concrete
1030
4.8%
Strength
SynMachine
557
8.9%
IF
Toxicity
908
5.5%
Toxicity
Yacht
308
9.7%
Resistance
RQ2: How sensitive the framework the choice of kernel and number of inducing points?
RQ3: Does i(r, v) correlate with spansity in contextual feature space?
RQ4: Can NS aid in clinical practice for assessing the normaley of the aortic diameter?
4.1 Data
Benchmark datasets. Table 1 summarizes the five UCT benchmark datasets, which include Abolone Nash et al. 1994), containing physical measurements of abalones used to predict their age; Concrete Compressive Strength (Yeh 1998), describing the properties of concrete mixtures with the target being the compressive strength; Synchronous Machine (UCI,20211 representing the behavior of synchronous motors in electrical systems; QSAR Fish Toxicity (Ballabio ct al. 2015), which assesses the acute toxicity of chemicals on fish; and Yacht Hydrodynamies (Gerritsma et. al.
1981 which involves predicting the hydrodynamic resistance of sailing yachts based on hull geometry and operating conditions.
Because those datasets do not contain contextual anomalics, we follow the experimental protocol introduced in Li & Van Leeuwen (003) which injects anomalies into the entire dataset before performing any train/tesi split. Given a dataset [Z,, V:), 1, ......,, NV, where each context consists of / variables, I; = (I.I...., Z,r) the anomaly injection process modifies the dataset as follows:
Normalize the behavioral feature y in the dataset to the range [0, using MinMax scaling, while keeping the contextual features (x1...., unchanged.
2. Select n instances (0 < < N) randomly from the dataset.
For each selected instance (za, yi), perturb the behavioral feature yi to be d1 = yi + es, where e sampled uniformly from the range [-0.5, [0.1, 0.5). The contextual features x; remain unchanged which ensures that the anomaly arises purcly due to the behavioral feature deviating from ils expected value given its context x,,
4. Replace the original instance (x,, yi) with the perturbed instance (I:,, D:)-
This scheme integrates the injected anomalies into the dataset while preserving the structure of the contextual features. Moreover, it addresses limitation of Song et al.) (2007)'s 's approach where some injected anomalies remained contextually normal. Table shows the percent of perturbed examples in each dataset.
Real-world dataset. We use a medical datasel on aorta normalcy (Frasconi el al. 2021). Aortic dilation is a risky condition with potentially deadly outcomes such as regurgitation or dissection. The dataset consists of 1110 normal subjects, and 351 patients with bicuspid aortic valve. The contextual variables are age, gender, weight, and height. We consider, separately, two behavioral variables, i.c., aorta diameters measured at two different locations: sinus of valsalva (SoV) and ascending aorta (AA).

---

Published in Transactions on Machine Learning Research (01/2026)
4.2 Experimental Setup
We compare our approach to three types of anomaly detection methods. First, we consider the contextual approaches QCAD and ROCOD, using the implementation from the QCAD repository F with the hyperpa- rameter settings as the original paper (number of Trees in the quantile regression forest, maximal features, minimum node size, and number of conditional quantiles). Second, we include regression-based contextual basclines. For the Z-score in Equation [ we fit linear regression model on the training set and use single global residual scale S estimated from the training residuals. We further include the linear variance- estimation approach of Altman (9933) and, to isolate the contribution of heteroscedassicitty, a homoscedastic variant of our method, denoted NShom, which uses single GP. Finally, we use the standard non-contextual anomaly detection algorithms Isolation Forest (IForest) (Liu et al.) 2008), Local Outlier Factor (LOF) (Bre-
unig et al. (2000), and Histogram-based Outlier Score (HBOS) (Goldstein & Dengel 2012) from the PyOD library (Zhao et al. 2019) with their default configurations.
The proposed NS relies the heteroscedastic Gaussian Process Regressor (HGPR) model, implemented using the GPFlow library [Matthews et al. 2017/. Key aspects of the HGPR model configuration include the use of a Rational Quadratic kernel [Rasmussen & Williams/ 2006), which is well-suited for capturing both smooth and non-smooth variations in the data. The hederoscedastlic prior is modeled with a constant mean and very low constant variance of 10-5 Inducing variables are initialized to the first 5% of the training data and updated during optimization, with the number of inducing points set proportionally to the dataset size ensure scalability. The model is trained using a hybrid optimization approach that combines Natural Gradient Descent with 7 = 0.02 and the Adam optimizer with learning rate of 0.01. We trained models for maximum of 40,000 epochs.
To evaluate anomaly detection performance, we use a 5-fold cross-validation strategy. This ensures that every data point is evaluated as part of the test set once. We adopt widely accepted metries in the anomaly detection literature [Liang & Parthasarathy 2016; Aggarwal 2017 Kuo et al.) 2018 Li & Van Leeuwen 2023) like area under the ROC curve (ROC AUC) and area under the precision recall curve (PR AUC).
4.3
Results
RQ1:
How does NS perform in detecting contextual anomalics compared to other methods?
Table 2 reports the average over five independent runs (each repeating the injection procedure with a different seed) together with the standard deviation for both metrics across the benchmark datasets. The NS anomaly score is computed as the absolute value of the expected NS, as detailed in Section 3.2 so that deviations in either direction are treated symmetrically.
NS secures the top ROC AUC and PR AUC on Abalone, SynMachine, Toricity, and Yacht, with Concrete remaining the lone dataset where QCAD holds a narrow edge. NS attains perfect scores on SynMuchine; its strong 0.97/0. /0.88 ROC AUC/PR AUC on Yacht, together with competitive PR AUC values of 0.65 on Abalone and 0.67 on Toricily, underscores the method's robustness across heterogeneous datasets.
Both NShom and the Altman method deliver strong results on several datasets, while NS remains the most consistently strong approach overall. Nonetheless, it is striking that an off- the-shelf Z-score baseline, over- looked in prior work, already proves remarkably competitive on this benchmark. To the best of our knowl- edge, this is the first work to include such a Z-score baseline in a contextual anomaly detection benchmark, and its strong results show it should be treated as a meaningful reference point.
Non-contextual methods, including Isolation Forest, LOF, and HBOS, consistently underperform across all datascts and metrics. This result underscores the critical role of explicitly incorporating contextual information into anomaly detection models. Importantly, the gap is driven by the problem formulation. These methods model p(æ, y) rather than the conditional p(y | x), and the effect is not due to the particular choice of detector. We therefore use standard non-contextual basclincs commonly reported in prior CAD studies.
htape:://githuu..cm///ooooo
8

---

Published in Transactions on Machine Learning Research (01/2026)
Overall, the results highlight the robustness and versatility of NS, which consistently performs well across diverse domains and balances performance across diverse evaluation metrics.
Table 2: Average (+ std) ROC AUC and PR AUC computed across five independent anomaly injections
(each evaluated with 5-fold validaanii.
Ahalone
Conarete
SynMachine
Toxicity
Yucht
Method
ROC AUC
PR AUC
ROC AUC
PR AUC
ROC
AUC
PR AUC
ROC AUC
PR AUC
ROC AUC
PR AUC
NS
0.04
1.
1.00
1.00
0.02
0.04
0.97 1: 0.02
0.88 1: 0.06
N8mm
0:95 + 0.03
0.64 + 0.06
0.56 + 0.02
0.52 + 0.03
1.00 + 0.00
1.00
+ 0.00
0.92 + 0.02
0.63
+ 0.05
0.95 + 0.01
0.57
+ 0.06
Z-score
0:95 0.01
0.57 1 0.06
0.56 1 0.03
0.55 1 0.04
1.00 : 0.00
1.00
1: 0.00
0.91 0.02
0.57 1 0.05
0.82 1 0.04
0.53 1 0.09
Altman
0.95 0.01
0,18
+
0.01
0.96
0,91 0.02
0.61 0.03
0.81
=
QCAD
0:90 + 0.01
0,28 = 0.04
0.93 + 0.02
0.64 + 0.04
0.98 = 0.01
0.96 + 0.02
0.86 0.01
0.45 + 0.07
0.96 + 0.03
0.83 = 0.11
ROCOD
0:93 0.01
0.40 :: 0.05
0.79 1 0.02
0.34 +: 0.05
0.90 : 0.04
0.79 4: 0.08
0.84 0.03
0.46 0.07
0.78 4 0.03
0.31 + 0.05
Horest
0.77 0.01
0.05
0.02
0.03
0.35 0.06
0,71
0.10 0.02
=
LOF
0.92 + 0.00
0.39 = 0.03
0.19 + 0.06
0.06 = 0.01
0.92
0.02
0.78 + 0.04
0.61 0.05
0.08 + 0.02
0.71 + 0.01
0.20 = 0.01
HBOS
0.67 + 0.01
0.04
0.00
0.74 + 0.05
0.27 + 0.05
0.67
0.03
0.26 + 0.04
0.69 0.04
0.10
0.02
0.76 + 0.05
0.34 + 0.08
Assessing the reliability of HDI We evaluate using NS in learning with abstention setting, where model only offers a prediction on the lest examples for which it is most confident. By abstaining when the model is uncertain, its performance on those examples for which it does offer a prediction should improve. We consider two different approaches for abstention when using NS. First, we use the HDI length i(0, y): we sort all test points by their HDI length and abstain on the 5% with the widest HDI. Second, we fit an Isolation Forest on the contextual variables and abstain on the test. points that have the top 5% most "outlying" contexts. Table 3 reports the ROC AUC and PR AUC computed over the 95% most confident examples identified by each approach. Across datasets the HDI-based abstention stralegy yields superior or equal performance compared to the contextual baseline, supporting the use of i(z, y) as reliable indicator of epistemic uncertainty.
RQ2: How sensitive is the framework to the choice of kernel and number of inducing points?
To assess the robustness of the NS model, we evaluated the impact of different kernel functions and numbers of inducing points on predictive performance. TTable/]summariee the variations in average expected NS and i(z,8) when switching among kernel choices Rational Quadratic (RQ), Matérn 5/2, and RBF. The model proves stable across these kernels, showing only minor differences in performance, which confirms that the method does not rely heavily on a specific kernel assumption.
Table 5 illustrates the effect of increasing the number of inducing points from 5% to 20% of the training data. We observe that even with a small number of inducing points (5%), the model maintains reliable estimates of both mean and uncertainty, with only marginal benefits from further increascs. This highlights a favor- able trade-off between computational efficiency and predictive robustness, as the 5% configuration already achieves strong and consistent performance across datasets. However, training time increases significantly with more inducing points, scaling from approximately 10 minutes (5%) to around 70 minutes (20%) on an NVIDIA A6000 GPU.
RQ3: Does i(.. y) correlate with sparsity in contextual feature space?
We expect EU to concentrate in underrepresented (sparse) regions in the contextual space since the model has fewer training examples to rely on. In this experiment, we aim to verify that i(x, does indeed align with sparse contextual regions. Specifically, scores s()), computed on contextual variables only. Here s(z) is obtained by applying three classic anomaly detectors: IForest, LOF and HBOS. The association is quantified by the Weighted Kendall Tan correlation coefficient (Shieh 1998). This rank-based metric is particularly suitable for comparing continuous variables, as it accounts for the magnitude of rank differences by assigning weights to the comparisons. In this version of the Kendall Tau coofficient, greater importance is given to elements higher in the rank. The coefficient is defined as:
evaluate the ordinal association between i(z, y) with the anomaly
The Eies Wii sgn[(z; xz)) (wi 2;)1
Eie, Wij
where I4 and xy are the values of one variable, yi and By are the values of the other variable, and sgn denotes the sign function. Weights wiy are used to emphasize certain pairs (i.j) based their significance
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: ROC and PR AUC when making predictions on the 95% most certain test instances as determined by i(x,y) and [Forest on contextual space when using the NS.
Dataset
Abstain using ( y) ROC AUC
Abstain using FForest
PR AUC
ROC AUC
PR AUC
Abalone
0.97
0.71
0.96
0.66
Concrete
0.92
0.65
0.86
0.55
SynMachine
1.00
1.00
1.00
1.00
Toxicity 0.95
0.74
0.92
0.70
Yacht 1.00
0.95
0.97
0.87
Table 4: Impact of kernel choice
NS statistics.
Kernel comparison
AE(NS]
Ai(x,y)
Matérn 5/2 vs RQ
0.07
0.02
RBF vs RQ
0.04
0.01
Table 5: Effect of inducing points ratio on NS and train time. Inducing pts ratio
AE[NS] (with 5%)
train time
5%
10 min
10%
0.06
40 min
20%
0.10
70 min
Table 6: Weighted Kendall Tau correlation between i(x, v) and anomaly scores s(z).
Dataset
Forest
LOF
HBOS
Abalone
0.71
0.66
0.62
Concrete
0.66
0.66
0.64
SyuMachine
0.65
0.62
0.60
Toxicity
0.73
0.68
0.63
Yacht
0.68
0.70
0.65
or importance in the dataset, often assigning larger weights to pairs involving elements with higher ranks. The results in Table 6 show consistent positive correlation, confirming that that i(z,z) can effectively capture sparsity in the contextual fcature space.
4.4 Results on the Real-world Medical Data
RQ4: Can NS aid in clinical practice for asscssing the normalcy of the aortic diameter?
For this experiment we train on normal subjects and test on adult patients with diagnosis of bicuspid aortic valve, condition typically associated with aorta dilations. We use the 40mm dilation threshold defined by current medical guidelines [lsselbacher et al. 2022) to assign patients to the positive class. This threshold is used solely to determine the ground truth labels needed for the ROC analysis and qualitative interpretation. NS assesses normalcy as a continuous quantity conditioned on patient context and not trained on or tuned to this 40 mm cut-off.
Figure 2 shows the ROC for detecting anomalous SoV and AA aortic diameters, respectively. In both cases, our proposed approach results in a superior ROC curve and higher ROC AUC values (reported in the figure's legend) than its contextual competitors. To verify the significance of the difference between the ROC curves, the Delong Lest. (DeLong et al. 1988) was applied, revealing statistically significant differences with QCAD and ROCOD (p < 0.001) for both SoV and AA.
One of the key advantages of NS its ability to quantify EU via i(x, ()..)) Heatmaps in Figure 3 visualize ((x y) (males and females separately) at fixed diameter of 32 mm (SoV and AA). To reduce the number
10

---

Published in Transactions on Machine Learning Research (01/2026)
1.0
0.6
0.4
0.4
(AUC
NS
0.2
Z-score (AUC 0.881)
0.2
Z-score (AUC 0.875)
QCAD (AUC 0.718)
QCAD (AUC 0.740)
ROCOD (AUC 0.834)
ROCOD (AUC 0.783)
0.0
0.4
0.6
1.0
FPR
FPR
(a) SoV
(b) AA
Figure 2: ROC curves for detecting aortic dilation for SoV and AA diameters.
of variables and enable 2D visualization, we use the body surface area (BSA, proportional vW x H) the plots. Dark red regions indicate subpopulations with greater EU and correspond regions of sparsity in the contextual space. This information can help clinicians identify and flag patients for whom the model's predictions are less reliable, thus prioritizing additional diagnostic tests or closer monitoring. Such insights are especially valuable in clinical decision-making, where accurate and interpretable risk assessments are critical.
80
80
3.6
3.2
3.2
2.9
60
2.8
60
2.5
2.4
2.2
2.0
Age
1.8 m
40
40
1.6 2
1.4
1.2 5
1.1 5
20
0.8
20
0.7
0.4
0.4
10
10
0.0
0.0
1.8
2.2
1.2 1.6 1.8 2.0
BSA
BSA
(a) SoV, Males
(b) SoV, Females
80
80
3.6
4.0
70
70
3.2
3.6
2.8
60
3.1 5
50
2.4
50
2.7
2.0
Age
2.2 g
1.6
1.8
30
1.2
30
1.4 S
0.82
0.9
0.4
0.5
10
10
1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4
10 1.2 1.4 1.6 1.8 2.0 2.2 2.4
BSA
BSA
(c) AA, Males
(d) AA, Females
Figure 3: Heatmaps of i((Age, BSA), y) at fixed diameter 32mm for SoV and AA diameters.
Figure 4 highlights four representative cases where the linear Z-score predicts an anomaly score below the clinical threshold Z for least one of the two aortic diameters (SoV or AA). However, the NS interval reveals significant diagnostic uncertainty, indicating that these patients may have non-negligible probability of exceeding 7 2.
11

---

Published in Transactions on Machine Learning Research (01/2020)
3.25
Age: 79,
Sex: Age:
Height: 1.68 m,
Height: 1.84 m.
Weight: 75.0 Kg
Weight: 100.0
SoV: 40.0 mm
2.75
SoV: mm AA: 31.0 mm
AA: 45.0 mm
Score
3
Z-score
2.50
Z-score
NS NS interval
2.25
NS NS interval
I
I
2.00
1.75
1.50
5oV
AA P(NS=2)=1.0
SoV
AA P(N5=2)=0.88
P(NS=2)=0.16
P(N5=2)=1.0
(a) Patient 208
(b) Patient 229
2.50
Age:
Sex: M, 76, Height: 1.78 m. Weight: 78.0 Kg SoV:
2.25
Height: 1.69 m,
2.5
Weight: 109.0 Kg
2.00
AA: 41.0 mm
2.0
AA: 37.2 mm
1.75
Z-score
Z-score
1.50
NS
1.5
NS NS interval
1.25
I
NS interval
I
1.00
1.0
0.75
Sov
AA
SoV PINS>2)=0.39
AA P(NS>2)=0.06
P(N5=2)=0.14
PINS=2)=0.38
(c) Patient 293
(d) Patient 297
Figure 4: Bicuspid patients where NS interval highlights high diagnostic uncertainty. For each diameter, we show the Z-score (red), NS (blue), its HDI (error bar), and the corresponding probability P(NS > 2).
In clinical practice, it may also intuitive to report the posterior probability that NS exceeds a clinically meaningful threshold, rather than inspecting the full high-density interval. For the bicuspid cohort, we therefore compute the exceedance probability P(NS( y) > 2) for each patient and report it alongside the per-diameter intervals in Figure This summary help flag borderline cases beyond point estimates: among palients with E[NS] < 2, 82 (SoV) and 48 (AA) still have P(NS > 2) > 0 (i.e., a non-zero probability of crossing the anomaly threshold).
5 Conclusion
In this paper, introduced normaley score (NS), framework for contextual anomaly detection that explicitly models both aleatoric and epistemic uncertainty. By leveraging heteroscedastic Gaussian process regression, our approach goes beyond traditional Z-scores by treating the anomaly score as a random variable. This enables the computation of high-density intervals, offering a more robust and interpretable assessment of anomalies.
Through experiments on benchmark datasets and real-world application in cardiology, we demonstrated that NS achieves state-of-the-art performance comparing with other contextual anomaly detection methods such QCAD and ROCOD. Additionally, the ability to quantify uncertainty provides significant advantages in high-stakes domains like healthcare, where decision-making must often account for sparse unreliable data. Developing approaches that explicitly reason about predictive uncertainly is important for promoting trust in machine-learning systems. This is particularly true in high-stakes applications such as medicine, where predictions may influence decision-making and it paramount to know when an output should be treated with caution or further investigated. Although NS sometimes offers only a modest margin over plain 7-score, experiments are, to our knowledge, the first to reveal that such simple baseline can already
12

---

Published in Transactions on Machine Learning Research (01/2026)
be remarkably competitive contextual anomaly detection tasks, observation that future benchmarks should not overlook.
Future work will explore extending the NS framework to other domains where contextual information plays a key role, like in the aortic diameters scenario. In addition, the case of vector-valued behavior needs to be addressed by taking into account the relationships among different. behavioral variables. For example, the overall shape of the aorta might be anomalous even though every individual diameter (such as So AA) may be normal. This can be addressed with multi-output/ pullti-ontpuut/muuttttttttt mmulti-takk Gaussian processes (Bonilla et al. 2007), and more structured constructions that place Wishart process priors over context-dependent covariance matricos to explicitly capture correlations between behavioral variables (Hcankulani & van der
Wilk 2019
Overall, NS represents a significant step forward in contextual anomaly detection, providing a robust, inter- pretable, and uncortainty-aware solution this challenging problem.
Acknowledgments
JD acknowledges the support of the Flemish government under the "Onderzoeksprogrammaa Artificièle In- telligentic (AT) Vlaanderen" programme. special thanks to Curzio Checcucci for his valuable feedback during the initial formulation of the problem.
References
Charu C Aggarwal. Outlier ensembles. Springer, 2017.
Douglas G. Altman. Construction of age-related reference centiles using absolute residuals. Statistics in Medicine, 12(10):917 (1449) 1993. doi:
Tatjana Ammer, André Schützenmeister, Ulrich Prokosch, Manfred Rauh, Christopher M. Rank, and Jakob Zierk. A pipeline for the fully automated estimation of continuous reference intervals using real- world data. Scientific Reports, 13(1):13440, 2023. doi: 14.1088//41508--224400000
Federico M. Asch, Jose Banchs, Rhonda Price, Vera Rigolin, James D. Thomas, Neil J. Weissman, and Roberto M. Lang. Need for a global definition of normative echo values rationale and design of the world alliance of societies of echocardiography normal values study (wase). Journal of the American Society of Echocardiography, 32(1):157-1622 122.e2, 2019. doi: (t00///.cho.2018.1000000
Davide Ballabio, Matteo Cassotti, Viviana Consonni, and Roberto Todeschini. QSAR fish toxicity. UCI Machine Learning Repository, 2015.
Edwin V Bonilla, Kian Chai, and Christopher Williams. Multi-task Gaussian process prediction. In Adv. Newral Inf. Process. Syst., volume 20, 2007.
G. E. P. Box and D. R. Cox. An analysis of transformations. Journal of the Royal Statistical Society: Series B (Methodological), 26(2):211 243, 1964. doi: 001111/)2917-60111196..... 100.0055......
Markus M Breunig, Hans-Pete Kricgel, Raymond T Ng, and Jörg Sander. LOF: identifying density-basod local outliers. In Proc. of the 2000 ACM SIGMOD Inl. Conf. on Management of data, pp. 93 104, 2000.
Laurence Campens, Laurent Demulier, Katya De Groote, Kristof Vandekerckhove, Daniël De Wolf, Mary J. Roman, Richard B. Devereux, Anne De Paepe, and Julie De Backer. Reference values for echocardiographic assessment of the diameter of the aortic root and ascending aorta spanning all age categorics. The American Journal of Cardiology, 1 114(6):914 920, 2014. doi: 10. n..10166j.amjaatt www.jarrdd.01..6...... 20....06......
Ferruccio Ceriotti. Establishing pediatric reference intervals: A challenging task. Clinical Chemistry, 58(5): 108-810, 2012. doi: 10.1373/clinee cwwnchem..mmmm crrthhem.2012..0. 2012.183483
Henry Chubb and John M. Simpson. The use of Z-scores in pacdiatric cardiology. Annals of Pediatric Cardiology, 5(2):179, 2012. doi: 10.4103//0944-2099.9922
13

---

Published in Transactions on Machine Learning Research (01/2026)
Steven D. Colan. The why and how of Z scores. Journal of the American Society of Echocardiography, 26 (1):38 40,8840,450 2013. doi: 10.1016/j.echo.2011 2012.11.00
Alexander Curtis, Tanya Smith, Bulat Ziganshin, and John Elefteriades. The mystery of the Z-score. AORTA, 04(04):124-330, 2016. doi: 10. 2..124//..oorta.211..66.
Elizabeth R DeLong, David M DeLong, and Daniel L Pearkon Comparing the areas under two more correlated receiver operating characteristic curves: nonparametric approach. Biometrics, pp. 837 845, 1988.
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi- Velez, and Steffen Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International conference machine learning, pp. 1184 1193. PMLR, 2018.
Paolo Frasconi, Danicle Baracchi, Betti Giusti, Ada Kura, Gaia Spaziani, Antonella Cherubini, Silvia Favilli, Andrea Lenarda, Guglielmina Pepe, and Stefano Nistri. Two-dimensional aortic size normalcy: A novelty detection approach. Diagnostics, 11(2):220, 2021. doi:
J. Gerritsma, R. Onnink, and A. Versluis. Yacht. Hydrodynamics. UCI Machine Learning Repository, 1981.
DOI: https://doi..rr///// 10. /tt../////....22//5/G.... /CXGGRR.
Paul W. Goldberg, Christopher KT Williams, and Christopher M. Bishop. Regression with input-dependent noise: A Gaussian process treatment. In Adv. Neural Inf. Process. Syst., volume 10, pp. 493 499, 1997.
H. Goldstein. The construction of standards for measurements subject to growth. Human Biology, 44(2): 255-261, 1972.
Markus Goldstein and Andreas Dengel. Histogram- based outlier score (HBOS); A fast unsupervised anomaly detection algorithm. KI-2012: poster and demo track, 1:59 63, 2012.
M. J. R. Healy. Statistics of growth standards. In Principles and Prenatal Growth, pp. 169-181. Springer US, Boston, MA, 1978. doi: 10.1007 1-77-----64---0-----55
Creighton Heaukulani and Mark van der Wilk. Scalable bayesian dynamic covariance modeling with vari- ational wishart and inverse wishart processes. Advances in Neural Information Processing Systems, 32, 2019.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457-500 2021. doi: 10.1007/ 510994-001--59466-3
Eric M. Isselbacher, Ourania Preventza, others, and Y. Joseph Woo. 2022 ACC/AHA guideline for the diagnosis and management of aortic discase: A report of the american heart association/ameen american college of cardiology joint committee on clinical practice guidelines. Circulation, 146(24))e334-e444) e482, 2022. doi:
10.1161/CIR.. /CI...0.0000000000000000
John K. Kruschke. Doing Bayesian Data Analysis. Academic Press, Boston, 2nd edition, 2015. doi: 10. (ttg:/9977--11---088---00000
Yu-Hsuan Kuo, Zhenhui Li, and Daniel Kifer. Detecting outliers in data with correlated measures. In Proc. of the 27th ACM Inl. Conf. on information and knowledge management, pp. 287 296, 2018.
Patrizio Lancellotti, Luigi P. Badano, others, and Gerald Maurer. Normal reference ranges for echocardiog- raphy: Rationale, study design, and methodology (norre study). European Heart Journal- Cardiovascular Imaging, 14(4):303-308 2013. doi: 10.1093/chi /11093/hjje//jct00..
Quoe V. Le, Alex J. Smola, and Stéphanc Camu. Hoteroscodastic Gaussian process regression. Proc. of the 22nd Int. Conf. on Machine Learning ICML '05, pp. 489-496 ACM Press, 2005. doi: 10.1145/1102311. 1102413.
14

---

Published in Transactions on Machine Learning Research (01/2026)
W. Lee, T. Riggs, V. Amula, M. Tsimis, N. Cutler, R. Bronsteen, and C. H. Comstock. Fetal echocardiog- raphy: 7-score reference ranges for large patient population. Ultrasound in Obstetrics & Gynecology, 35 (1):28 34, 2010. doi: 10.1002/uog.7444 7483.
Zhong Li and Matthijs Van Leeuwen. Explainable contextual anomaly detection using quantile regression forests. Data Mining and Knowledge Discovery, 37(6):2517-2663, 2023. doi:
Jiongqian Liang and Srinivasan Parthasarathy. Robust Contextual Outlier Detection: Where Context Meets Sparsity. In Proc. of the 25th ACM Int. Conf. on Information and Knowledge Management, pp. 2167-2172.
ACM, 2016.
doi: 10.1145. 10.1145/298332...
Fei Tony Liu, Kai Ming Ting, and Hua Zhou. Isolation forest. In Highth IEEE Int. Conf. on Data Mining, pp. 413-422, 2008.
Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke. Fujii, Alexis Boukouvalas, Pablo León-Villageee Zoubin Ghahramani, and James Hensman. GPflow: A Gaussian process library using TensorFlow. Journal of Machine Learning Research, 18(40):1-6, 6, 2017.
Wadi Mawad, Christian Drolet, Nagib Dahdah, and Frederic Dallaire. A review and critique of the statistical methods used to generate reference values in pediatric echocardiography. Journal of the American Sociely of Echocardiography, 26(1):29 37, 2013. doi: 10.1016/j.echo.2U00 http::/.eccco.222.0...... 2012.09.021.
Nicolai Meinshausen and Greg Ridgeway. Quantile regression forests. Journal of Machine Learning Research, 7(6), 2006.
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1994.
Zeel B. Patel, Nipun Batra, and Kevin Murphy. Uncertainty disentanglement with non-stationary het- eroscedastic Gaussian processes for active learning. In NewrIPS Workshop Gaussian Processes, Spa- tiotemporal Modeling, and Decision-making Systems, 2022. doi: 10.48550 20.:500/XXi.22200100000
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT
Press, Cambridge, Mass, 2006.
Fabrizio Ricci, Nay Aung, others, and Steffen E. Petersen. Cardiovascular magnetic resonance reference values of mitral and tricuspid annular dimensions: The uk biobank cohort. Journal of Cardiovoscular Magnetic Resonance, 23(1):5, 2021. doi: 10.1886/112008--2---00-88
Mary J. Roman, Richard B. Devereux, Randi Kramer-Foo and John O'Loughlin. two-dimensional echocar- diographic aortic root dimensions in normal children and adults. The American Journal of Cardiology, 64 (8):507-512, 1989.
Patrick Royston. Constructing time-specific reference ranges. Statistics in Medicine, 10(5):675 690, 1991. doi: 10.1002/sim.4780111
Robin Senge, Stefan Bösner, Krzysztof Dembezyíski, Jörg Haasenritter, Oliver Hirsch, Norbert Donner- Banzhoff, and Eyke Hüllermeier. Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255:16-20 2014. doi:
2013.07.030.
Grace S Shich. A weighted Kendall's tau statistic. Statistics & Probubility Letters, 39(1):17-24, 1998.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. Adv. Neural Inf. Process. Syst., 18, 2005.
Xiuyao Song, Mingxi Wu, Christopher Jermaine, and Sanjay Ranka. Conditional Anomaly Detection. IEEE Transactions on Knowledge and Data Engineering, 19(5):631 645, 2007. doi: 10.1109/TKEE.200...
Guanting Tang, James Bailcy, Jian Pei, and Guozhu Dong. Mining multidimensional contextual outliers from categorical relational data. In Proc. of the 25th Int. Conf. on Scientific and Statistical Database Management, pp. 1-4. ACM, 2013. doi: 00.1145/2484838.248
15

---

Published in Transactions on Machine Learning Research (01/2026)
Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Artificial intel- ligence and stalislics, pp. 567-574, 2009.
UCI. Synchronous Machine. UCI Machine Learning Repository, 2021.
Michal Valko, Branislav Kveton, Hamed Valizadegan, Gregory F. Cooper, and Milos Hauskrecht. Conditional anomaly detection with soft harmonic functions. In 2011 IEEE 11th Int. Conf. on Data Mining, pp- 735-
743, 2011. doi:
WHO Multicentre Growth Reference Study Group. WHO child growth standards based on length/heighe 200.::6666--2222200660.222333330 I-Cheng Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 1998.
weight and age. Acta Paediatrica, 95($450):76 85, 2006. doi:
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96):1 7, 2019.
Jakob Zierk, Markus Metzler, and Manfred Rauh. Data mining of pediatric reference intervals. Journal of Laboratory Medicine, 45(6):311-317, 2021. doi:
16


Reviews:
[{"review_id": "p8YNjHkQaP", "reviewer": "Reviewer_DdVc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors propose the use of heteroscedastic gaussian process regression as a model for contextual anomaly detection- a setting where characterization of anomalies in so called \"behavioral variables\" depends upon the value of simultaneously observed \"contextual variables\". \n- After fitting gaussian process models to data using a variational approximation based on inducing points, the authors define various statistics from the heteroscedastic GP regression fits which can be used to detect anomalous samples, built around a normalcy score (NS): a distributional extension of traditional Z scoring. \n- With these statistics in hand, they evaluate the performance of NS based point estimates in detecting contextual anomalies, compared to other methods formulated specifically for contextul anomaly detection; e.g. Quantile Contextual Anomaly Detection (QCAD), Robust Contextual Outlier Detection (ROCOD), and a simple Z scoring baseline, as well as non-contextual methods for outlier detection such as isolation forests. They show that NS compares favorably to other methods in a large majority of cases in various datasets into which anomalies are artificially injected through perturbations of real observations.\n- The authors then go on to characterize the variance in the normalcy score distribution, showing that it appears to widen (reflecting more epistemic uncertainty) in locations where data are sparse. \n- Finally, they apply their normalcy score based analyses to real-world medical data, in which anomalies in aortic diameter were analyzed using normalcy score based metrics, as well as other competing methods. Normalcy score was found to perform well in detecting anomalies against human defined ground truth, and likewise was used to characterize model prediction confidence as a function of various contextual variables. \n\n\nA major strength of the method proposed by the authors is interesting in that it appears conceptually more straightforwards and effective than many previously proposed methods in the field. \n\nA major weakness of the paper is lack of clarity. It is difficult to evaluate claims made about the model's ability to separate different sources of uncertainty as the principal driver of the model's success.", "claims_and_evidence_yn": "No", "claims_explanation": "- As stated above, I found that lack of clarity made it difficult to evaluate the actual details of the proposed method, and alignment with the claims made, especially around the importance of separating epistemic and aleatoric uncertainty. I will group together specific examples related to this point in the requested changes section below.\n- I attempted to locate appendices where experimental details might be explained in greater detail, but I was unable to find them. Please let me know if I have missed something.", "audience_yn": "Yes", "audience_explanation": "The method presented here appears to be at least as effective as previously discussed methods for contextual anomaly detection in a wide variety of tasks, while presenting a Bayesian approach which is conceptually more straightfowards than existing proposals for this problem. I believe these qualities would make this work interesting for the TMLR audience.", "requested_changes": "### Methodological details. \nThe following details should be specified somewhere in the main manuscript (or clearly referenced into an appendix), as I believe they are important to evaluate the claims made. \n- **Implementation details of QCAD.** The default implementation of QCAD from Li and van Leeuwen 2022 describes hyperparameter settings which reflect a balance of logistical constraints and effective performance (e.g. number of trees = 10, maximal features = total features). Were these parameters explored in your comparison? I believe this question to be important because performance differenes between NS and QCAD are quite small in a variety of cases.   \n- **Z scoring baseline.** My understanding of this model is that it represents the z score estimated by fitting a linear regression model from contextual variables to behavioral variables. It would be useful to explicitly define this model, as claims are made as the surprising effectiveness of this simple baseline. \n- **Fitting paradigm.** Please describe in greater detail the fitting procedures used to infer gaussian processes $f_1,f_2$ in each experiment, as well as corresponding inference procedures for other methods. In particular:\n\t- It is not clear from the main text that $f_1,f_2$ are fit jointly as a single, heteroscedastic gaussian process, and that the parameters of $f_1,f_2$ are thus linked through inference of a shared posterior process. Please make this explicit in your description of the inference procedure. \n\t- For the UCI datasets, are anomalies injected into both the training and test set, or only the test set? \n\t- Relatedly, how are methods like QCAD, which do not explicitly separate out a training and test set, applied to the aorta diameter dataset?\n\t- In the description of the medical dataset, it is stated that `We use the 40mm dilation threshold defined by\ncurrent medical guidelines (Isselbacher et al., 2022) to assign patients to the positive class.` If I understand correctly, this task on its own would be an example of non-contextual anomaly detection, as the human defined threshold is not dependent upon any contextual variables. In general, more detail about the specific goals of the task, the available contextual variables, and the provided examples are necessary to make the value of NS in the example application clear. How should we interpret the example patients in Figure 4, where NS and Z scores give different anomaly scores? Does the fact that we are given the ground truth SoV and AA diameters for each patient provide a ground truth against which we should assess the anomaly scores? Or rather, are these simply illustrative examples of differences in model comparison? \n\t- Please comment on differences reported in scores from Table 6, Li and van Leeuwen and Table 2 in this paper, some of which are quite significant.  \n### Understanding the effectiveness of NS\nThe introduction to this paper closely relates its contributions to the notion of disentangling epistemic and aleatoric uncertainty. This connection is references a few times throughout the paper, but I found the specifics of this connection to be somewhat unclear. I have organized my questions below: \n\n**Disentangling AU and EU.** Section 3.1 states that AU is captured by the second GP $f_2$, while EU is captured by the posterior of both $f_1,f_2$. Can we be more precise here? In particular:\n- I would expect that the posterior *mean* of $f_2$ might not shrink with more data (reflecting AU) but that the posterior *variance* of $f_1$ and $f_2$ would (reflecting EU). If this is what is intended, it would be useful to provide a mathematical statement for precision. As much seems to be stated in section 3.2 (\"`As a result, p(NS(x, y)) is not normal. Still, we wish to summarize it in terms of position and dispersion,\nthat quantify AU and EU, respectively.`\") but at the moment these statements appear to be in conflict. \n- The introduction states that anomalous points are exactly those with large AU. Thus, it would make sense to choose an anomaly metric which captures only aleatoric uncertainty. While $s(x,y)$ as defined in the paper seems very effective, wouldnt it be more appropriate to consider only the mean of $f_2$, instead of including the posterior variance variance in the provided pointwise estimate? Stated differently, I would expect $s(x,y)$ to shrink as we add more training data, which seems inappropriate for an estimate of AU. \n\n**Important baseline models**. As noted by the authors, it is significant that a simple (linear?) z scoring baseline appears to perform very well in contextual anomaly detection tasks. This results suggests that at least some part of the effectiveness of NS may be shared with other methods which employ **any** kind of Z-scoring based formulation, as opposed to other methods which attempt estimation of contextual quantiles, or contextual neighborhoods (e.g. QCAD). It would be useful to consider how NS compares to the following simple baselines to understand how important the separation of EU and AU is to the overall success of this model. \n- **Homoscedastic GP.** If one was to fit a simple homoskedastic GP to training data, and perform hyperparameter search on the marginal standard deviation, one could estimate z scores on test data similarly to how one does in a linear model, computing a residual relative to the mean function. Understanding how this model performs in anomaly detection would be useful as a model of intermediate complexity between NS and Z scoring.  \n- **Altman model.** As noted by the authors, Altman (1993) proposed variance estimation within a linear regression framework. Given the success of the simple linear Z score model, it would be useful to consider a simple alternative to NS anomaly detection based on this model. How well would such a model do without the nonparametric expressiveness of GP regression? \n\n### Differences in problem setting \nIt would be useful to discuss differences in problem setting between this model and others which have been discussed for contextual anomaly detection. \n- **Categorical contextual variables**. Are categorical contextual variables considered in the study? If not, it would be good to include this in the discussion, as a difference in problem setting to other methods which are based on trees or other models which work well with tabular data. \n- **Multiple behavioral variables**. Other contextual anomaly detection methods consider multidimensional outputs, and devote a significant amount of attention to the question of how to attribute anomaly scores to different behavioral variables. While vector-valued behavior is discussed briefly at the end of the work, it should be made clear that this concern is a major part of the problem formulation in other studies of contextual anomaly detection. \n\n### Others. \n\nI believe the following requests would be useful for interested readers of the paper, but are not critical to securing my recommendation. \n- An alternative method of building heteroscedastic Gaussian Processes which generalize well to higher dimensional output could be through the use of Wishart processes or inverse Wishart Processes, for which there are also efficient variational inference schemes (Heaukulani and van der Wilk, 2019).\n- Another example of useful decompositions of uncertainty into epistemic and aleatoric components comes from Depeweg et al. 2018, which may be a useful reference.\n- bottom of page 9: `Dark red regions indicate subpopulations with greater AU and correspond to regions of sparsity\nin the contextual space.`: shouldn't this be EU?\n\n## References\n- Heaukulani, Creighton, and Mark van der Wilk. \"Scalable Bayesian dynamic covariance modeling with variational Wishart and inverse Wishart processes.\" Advances in Neural Information Processing Systems 32 (2019).\n- Depeweg, S., Hernandez-Lobato, J. M., Doshi-Velez, F., & Udluft, S. (2018, July). Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In International conference on machine learning (pp. 1184-1193). PMLR.", "broader_impact_concerns": "Uncertainty estimation is often framed with applications towards high-risk settings, such as clinical practice as described in this work. Considering the ethical implications of trusting the system under consideration with such decisions would be useful to describe in a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VgSIQAKhv0", "reviewer": "Reviewer_KHaV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method for contextual anomaly detection. \nThe method uses two Gaussian Processes to model mean and log standard deviation of the target variables conditioned on the context. \nThe proposed methods can be used to obtain point estimates and high-density intervals.", "claims_and_evidence_yn": "No", "claims_explanation": "The experiments are limited to five small datasets and six shallow baselines. The proposed method only shows slight improvements over the next best baselines on three out of five datasets. Claims of consistent and superior performance would be much more convincing with more baselines, in particular deep baselines, and more datasets.", "audience_yn": "Yes", "audience_explanation": "The proposed method is novel, straight-forward, and effective.", "requested_changes": "The claims of the paper would be much stronger with extended experiments including more recent baselines and datasets.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kKLn3CefFQ", "reviewer": "Reviewer_mKdL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a Gaussian Process Regression (GPR) based framework, Normalcy Score (NS), for the Contextual Anomaly Detection (CAD) task, predicting not only the level of anomaly but also the level of uncertainty for the prediction. Instead of outputting a single anomaly score, such as z-score, the framework regards it as a random variable and outputs a distribution of it, allowing computation of a confidence interval (HDI). It also explicitly considers and disentangles two sources of uncertainty: aleatoric uncertainty (AU) from the nature of the task, and epistemic uncertainty (EU) from the lack of datapoints. In practice, this could greatly improve the interpretability of the predictions by flagging which predictions are not as confident, as well as how large the confidence interval is.\n\nStrengths\n1. Quantitatively modeling confidence level of predictions is conceivably helpful in practice.\n2. The proposed framework is mathematically solid and neat.\n3. The paper is well written, making the logic rather easy to follow.\n\nWeaknesses\n1. Lack enough analyses of the reliability and superiority of the estimated confidence level (HDI), one of the major contributions of the paper. Some analyses could be:\n- Does the 95% HDI truly cover the real z-score calculated or approximated from real data about 95% of the time?\n- Does HDI-based confidence estimation outperform other competitive approaches (or simple baselines such as sample density thresholding in feature space)?\n2. Regarding interpretability, HDI is logically interpretable but not necessarily practically. For instance, the probability of a data point being anomalous (e.g. probability of having a z-value > 2.0) may be practically more straightforward, and could actually be calculated from the already-predicted distribution of z-score.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Despite lack of enough analysis and comparisons for the proposed HDI, the proposed framework is mathematically solid and neat. Furthermore, some sanity check analysis are conducted to show the alignment between model predictions and human interpretation, e.g. high correlation between prediction confidence and data point density in the feature space.", "audience_yn": "Yes", "audience_explanation": "The proposed framework that predicts anomaly along with a confidence score (HDI) improves prediction interpretability, which is conceivably helpful in practice.", "requested_changes": "The following are recommendations for further strengthening the work, rather than requests.\n1. Include some analyses of the reliability and superiority of the estimated confidence level (HDI). This could help link model predictions to real world data, and establish a solid baseline for the path that future works can compare against. Some candidate analyses include:\n- Does the 95% HDI truly cover the real z-score calculated or approximated from real data about 95% of the time?\n- Does HDI-based confidence estimation outperform other competitive approaches (or simple baselines such as sample density thresholding in feature space)?\n2. Propose some variables that can be quickly derived from the predicted distribution of z-scores that are practically more straightforward than HDI, such as the probability of a data point being anomalous (e.g. probability of having a z-value > 2.0).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)