Sequential Harmful Shift Detection Without Labels
Authors: Salim I. Amoukou
Tom Bewley
Saumitra Mishra
Freddy Lecue
Daniele Magazzeni
Manuela Veloso
Venue: NeurIPS 2024 poster (2024)
Keywords: distribution shift
sequential testing
confidence sequences
uncertainty quantification
Primary area: safety_in_machine_learning

Abstract: We introduce a novel approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. It builds upon the work of Podkopaev and Ramdas [2022], who address scenarios where labels are available for tracking model errors over time. Our solution extends this framework to work in the absence of labels, by employing a proxy for the true error. This proxy is derived using the predictions of a trained error estimator. Experiments show that our method has high power and false alarm control under various distribution shifts, including covariate and label shifts and natural shifts over geography and time.

Sequential Harmful Shift Detection Without Labels
Salim I. Amoukou
Tom Bewley Saumitra Mishra Manuela Veloso
Freddy Lecue Daniele Magazzeni
J.P. Morgan Al Research
Abstract
We introduce novel approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires access to ground truth data labels. It builds upon the work of Podkopaev and Ramdas [2022], who address scenarios where labels are available for tracking model errors time. Our solution extends this framework to work in the absence of labels, by employing proxy for the true error. This proxy is derived using the predictions of trained error estimator: Experiments show that our method has high power and false alarm control under various distribution shifts, including covariate and label shifts and natural shifts over geography and time.
Introduction
When deploying a machine learning model in production, it is common to encounter changes in the data distribution, such as shifts covariates [Shimodaira, 2000], labels [Saerens al., 2002, Lipton et al., 2018] or concepts [Gonçalves Jr et al., 2014]. Many methods exist for detecting such distribution shifts. However, distinct but equally important challenge assessing whether shift has a harmful impact on the prediction eтror of a given model. which may necessitate interventions such as ceasing production or retraining the model. Not all distribution shifts are harmful, but traditional methods for shift detection are unable to distinguish harmful and benign shifts.
While some approaches address the specific issue of performance shift, most require access ground truth data labels in the production environment [Gama et al., 2013, 2014, Bayram et al., 2022]. In scenarios where predictions concern future outcomes, such as medical diagnosis or credit scoring, immediate access to labels in production is not feasible. This work focuses on the challenge of detecting harmful distribution shifts those that increase model error in production requiring access to labels. As Trivedi et al. [2023] note, current methods for harmful shift detection without labels rely on disparate heuristies, often lacking solid theoretical foundation. Such methods include proxies based on aggregate dataset-level statistics [Deng and Zheng, 2021], ], optimal transport mappings between training and production distributions [Koebler et al., 2023], and model-specific metrics such as input margins [Mouton et al., 2023], intturbation-eensiivie disagreement-metries [Chen al., 2023, Ginsberg et al., 2022], and prediction confidence [Guillory et al., 2021, Garg et al., 2022]. While such methods may be practically effective in certain contexts, they rely on assumptions and correlations that do not hold universally, so can provide no guarantees.
without
[Ng et al., 2023],
Furthermore, conventional methods rely on two-sample or batch testing, which involves comparing the statistical properties of a production dataset with those of a control sample. These methods have inherent limitations, as the sample size is prespecified. This is problem because the necessary amount of data detect any given shift is unknown beforehand. Furthermore, real-world scenarios, data typically arrive sequentially over time and shifts may occur either suddenly gradually. In such scenarios, may be desirable detect harmful shifts as early as possible. Batch testing is ill-suited to the sequential context [Maharaj et al., 2023], it does not accommodate the collection of additional data for retesting without adjusting for multiple testing, leading to diminished power.
Correspondence to: Salim I. Amoukou
jpmorgan.com com>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The most principled and relevant work to our problem is that of Podkopaev and Ramdas [2022]. which tackles the problem of sequential harmful shift detection with false alarm control but assumes the availability of ground truth labels in production. Our work builds on the foundation established by Podkopaev and Ramdas [2022], extending the methodology to detect harmful shifts in unlabeled production data while effectively managing false alarms.
Our approach leverages a secondary model to estimate the errors of the primary model. While learning such a model might seem challenging at first, consider a situation where the primary model performs well overall but struggles with specific data subgroups. Sagawa et al. [2019] demonstrate that this phenomenon can occur in natural distributions. In such cases, learning to predict "error given X" might be easier than the primary task of predicting "Y given X", because the error estimator only needs to identify those subgroups where the primary model struggles. This approach has shown promise in recent studies [Zrnic and Candès, 2024, Amoukou and Brunel, 2023]. More generally, Zrnic and Candès [2024] note that predicting the magnitude of the error, rather than its direction, often easier. Furthermore, our approach is based on estimating the proportion of high-error observations over time. For this task, the error estimator does not need to be very accurate; it only needs to assign higher values to observations with higher errors. That is, the estimator only needs to correctly order most observations from low to high error, which is easier than precisely predicting the error itself. We demonstrate in Section 4.1 that even a relatively inaccurate error estimator can be effective at identifying high-error observations, and thus provides the functionality required by our framework. Although this paper uses a learned error estimator's 's predictions as a proxy for error, note that any scalar function correlated with error could suffice to isolate high-error observations. For example, for a well-calibrated binary classification model, we could instead use that model's predicted probability, tracking observations with predictions near 0.5 to identify uncertain predictions.
Figure gives an overview of our approach. We first fit the secondary error estimator model to predict the error of the primary model, then use labeled data to calibrate estimated er- ror threshold (---) that separates observations
Labeled calibration data
Unlabaled production data
Prediction of erro e estimmooo
Calibrated tiresholl
with low and high true error fully as possible. We run the error estimator on all
observations encountered in production and con- tinually monitor the proportion of observations whose estimated error falls above the threshold. We raise an alarm when this exceeds the rate of high-error observations set plus a tolerance threshold éndl and correction terms to deal with the sequential setting and ac- count for uncertainty in the estimates. In the example shown, this occurs at time t 10.
6/10 = 10R10+10)
Got conrections
ALARM
True model error
production
Figure 1: Overview of the proposed approach. Left: calibrating an estimated error threshold to separate low/high true errors. Right: sequentially tracking production data exceeding the threshold and raising an alarm upon a significant increase.
in the calibration
The rest of this paper is organized as follows. Section 2 outlines the problem definition and Section provides an overview of the foundational work of Podkopaev and Ramdas [2022] background. Section is dedicated to the presentation and theoretical analysis of our sequential statistical test. Sec- tion 5 demonstrates the empirical efficacy of our method, showcasing its strong detection capabilities and controlled false alarm rates across various types of harmful shift.
Problem Definition
Let X and Y' be input and label spaces. f X + + be a predictive model, and E )/2 - E be measurable and bounded error function that is selected for monitoring purposes. The model's error on specific observation (X, Y) E X' x V, drawn from joint distribution P(x,r) Px Py|x is represented by the random variable E = ((f((x))))))))))))))())))) Y). The probability distribution of the error is denoted by Pg. As discussed above, our focus is not on detecting shifts in covariates or labels per se, but rather changes in the error distribution Pg. Error changes
be caused by various types of shift in the underlying joint distribution, including changes in Px while the conditional label distribution Prix remains constant (covariate shift) or changes in Py while Pxly remains constant (label shift).
We assume access to a dataset D, = ((X,, )² sampled independently from source distribu- tion P0x P(x.y). In addition, we have sequence of data (X e, Yi)(21 drawn independently from time-
2

---

varying distribution encountered by the model in production, (x.Yy We model the ocurrence of a shift in production by assuming this distribution is equal the source before some time T E N U {oc} (i.e., ((.,r) P (x,
ry, VI < T) and different thereafter (i.e., (x,Y) # P (X.Y). VI 2 T).
Our goal whenever there is shift, (i.e., T < 20), is to decide if this shift is harmful to the model error. To formalize this, we introduce 0: P(E) + IR+ as mapping from probability distributions on the error space& & to a real-valued parameter. This mapping could, for instance, map the distribution its mean or a certain quantile. We aim to construct a sequential test for the following pair of hypotheses:
Vo:V1s 2 1, ((P)) - ((Pg) + Etol;
H1 :1:32 T (+ (EP > ((Pg) + fool:
where Ph denotes the error distribution at at time the running risk Ei= ((P)) is the average value of the error parameter up to time i, and Gut 2 0 is tolerance level. Intuitively, Ho holds if the running risk remains below that of the source distribution (+)) for all time throughout production, and H1 holds if this condition is violated.
Objective. Construct a a-level sequential test, defined by an alarm function + : U X + {0, 1}. which time t uses the first observations X1 ...... X₁ to output 0 (no harmful shift so far) or (harmful shift; raise an alarm) with a controlled false alarm rate and high power, i.e.,
PHo (3t 2 1: ( X.) = 1) S a, and PH. (3t 2 1: ( Xx) = 1) R 1. (3)
We refer this problem definition as sequential harmful shift detection (SHSD).
3 SHSD with Production Labels
A work closely related to ours is that of Podkopaev and Ramdas [2022], which offers a solution for scenarios where the ground truth labels of the production data are available. This method leverages confidence sequences [Darling and Robbins, 1967, Jennison and Turnbull, 1984, Johari et al., 2015, Jamieson and Jain, 2018], which are uime-uniform (i.e., valid for any time) confidence intervals, allowing for the ongoing monitoring of any bounded random variable. With access to labels, it is possible to calculate the true errors on the production data over time and monitor the running risk.
Choosing the mean the error parameter i.e. 0(Pk) = Ep [E] Podkopaev and Ramdas [2022] use the empirical production errors E1 = ()) Yi).... E, = {(((x), Yt), to construct a confidence sequence lower bound L for the running risk, satisfying chosen miscoverage level Aprod E (0, 1):
P vt 1 1, (#Ei)
Ea) > 1 aprod-
(4)
This equation guarantees that the lower bound remains valid over time with high probability. Fur- thermore. given the errors on the source data E (xf y?).... E. = ((f(x)) ). Y."), either another confidence sequence or a traditional confidence interval method [Howard et al., 2021. Waudby- Smith and Ramdas, 2020] can be used to construct fixed- time upper confidence bound Ú for the mean error P. For miscoverage level Ossurcee E (0, 1), Ü satisfies the following condition:
Vn P 1, IP (0(P2) S ŪEE b..)) > 1 Osource
(5)
An alarm is raised when the lower bound of the running risk in production exceeds the upper bound of the source error plus tolerance Gvol Formally, this equates to defining the function 4 as follows:
i (E E.) {i .......... ... E.) ÚE ... E + Evol
(6)
where the subscripted m denotes that the mean is the error parameter being tracked. This methodology provides uniform control over the false alarm rate across time, i.c..
IP tr. 3 2 1 &m (E,, B.) = < Osource + O'podd
It also makes no assumptions about the data distribution or the type of shift. However, the reliance on immediate access to ground truth production labels at each time t limits the method's practical applicability. We now propose a solution that avoids the need for production labels.
3

---

4
Sequential Harmful Shift Detection without Production Labels
This section consists of two subsections, each detailing one of the two stages of our proposal. The initial stage consists of fitting an error estimator and calibrating it to identify high-error observations with few mistakes. Following this, we apply confidence sequence methods to track the proportion of high errors over time production, and develop a test for raising alarm based on this proportion.
4.1 Fitting and Calibrating the Error Estimator
The primary drawback of the Podkopaev and Ramdas [2022] method is its reliance on having ground truth labels for the production data, which are often unavailable in real-world scenarios. A straightforward solution is to use plug-in in approach: replace the true error in production with an estimated error obtained from a secondary predictive model, denoted as f X + E. This model trained to predict the true error of the primary model using any available labeled data. We can then reformulate the alarm function of Equation to deal with unlabeled production data as follows:
x,) = (f((()))........ f(X.)) > (E F.2) Gua
(8)
m
If f(-) is sufficiently accurate, the performance of this alarm mechanism should align closely with what would be achieved if ground truth labels were available. However, even if the estimator f exhibits strong performance on its training distribution, the absence of labels in production makes difficult to conclusively determine the alarm's 's reliability in shifting production environment.
Our strategy to address this issue consists of using calibration step derive more reliable statistic from the imperfect estimator f(-), Specifically, we propose to track the proportion of observations above a carefully-selected quantile of estimated error, rather than the mean value as in the original method of Podkopaev and Ramdas [2022]. The fundamental hypothesis here is that an estimator, even if not particularly accurate predicting error magnitudes, may still effectively distinguish between the lowest and highest errors across dataset, thereby preserving most ordinal relationships between observations. For example, if f()) has correctly represented some underlying patterss predict the errors, and if k-th and (-th ranked errors are significantly different, then it is highly probable that ((X()) S f(X ())). Focusing on the aggregate distinction of low and high errors rather than the prediction of specific magnitudes allows us to utilize an imperfect estimator F more effectively.
Our proposed calibration process is as follows. Given the labeled source data D, and a trained error estimator i, we identify an empirical quantile of the true errors, g = 2(p. { {E 1= p E [0.5, 1), and empirical quantile for the estimated errors q = 2(ù, (f(X)) X E D.),, (0, 1). such that the selector function Sx.4(X) I (f(X) > i) reliably distinguishes between observations with true error below and above q. Specifically, we seek to balance the statistical power and false discovery proportion (FDP) of the selector, which are defined as follows:
Power Em (x)) 1 > 4
E (x)( x 1 g
FDP
E
(E > ②
Σ Sp (x.)
We search over a uniform grid of quantile pairs (p. compute the associated thresholds (g. Ø), and identify those that achieve an FDP below a maximum value. Among these qualifying pairs, we select the one that maximizes the power. Figure 2 illustrates this process for toy example. In this case, thresholds are found that achieve selector power of 0.72 while keeping FDP below the specified maximum of 0.2.
Sae Sereccion TBP
Selector poow
030 020
05
Figure 2: Calibration toy example. Left: threshold grid created by sweeping p E [0.5, 0.95] at increments of 0.05 and á E [0.1, 0.9 at increments of 0.1. Middle: FDP of selector for each (p. pair. Black outline indicates pairs for which FDP < 0.2. Right: selector power for each (p, p) pair. Green dotted outline indicates the pair that maximises power subject to the FDP < 0.2 limit.
We now present empirical ev- idence that it is possible to achieve high power and a con- trolled FDP in realistic settings,
Corresponding thresholds (q, i) shown as thick lines in left plot.

---

using the California house prices [Dua and Graff, 2017], Bike sharing demand [Fanaee-T, 2013], HELOC [FICO, 2018] and Nhanesi [CDC, 1999-2022] datasets. We partition each dataset into training (60%), test (20%) and calibration (20%) sets and use the training data to train random forests (RFs) as the primary models. However, we first ablate the training data in various ways to ensure the models perform poorly on certain subgroups. The ablation is done on per-feature basis. For continuous features, we exclude 80% of observations with values either above or below the median. For categorical features, we exclude data from one category. We then simulate production environments by gradually reintroducing these previously excluded observations alongside the test set. For each dataset, the number of distribution shifts studied equals the number of features times the number of splits: two for continuous features and the number of categories for discrete ones. We use half of the calibration sets to train RF regressors as the error estimators, then use the remainder to calibrate true and estimated error thresholds using the grid search process described above.
In Figure 3, we present the distribution of the FDP and power across all datasets and shifts, relative to the performance of the error estima- tor, as measured by the R-squared score on the source/calibration data. The R-squared score is binned into quantiles, with 10 bins used. We observe that the estimators generally highly imperfect, with R-squared values consistently below 0.3. Despite these low predictive accura- cies, we can still find threshold pairs that achieve an FDP below 0.2 in the source data (shown next to the red boxplot). The power ranges from 0.4 for the least accurate estimators to 0.9 for the most accurate. Crucially, when we apply the calibrated thresholds in the production environments, we achieve similarly low FDP values (shown in red), almost always below 0.25 (though some reach 0.4), while the power remains similar to the source data, ranging from 0.4 to 0.9. This consistency of the FDP/power even when error estimators are not particularly accurate promising for shift detection.
0.4
1.0
Selectrr F56
0.3
Saloctor Power
0.2
0.4
0.1
0.2
0.0
0.
Risquared
R.squared
Figure 3: Selector FDP (left) and power (right) vs estimator accuracy. Results on source data in blue; results on production data in red.
4.2 Sequential Testing Framework and Performance Guarantees
We can now state the specific objective of our sequential testing framework. During production, we propose test if there is an increase in the proportion of observations exceeding the true error quantile obtained in calibration. This is formalized in terms of the following hypotheses:
Ho : Vt 2 1, P (E > q) S po E 9) + Epl.
(10)
H : 31 2 T: E (F > n) > P E > q) + fwo,
(11)
where pt denotes a probability taken under distribution ph Note that this a special case of the general test in Equations 1 and 2, with the probability 0 ((P)) = P P (E > q) as the error parameter.
Since we cannot observe production errors directly, we use the selector function (x,g(X) as proxy for a check on the true error E > q. The effectiveness of the sequential test under this substitution depends on how well the selector's power and FDP properties generalize from the source distribution to the production environment. In particular, we can show that the method outlined below provably controls the false alarm rate given in Equation if the following assumption holds:
Assumption 4.1. V 2 1, Ei_ PPA (X) E, < 9) Pp (X) = 1,E5
Referring back to the example in Figure 2 (left), this assumption implics that at all times during production, the proportion of data observed so far falling the quadrant above and to the left of the calibrated thresholds (---) does not exceed that observed under the source distribution. While we do not claim that this assumption always holds exactly, we find that it is only violated to small extent in realistic settings (see Appendix A for more discussion and experimental analysis). If this is the case, and thresholds (g. ?) have been found that yield small number of false discoveries in calibration, P pe (X) = 1, E < q). then the number of false discoveries in production will also remain low. substantial increase in false discoveries in production would require a shift specifically targeting those rare observations with low error but high estimated error.
With this foundation established, we can now describe our testing methodology. Following a similar approach to that used by Podkopaev and Ramdas [2022], we construct:
5

---

1. A lower bound of Ei= (p > q) using confidence sequence.
2. An upper bound of P. Po (E q) using a traditional confidence interval.
To construct the lower bound, we rewrite the target quantity as follows:
> 9) ; Ei= (S X): 1, g) Pp (5) (12)
2 / Σ k Ppr ( (X) = 1, g)
(13)
Σ k= P Pa Sp o (X) 1) - Ek- Pp (5r. (X) 1,E 9) (14)
2 P (X) = Po P (X)
= q).
(15)
The last inequality uses Assumption 4.1 substitute the probability of false discovery in production with the probability on the source. As we can empirically estimate both Ppt (S+,4 (X) 1) and po g 1, E S q) (via the labeled source data P,), we can use confidence sequence to construct a valid fime-unifore lower bound of their sum. Specifically, we define the bound Lg as
- (
1 1 E.= 1 ( (x? 1, E S 9 Wi Wny (16)
where W2 and w,, are the widths of the lower and upper bounds of : ا Pp (S 1) and r (SA.g = E S q) with miscoverage levels o₁ and a2 respectively, such that for a total miscoverage level Oprrod = a1 a2 E (0, 1).
( 1:
4) Lgg 2 1 - ppdd-
(17)
The specific values of wi and w. used in our experiments are given in Appendix B. Respectively, these choices correspond to the predictably-mixx mixed empirical- Bernstein (PM-EB) confidence sequence described by Podkopaev and Ramdas [2022], and the classic Hoeffding interval.
We similarly compute an upper bound Uú for TP Po EE > q) as follows:
0, = 1 n E.= 1 g) + wh.
(18)
where Wn is the same as above. This bound satisfies a miscoverage level Orgource E (0, 1), such that
P (Fro > g) s U.) N
assurce
(19)
Finally, we define our sequential test using the following alarm function:
Dy (X....., X,) = 11 {L > U, + emo {m))
(20)
where the subscripted q denotes that we are now detecting shifts in error across a particular quantile, rather than the mean. In Appendix C, we provide a proof of the following statement:
Theorem 4.2. Under Assumption 4.1, Lg and Ug satisfy Equations 17 and 19. Therefore, the function by has false alarm control, i.e.,
PHo (3t 2 1 (XI.......... ..). = 1) S Qosurce + Opprad-
(21)
While a controlled false alarm rate is a desirable property, the power of Dg may be limited
the degree of error change is not large. Noting that (1/t) Σi= I' (E > 9) is lower-bounded (S
= E, > 4), detecting a change requires this probability to exceed Po E > q). Thus, we also propose to compare >i= P = 1, E > q) directly with
by (1/l) Et=1 P
po (S (X) 1, E. > q). This leads to a second test with higher power. It uses an upper bound of
Po Sp. X 1, F > q). defined as:
こ = 1, En 1(5 = 1, E > o) + Wn:
(22)
satisfying
Ppr (Sp (X)= = =1 9) < 022 > 1 assource
(23)
The alarm function for the second test is defined as:
X x,) 1 {L, > 02 Gol
(24)
q
Through an almost identical proof, we can similarly show that &2 also has false alarm control for comparing (1/f) Ei... (Sr 1, F. > q) with PP (Sp (X) = 1, F. > g).

---

5 Experiments
In this section, we compare the performance of the plug-in approach of Podkopaev and Ramdas [2022]'s method (Equation 8), which is designed to detect a change in the mean error, and our approach, which determines if an increasing number of observations fall beyond certain quantile. We focus on the second test (Equation 24) to simplify the comparison with the mean detector and because it consistently outperforms the first statistics. Results for the first test are reported in Appendix E. We conduct three experiments using a variety of datasets and setups. The first experiment aims to illustrate the different approaches and demonstrate the applicability of our method to image data and deep learning models. The second experiment returns to the tabular datasets studied in Section 4.1, going into more detail by comparing the mean and quantile detection approaches in terms of power and FDP on the numerous generated shifts. The final experiment also consists of a large-scale evaluation of the approaches, this case on natural shifts due to temporal and geographical changes. Although the focus of this paper is on the sequential or online setting, we provide an analysis using tate-of-the-art methods in the batch setting in Appendix F.
5.1 Illustrative Example on Image Dataset
The first experiment replicates the setup of Saerens et al. [2002] using the CelebA dataset [Liu et al., 2015]. They demonstrate that ResNet50 model [He et al., 2016] trained this dataset performs poorly on "males with blond hair" due to spurious correlations. We split this dataset into a training set (60%), test set (20%) and calibration set (20%), and train a ResNet50 on the training set. Using half of the calibration set, we train another ResNet50 (with a regression head) as an error estimator. The remaining half is employed determine the empirical quantiles p E [0.5, 1), P E (0, 1) at which we achieve maximum power while keeping the FDP below 0.2. We create a harmful shift in production as follows. For each time step up to t 4990, we sample an observation uniform- renioollr from the test set. Thereafter, we begin to oversample instances of males with blond hair, sampling such an observation with probability B, 1/(1 + exp( 4990))), ). and a random observation otherwise.
The objective of this experiment is to visually observe how the methods can be used to monitor performance shift over time and evaluate how each method compares an idealised version with access to true production errors. Both Podkopaev and Ramdas [2022]'s method (mean detector) and our approach (quantile detector) involve comparing a lower bound an upper bound. For both methods, Figure 4 displays the lower bound in blue and the version calculated with true production errors in gray. For the quantile detector, the blue line corresponds is the estimated lower bound of & Σk=1 px (E > q) with estimated production errors. The gray line represents the lower bound of this same quantity, except computed using the true errors. The blue lower bound of the plug-in approach of the mean detector is defined as (f f(X.)). the lower bound of the original mean detector using true
Ly of Equation 16, which
The gray line represents F.. Fe), errors. The upper bound that needs to be surpassed for each method to raise an alarm is depicted in red. For the quantile detector, this is the second lower bound f.3. and for the mean detector, is F). For the quantile detector, we also plot in pink the upper bound of the first statistic P, (Equation 20).
Mcan detector
Ouantle detector
Shift started 4980
Shift started 4950
True detected 9932 Plup-in not detected
Truc detected 6803
Plug-in debected at 6220
Gif El
2.8
(RKX)... RX.))
0.8
0.5
(D
L using labels
0.2
0.2
0.1
0.2
2000
4000
60000
8000
10003
2000
4000
6000
8000
10000
Time production
Timee productiion
Figure 4: Evolution of bounds in production for mean detector (left) and quantile detector (right).
The R-squared value of the error estimator on the source distribution is 0.35, which is not especially high. By analyzing the upper bounds for each method in Figure 4, we observe that all bounds remain roughly constant before the shift starts. Unsurprisingy, we observe the mean detector using true
7

---

production errors quickly detects the shift (gray line). In contrast, its plug-in version raises an alarm with a significandly delayed detection. For the quantile detector, there much smaller difference between the lower bound of the plug-in and the one using true production errors. This observation validates our expectation that the FDP remains relatively stable post-shift. Additionally, as expected, the plot shows that the lower bound of the quantile detector crosses the upper bound of the second statistic (red line) much earlier than that of the first statistic (pink line).
This experiment suggests that in scenarios with a less accurate error estimator, targeting quantile changes is more effective for detecting harmful shifts than focusing on mean change. Additional experiments on image datasets confirming this observation can be found in Appendix D. Larger-scall analyses in the following subsections examine the advantages of the quantile detector more depth.
5.2 Synthetic Shifts on Tabular Datasets
In this section, we conduct a large-scale experiment to evaluate the effectiveness of both methods in detecting harmful shifts while maintaining their ability control false alarms. We also analyze how these metries relate to the performance of the error estimator. We use two regression datasets (California house prices and bike sharing demand) as well as two classification datasets (HELOC and Nhanesi). We follow the feature-splitting setup of Section 4.1 to generate synthetic distribution shifts, excluding splits that result in subsets with fewer than 10 observations, and repeat each split 50 times with different random seeds.
Table shows the number of generated shifts and the number of harmful shifts detected by each method using the true errors (H-M for mean detector and H-Q for quantile detector). A shift is considered harmful by each method as soon as the lower bound exceeds the upper bound plus Eool = 0.
Table 1: Description of the shifts generated.
Data
# Generated Shifts
H-M
H-Q
california bike
62
10
48
2129
57
961
heloe
3385
774
1283
nhanesi
1697
377
679
The left plot of Figure displays the aggregated results across all distribution shifts for mean detection (red) and quantile detection (green) on the different datasets. 'hee points labeled "all-[method]" represent the average results across the datasets. The quantile method achieves a significantly better power-FDP balance: (power 0.83, FDP 0.11) compared to the mean method: (power 0.67, FDP 0.41) across all experiments. An exception is observed for the Nhanesi dataset, where the mean detection shows slightly better power. However, overall, the quantile detection demonstrates a superior trade-off between power and false alarms. A similar trend is observed in the middle plot, which analyzes the absolute difference in detection time between each method using estimated errors and the same method with access to true errors. In the right plot, we compute how the power across datasets varies when we increase the threshold which we consider the true shift harmful (Etot). Across varying intensities shift, the quantile detector consistently outperforms the mean detector, with false alarm rates ét! O being 0.41 and 0.11, respectively.
0.8
0.6
0.6
eecocrion
0.4
0.44
0.2
cwancie-dcttctor
mean-cetaacto
0.0
6.0
caliomni
bae
mares
10"
Harmhuusess thweshots
Figure 5: Left: Power/FDP when Gool 0 for all datasets. Middle: Absolute detection time
difference vs. the methods using true errors. Right: Power values for different harmfulness thresholds (Coo))
In Figure 6, we further investigate the relationship between the power (top row) and FDP (bottom row) of each method and the error estimator's performance binned into 10 quantiles for each dataset. The error estimator performance, measured by R-squared values, is generally low across all experiments (0.10- 0.26). Notably, the quantile detector consistently maintains a lower FDP compared to the
8

---

mean detector across all eтrоr estimator values. Regarding power, excluding the Nhanesi dataset, the quantile detector performs better than or equal to the mean detector.
talforte
200
313
ELantiie detention
A12
100
312
Figure 6: Power and FDP by error across all datasets.
5.3 Natural Shifts on a Tabular Dataset
In the last experiment, conduct another large-scale evaluation of our approach on natural shifts within the Folktables dataset [Ding et al., 2021]. This dataset is derived from US Census data spanning all fifty states within the US (plus Puerto Rico), each with unique data distribution. Furthermore, includes data from multiple years (from 2014 to 2018), introducing a form of temporal distribution shift in addition the variations between states. We select the income feature as the target label, specifically predicting whether income exceeds $50, 000. We first split the dataset of each state in the year 2014 into training (50%), and calibration (50%). Then, we train separate RF classifier in each state in the year 2014, and an RF regressor to learn the error of the primary model on the calibration set. Subsequently, we evaluate the model's error on all the remaining 50 states over years, effectively creating 250 production datasets. We consider shift to be harmful if the model's error production exceeds the error on the calibration dataset plus Enol = O. We introduce the shift in all datasets starting at time t 3300.
Table 2 summarizes the results for both methods, demonstrating that the quantile detector consistently outperforms the mean detector across all metrics.
Table Comparison of detection methods on Folktables data.
Method
Power
FDP
Mean detection time
Quantile detector
0.48
0.019
3727 4945
Mean detector
0.01
0.19
Figure 7 plots the sensitivity of each method relative to the shift harmfulness threshold. We observe that the quantile detector maintains su- perior performance across all threshold values.
ua andidl strn
Overall, this experiment provides good evidence that our proposed method is effective under the kinds of natural shift encountered in realistic production environments.
02
10n mio
0226 aiian 31%% 8i5e 1.15 8203
Figure 7: Power corresponding to different levels of the harmfulness threshold (Ezot) on Folktables.
6 Conclusion
We have introduced an approach identifying harmful distribution shifts in continuous production environments where ground truth labels are unavailable. Utilizing plug-in strategy that substitutes true errors with estimated errors, alongside threshold calibration step, our method effectively controls false alarms without relying on perfect eтror predictions. Experiments on real-world datasets demonstrate that our approach effective terms of detection power, false alarm control and detection time across various shifts, including covariate, label, and temporal shifts. In future work, we plan to apply interpretability techniques to the quantile detector to understand where and how the data are shifting in the input space, and to use this information to improve the primary model itself.

---

Disclaimer
This paper prepared for informational purposes the Artificial Intelligence Research group of JPMorgan Chase Co and its affiliates ("J.P. Morgan") and not product of Research Department J.P. Morgan. J.P. Morgan makes no representation and warranty whatsoever and disclaims liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment rescarch investment advicc, instrument, financial product service, be used in any way for evaluating the merits of participating any transaction, and shall not constitute solicitation under any jurisdiction or to any person, such solicitation under such jurisdiction or to such person would be unlawful.
recommendation, offer or solicitation for the purchase sale of any security, financial
References
Salim Amoukou and Nicolas JB Brunel. Adaptive conformal prediction by reweighting nonconfor- mity score. arXiv preprint arXiv:2303. 12695, 2023.
Firas Bayram, Bestoun S Ahmed, and Andreas Kassler. From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge- thowledge-Based Based Systems, 245:108632, 2022.
CDC. National health and nutrition examination survey, 1999-2022. URL atttps//wwwit cdc. gov/NcchsNhanes/ Daaeees/Defaane DDffautt...t aspx.
Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting Errors and Estimating Accuracy on Unlabeled Data with telf-training Ensembles, May 2023. URL 15728. arXiv:2106. 15728 [cs].
/tttp://axxiv.org////
Donald A Darling and Herbert Robbins. Confidence sequences for mean, variance, and median. Proceedings of the National Academy of Sciences, 58(1):66 68, 1967.
W. Deng and L. Zheng. Are labels always necessary for classifier accuracy evaluation? In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15064-15077 15073, Los Alamitos, CA, USA, jun 2021. IEEE Computer Society. doi: 10. L009//CVR44447.72200000000 URL httpp://ddo ieescompttersscieeyy orgp org/10. 10. 1109CCVPA64437 2021 01482.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http httt:://arccive..ic.. uci edu/ml.
Hadi Fanaee-T. Bike Sharing Dataset. https://bb.orr/10.4433//.
UCI Machine Learning Repository, 2013. DOI:
FICO. Fico. explainable machine learning challenge, 2018. URL s/explainable- macheee-maai-eennneee Leweeee-eeeaaannaaaaeeeeee
wfc.iitt..iii....o...... com/
Joao Gama, Raquel Sebastiao, and Pedro Percira Rodrigues. On evaluating stream learning algorithms.
Machine learming, 90:317-346, 2013.
João Gama, Indré Žliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-37, 2014.
Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie
Sedghi. Leveraging unlabeled data to predict out-of out-of-distributtoo performance. In International Conference on Learning Representations, 2022. URL https://ppenreeiee..eeeee hettrrooiiieeeeeo/e.....e.. o_HsiiPPh__x
Tom Ginsberg, Zhongyuan Liang, and Rahul G Krishnan. A learning based hypothesis test for harmful covariate shift. arXiv preprint arXiv:22/2.02742,, 2022.
Paulo M Gonçalves Jr, Silas GT de Carvalho Santos, Roberto SM Barros, and Davi CL Vieira. A comparative study on concept drift detectors. Expert Systems with Applications, (1(18):8144-811 2014.
10

---

Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with Confidence on Unseen Distributions. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1114-1124, Montreal, QC, Canada, October 2021. IEFE. ISBN 978-1-66552-812-5. doi: 10. 109/CCCV48922.2021111 100117. URL URI.ttps: http:/tt:://eeeeellorre ieee.. org/document/99 97700////
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.
pages 770-778, 2016.
Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparamet- ric, nonasymptotic confidence sequences. arXiv preprint 1eX0vv:810008244 2021.
Kevin Jamieson and Lalit Jain. A bandit approach multiple testing with false discovery control. arXiv preprint arXiv: 1e0..02295....... 2018.
Christopher Jennison and Bruce W Turnbull. Repeated confidence intervals for group sequential clinical trials. Controlled Clinical Trials, 5(1):33-45, 1984.
Ramesh Johari, Leo Pekelis, and David J Walsh. Always valid inference: Bringing sequential analysis to a/b testing. arXiv preprint arXiv: 151iv:151 2015.
Alexander Koebler, Thomas Decker, Michael Lebacher, Ingo Thon, Volker Tresp, and Florian Buettner. Towards explanatory model monitoring. In XAI in Action: Past, Present, and Future Applications, 2023. URL attps://openreviest .et.:/neeeeeee ftt////rnnd////uWh/aas
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http: //ww:///wwwwww/ cs. www.toootoo..moooooomoo edu/ cifar .html.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-3130. PMLR, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
Akash Maharaj. Ritwik Sinha, David Arbour, Ian Waudby-Smitte Simon Z Liu, Moumita Sinha, Raghavendra Addanki, Aaditya Ramdas, Manas Garg, and Viswanathan Swaminathan. Anytime- valid confidence sequences in an enterprise a/b testing platform. In Companion Proceedings of the ACM Web Conference 2023, pages 396-400, 2023.
Coenraad Mouton, Marthinus W. Theunissen, and Marelie H. Davel. Input margins can predict generalization too, 2023.
Nathan Hoyen Ng, Neha Hulkund, Kyunghyun Cho, and Marzyeh Ghassemi. Predicting out-of- domain generalization with neighborhood invariance. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:// hptp///pppeeeeeee///eee/e/ net/ ji=jdddjjji jiddjkkKJJJZT..
Aleksandr Podkopaev and Aaditya Ramdas. Tracking the risk of deployed model and detecting harmful distribution shifts. In International Conference on Learning Representations, 2022. URL
https://openreviex: http:////pppareeeeeeeeeeooorotop
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure. Neural computation, 14(1): 14(1):21-41,20) 2002.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint
2019.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-
likelihood function. Journal of statistical planning and inference, 90(2))227-244, 2000.
Puja Trivedi, Danai Koutra, and Jayaraman J Thiagaraaan. A closer look at scoring functions and generalization prediction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.

---

Ian Waudby-Smin Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. arXiv preprint arXiv:2010.09686, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXin:1008.07747, 2017.
Tijana Zmic and Emmanuel Candès, Active statistical inference. arXiv preprint arXiv:2403.03208, 2024.
12

---

A Discussion of Assumption 4.1
To formalize the statement in the main body of the paper, we do not expect Assumption 4.1 hold exactly, but we expect that in realistic settings, for all t 2 1, the inequality
Σ pa (Sr 1, S 9) S Ppo pe (S (X) = 1, E S 9) Stol
holds with a small deat. For instance, in Figure 8, we compute the empirical distribution estimate of 5 = Ek= Pp* (X),4(X)= = 1, E S 9) Po (X) 1, E < q) with t equals to the total number of production data across the different distribution shifts and datasets of Section 5.2 and the natural distribution shifts of Section 5.3.
0.08
0.04
0.01
0.00
-0.02
(a)
Figure 8: Distribution of ō across the different shifts and datasets of Section 5.2 (a) and the natural distribution shifts of Section 5.3 (b)
We observe that Assumption 4.1 is valid approximately 50% of the time for both experimental setups, corresponding to the cases where 8 is negative. In the other half of the cases where the assumption is not verified, we note that 5 is very small, often less than 0.01.
It should be noted that Assumption 4. allows for controlling false alarms when is zero negative. To control false alarms when 8 is positive, it is sufficient to always add 5 to the lower bound i. (Eq. 16) to have the false alarm guarantee. Specifically, under the assumption that for all t 2 1, D ps (S),(()) 1, E < q) S Po (Xx, 1, E < q) + 6 we can show, similar to Theorem 4.2, that the corrected bounds Lgorr Lo used in the following statistic:
corr
X.) 1 i > Ü, + 61ol
(25)
will have false alarm control. The proof identical to the proof of the Theorem 4.2, with Ly replaced by Lcorr.
However, in practice. we do not know the value of 8. Fortunately, in most realistic cases we have observed, very small, especially compared our maximum false alarm threshold of 0.2. Therefore, not adding this correction has very little impact on the statistics without correction (Equation 20).
13

---

B Bounds of the Confidence Sequences and Intervals Used
In our experiments, the confidence sequence bound wy is that of the Empirical Bernstein confidence sequence, as defined in the Theorem below. For a more detailed presentation of different confidence sequences, we refer the reader Howard al. [2021].
Theorem B.I. Let Dr = Ei= Xi, and suppose X; are bounded a.s. for each 2 1. Then, for each 10,
C 10, + wi} forms (1 -aaaa-llvve -level confidence sequence for E(pe),
Vi log low V. Ve (X, P4-1)² denotes empirical variance term and Co Vlog(1/a).
where Wi =
When we use confidence interval, we use the classic Hoeffding interval:
On (An + wn forms a (1 alleee confidence interval for E((n))
where W/n
log(2/ao) 2n
C Proofs
Theorem C.1. Under Assumption 4.1, Lg and Üg satisfy Equations 17 and 19, Therefore, the function &g has false alarm control, i.e.
PHo 23 1: (X(, X.) 1) S Soouree + Opprad
(26)
Proof.
Pho (Et 2 1: (NI...... X,) = 1)
=Pro {3t 21:1: > 0, au)
=Pro
21
L. (1/8) Σ (S, (0. (P=-Pp0(),,())=1))
pk (Sr. = -1 =1))
> md
(1/2)
Σ
h-1
PHo
#2
i.g (1/t))
pp (S) 11) ( 1)) 20}
$ PHo
1:
ing - (1/1) Σ 7 (S > PHo {(6o (P) oo
Oeaam + apod
The last inequality is due to Equation 17 and 19.
14

---

D Additional Experiments on Image Datasets
Here, we conduct two experiments using image datasets, specifically CIFAR-10 [Krizhevsky et al.] and Fashion MNIST [Xiao et al., 2017]. Similar to previous experiments, we remove some part of the data during training phase, here 90% of the observations with label for both datasets, and reintroduce them gradually during the production phase.
In Figure 9, observe the same behavior in Section 5.1. The quantile detector detects changes more quickly than the mean detector, and the performance of the former is closer to the true version than that of the latter.
detector
Quantile detector
2.00
1.0
Shift started at 2500 Detected at 4623
Shift started 2500
1.75
Detected at 4592
0.8
1.50
U(i(X)) .... RX
Üa
1.25
Ú(C}. CR)
0.6
0
1.00
[(P(X))..... i(X))
La
0.75
L(E)..... Ex)
0.4
?a using labels:
0.50
0.2
0.25
0.00
0.0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Cumulative Sample Count Overtime
Cumulaative Sample Count Overtime
Bounds on CIFAR-10
Mean detector
Quantile detector
2.00
Shift started at 2500
Shift started 2500
Detected at inf
Detected at 4769
Úli())) 1
Ü.
1.25
Ú(EP..... E
0
((P(X)), F(X))
0.75
İ(EI, Ex)
0.4
L, using labels
0.00
1000
2000
3000
4000
1000
2000
3000
4000
5000
Cumulative Sample Count Overtime
Curulaativ Sample Count Overtime
(b) Bounds on MNIST
Figure 9: Evolution the bounds in production for mean detector (left) and quantile detector (right).
15

---

E Comparison Between o and D
In this section, we will revisit the main experiments from sections 5.2 and 5.3, incorporating comparisons with the quantile detector using the first statistic &.- As expected, figure 10, we consistently observe that the first statistic @ achieves better FDP than the second statistic $ the cost of a much smaller power: In addition, 4, fails to detect any shift in the California dataset and has much higher detection time.
1.0
oxtecttion Sife
0.2
quantile detector v2 mest-detectto
0.
0.0
0.2
0.6
0.6
0.0
casteariie
bikk
hetoc
ditenos
0.00 0.6
0.10
0.15
0.00
0.9 0.10 0.15
Figure 10: Left: Power/FDP when exol
for all datasets. Middle: Absolute detection time
difference vs. the methods using true errors. Right: Power values for different harmfulness thresholds (Eta)).
In Figure 11, we have also computed the power relative to the harmfulness threshold Etol of the Folktables data from Section 5.3. The second statistic performs much better than the first in terms of power, although the FDP of the latter is smaller (0.004) compared to 0.019 for the former.
10
caantiaddetetion gearteeeeeetttr yy moan detection
0.033
0.305
0.20
0.000
0.75
0.20
Figure 11: Power corresponding to different levels of the harmfulness threshold (Ctar) on Folktables.
F Evaluations in Batch Setting
In this section, we compare our approach with a leading method, Detectron, proposed by Ginsberg et al. [2022], in batch setting. Directly comparing method (SHSD) to those in Ginsberg et al. [2022] presents challenges due to fundamental differences in their design. Their methods are tailored for offline batch setup, which requires complete batch of production data to compute statistics and trigger alarms. In contrast, our approach is optimized for an online setting where shifts may occur gradually and continuously, necessitating real-time decisions without the ability observe entire unlabelled batch upfront. Our methodology designed to detect harmful performance shifts on the fly, processing each observation sequentially without requiring access to the full production dataset.
Applying offline methods like Detectron an online setting would be both impractical and unfair, as these methods rely on training a model or computing statisties from a batch of data. Additionally, it would be computationally expensive since Detectron requires training a new model for each batch of production data. Consequently, deploying this approach online would entail training number of models proportional to the production data size.
To provide meaningful comparison, we evaluated our method alongside Detectron in batch setting, progressively increasing the size of the production or dus-i--distiiiution (OOD) data. We generated shifts in line with the setup described in Section 5.2, ensuring no shift within the first 1300 samples
16

---

of production data. We utilized the NHANESI classification dataset, as Detectron is specifically designed for classification tasks. Our experiments were replicated 50 times, yielding total of 10,200 shift instances.
Table 3: Comparison of Power and FDP metrics for Detectron and SHSD across different OOD sizes.
OOD Size
Power Detectron
FDP Detectron
Power SHSD
FDP SHSD
100
N/A N/A
1.00
N/A
0,00
1000
1.00
N/A
0.00
2000
0.96
0,61
0.40
0,02
3000
0.98
0.60
0.63
0.02
3500
0.98
0,60
0.67
0.02
8593
0.98
0.60
0.74
0.04
The results, summarized in Table 3, demonstrate that for smaller sample sizes (100 and 1000), our method did not detect any shifts, as expected given the lack of shifts in the initial 1300 samples. However, Detectron raised a significant number of alarms (1126 out of 1700 for sample size 100 and 1493 for sample size 1000), all of which were false alarms. For larger sample sizes, while Detectron shows high power in detecting shifts, it also produces high false discovery proportion (FDP). In contrast, our method exhibits lower power but significantly better control over false alarms, consistent with our objective of minimizing false positives.
These results confirm that our method performs robustly in both batch and online settings, effectively maintaining low false alarm rates while detecting harmful shifts they arise.
F1 Limitations of Disagreement-basee Detectors
In this section, we highlight some potential limitations of disagreement- detectors, such as Detectron, which may limit their effectiveness in certain contexts.
Ground (Cles Oenural frui (Choo
Wusk mats
Basc Cheorifir
951
3
Figure 12: Illustration of Disagreement-Basee Detector Failure Case
The primary concept behind Detectron is to train disagreement classifier that performs comparably to the original model on the training distribution while disagrecing with the original model's predictions production data. This approach is highly sensitive to the base model's performance, the choice of function class, and the size and nature of the production data. Although Detectron shows high power in detecting harmful shifts (as evidenced by experiments), it may raise false alarms when the shift benign.
In Figure 12, we illustrate failure case for the disagreement based detector. In this example, training data points are represented in red and green, with the ground truth shaded accordingly. The solid black line represents the decision boundary of base model, which we assume to be a perfect classifier. The data has shifted to the right, resulting in unlabeled production data that is still correctly classified by the base model.
17

---

We've also depicted the potential learnable classifier as a dashed line, representing the boundary of all possible functions, which depends on the model type, complexity used for the disagreement classifier, and the nature and size of the data. We have shown potential disagreement classifier in orange that performs similarly to the original model on training data but disagrees with the predictions of the base classifier in the production data. As shown, even with a benign shift, we can still find a disagreement classifier that performs well on training data but disagrees significantly in production, raising a false alarm.
G Experimental Compute Resources
We run all our experiments on an Amazon EC2 instance (cl.4xlarge) that consists of 16 vCPUs and 32 GB of RAM.
H Impact Statement
This research, focusing on developing algorithms to detect harmful distribution shifts in machine learning models, has significant and diverse practical impacts. It offers solution key challenge in the safe deployment of AI across various industries by detecting shifts without needing labeled data. For instance, in healthcare, the ability to identify harmful shifts in predictive models enhances the accuracy and reliability of diagnostic tools, which is especially vital as patient data continuously changes due to new diseases or demographic shifts. finance, the algorithms can detect market trends or consumer behaviour changes that might negatively impact forecasting models, leading to more adaptive and resilient economic models, improved risk management, and better-informed decision-making processes.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This work's 's main contributions can be found in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
be expected to generalize other settings.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: On lines 40-51, we discussed the potential difficulty learning second estimator and provided rationale and examples of when it is possible. We presented the assumptions under which our method should have false alarm control and discussed its validity in practice in appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: We have stated the theorems and the assumptions in the main body and provided the proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose all the information needed to reproduce the main experi- mental results, but we also plan to release the code to use the methods and replicate the experiments.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have used open-source datasets and added the references. We will also release the code with proper readme to use the methods.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
Sobllic/guidee/Coeeuubbbbi sow.sssabbsiiiooooooooooo.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ccc/ppu. wwtpccccppc/cceecccccccccc.ccllt wttpiiigggies///eeeeeeeeeeesseet ProdeeSumissiiono.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We replicated each experiment 50 times and displayed the distribution of errrr in most cases to illustrate how the errors vary within each dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided Section the type of compute workers we used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
cc. cctppppicccpppc/teciccccc ctw.iitieeedddddeeeeei
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Section H.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwoee.hooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We plan to release well-documented code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "Z5eGa2YiyW", "reviewer": "Reviewer_4vCN", "summary": "This work is an extension of Podkopaev and Ramdas. They propose a framework to detect the harmful distribution shift without accessing the true labels during detection. To do that, the authors introduce an error estimator model to measure the error scores. Besides, the authors propose a strategy to manage false positives by leveraging these imperfect error predictions. Experiments show that the method effectively controls false positives and performs well under various distribution shifts.", "questions": "- In lines 117-118, could the authors explain how they connect the upper bound of the training dataset to the lower bound of the test dataset to derive this probability inequality?\n- The proposed method does not rely on distribution assumptions and has very low accuracy requirements for error estimation (In section 4.1). Additionally,  there are no hyper-parameters that need to be preset (in section 4.2). However, there is no free lunch. Despite these excellent properties, does the method will face any other challenges?", "limitations": "- The proposed approach is to detect harmful distribution shifts in production environments, which is a very interesting problem setting. However, there are many out-of-distribution detection and robust learning algorithms that can cover or solve the harmful distribution shift problem, or even solve it directly, which makes the proposed method not so necessary in practical applications.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The label-free solution is essential for practical applications. \n- Algorithm-agnostic makes the approach more applicable.\n- The approach does not rely on pre-assumptions about data distribution.\n- The authors provide detailed mathematical derivations.", "weaknesses": "- In the experiments section, the authors only compare with a baseline scheme and ignore other existing distribution shift detection algorithms (e.g., [1-2]), although they may not claim to only detect harmful distribution shifts.\n\n### Reference\n1. Hinder F, Artelt A, Hammer B. Towards non-parametric drift detection via dynamic adapting window independence drift detection (dawidd)[C]//International Conference on Machine Learning. PMLR, 2020: 4249-4259.\n2. Frias-Blanco I, del Campo-Ávila J, Ramos-Jimenez G, et al. Online and non-parametric drift detection methods based on Hoeffding’s bounds[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 27(3): 810-823.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review and acknowledge our work.\n\n> In the experiments section, the authors only compare with a baseline scheme and ignore other existing distribution shift detection algorithms (e.g., [1-2]), although they may not claim to only detect harmful distribution shifts.\n\nRegarding the suggested paper, neither would be a suitable baseline for our method as they either attempt to detect shifts in general or require access to labels to assess model performance. Comparing them with our method would be unfair since they are likely to have high false alarm rates in a context where there are both harmful and benign shifts. Moreover, the primary purpose of our methodology is to detect harmful performance shifts without using labels.\n\nHowever, the closest baseline to ours is the one suggested by Reviewer KyAy, although it is designed for batch or offline contexts rather than the sequential or online context of our work. We provide comparisons (please refer to our response to Reviewer KyAy) showing that our method also works in the batch context.\n>  However, there is no free lunch. Despite these excellent properties, does the method will face any other challenges?\n\nRegarding the limitations, we acknowledge that our method may perform less effectively when there is a significant concept shift or when we do not have enough calibration data to train and calibrate our proxy. We will include this consideration in the final version of the paper.\n> In lines 117-118, could the authors explain how they connect the upper bound of the training dataset to the lower bound of the test dataset to derive this probability inequality?\n\nBelow, you will find the detailed derivation of Equation 7 (Lines 117-118):\n\n$$\n P_{H_0}\\left\\[ \\exists t \\geq 1: \\Phi_m(E_1, \\dots, E_t) = 1 \\right\\] \n$$\n$$\n= P_{H_0}\\left\\[\\exists t \\geq 1: \\hat{L}(E_1, \\ldots, E_t) > \\hat{U}(E_1^{0}, ..., E_1^{0})+ \\epsilon_{\\text{tol}}\\right\\]\n$$\n$$\n= P_{H_0}\\left\\[\\exists t \\geq 1:  \\left(\\hat{L}(E_1, \\ldots, E_t) -\\frac{1}{t} \\sum_{k=1}^t \\theta(P_{E}^{k})\\right) - \\left(\\hat{U}(E_1^{0}, \\ldots, E_n^{0})-\\theta(P_{E}^{0})\\right) >  \\epsilon_{\\text{tol}} - \\left( \\frac{1}{t} \\sum_{k=1}^t \\theta(P_{E}^{k}) - \\theta(P_{E}^{0}) \\right) \\right\\]\n$$\n$$\n\\leq P_{H_0}\\left\\[\\exists t \\geq 1:  \\left(\\hat{L}(E_1, \\ldots, E_t) -\\frac{1}{t} \\sum_{k=1}^t \\theta(P_{E}^{k})\\right) - \\left(\\hat{U}(E_1^{0}, \\ldots, E_n^{0})-\\theta(P_{E}^{0})\\right) >  0 \\right\\]\n$$\n$$\n\\leq P_{H_0}\\left\\[\\exists t \\geq 1:  \\left(\\hat{L}(E_1, \\ldots, E_t) -\\frac{1}{t} \\sum_{k=1}^t \\theta(P_{E}^{k})\\right) >  0 \\right\\] + P_{H_0}\\left\\[\\left(\\hat{U}(E_1^{0}, \\ldots, E_n^{0})-\\theta(P_{E}^{0})\\right) <  0 \\right\\]\n$$\n$$\n\\leq \\alpha_{\\text{source}} + \\alpha_{\\text{prod}}\n$$\nThe third last line is because we are under $H_0$ and the second last line is due to if $\\exists t \\geq 1:  \\left(\\hat{L}(E_1, \\ldots, E_t) -\\frac{1}{t} \\sum_{k=1}^t \\theta(P_{E}^{k})\\right) - \\left(\\hat{U}(E_1^{0}, \\ldots, E_n^{0})-\\theta(P_{E}^{0})\\right) >  0$, then either $\\exists t \\geq 1:  \\hat{L}(E_1, \\ldots, E_t) -\\frac{1}{t} \\sum_{k=1}^t \\theta(P_{E}^{k}) >  0$ or $\\hat{U}(E_1^{0}, \\ldots, E_n^{0})-\\theta(P_{E}^{0}) <  0$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cMZrjNYGf", "reviewer": "Reviewer_KyAy", "summary": "This paper proposes a new method for identifying harmful distribution shifts when no labels are available at test time. This work is a good contribution to the field of ML and has practical importance.", "questions": "The questions I have pertain to the first point I mentioned in the weaknesses section. Could you clarify that point for me? I would be willing to increase my score if I understand how your work builds on the previous research by Ginsberg et al. and recognize a significant difference.", "limitations": "Limitations are discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper proposes a theoretically motivated method for the detection of harmful distribution shifts with no labels at test time;\n- The paper discusses the conditions under which the method controls Type-I errors;", "weaknesses": "- In my view, there are two main weaknesses in this paper:\n  1. The error prediction function is a function of the covariates X. That implies that this method only works in detecting harmful shifts that involve shifting the distribution of X (e.g., covariate shifts). If the shifts happen only in P_{Y|X}, instead, the method would have low power. However, this limitation is not discussed by the authors (Please let me know if I have any misunderstanding here). If this is true, the authors should compare their work with [1], because it's not clear how their work is different in nature from that work. If it turns out that both works have a similar nature, the new method would have to be superior in terms of performance;\n  2. In the experiments, the authors do not compare their method with any other competitors. You could, for example, compare your method with covariate shift detection methods [1,2] (which do not rely on the presence of labels).\n\nReferences:\n\n[1] Ginsberg, Tom, Zhongyuan Liang, and Rahul G. Krishnan. \"A learning based hypothesis test for harmful covariate shift.\" arXiv preprint arXiv:2212.02742 (2022).\n\n[2] Polo, Felipe Maia, Rafael Izbicki, Evanildo Gomes Lacerda Jr, Juan Pablo Ibieta-Jimenez, and Renato Vicente. \"A unified framework for dataset shift diagnostics.\" Information Sciences 649 (2023): 119612.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper and for providing valuable references that we were not aware of. We appreciate the opportunity to clarify our contributions in light of these references.\n\nFirst, we note that paper [2] focuses on detecting distribution shifts in general and not specifically harmful shifts, unlike our method and the first suggested paper [1]. Consequently, using [2] as a benchmark in cases where there are harmful/benign shifts would be unfair since it does not differentiate between harmful and benign shifts and would likely result in a much higher rate of false alarms. Therefore, the first suggested paper is more relevant to our work as it directly targets harmful shifts.\n\nHowever, we must highlight that comparing our method directly with those in [1] is challenging due to fundamental differences in setup. The methods in [1] are designed for an offline setup, requiring a batch of production data to train their statistics and raise alarms for a batch of production data. In contrast, our methodology was designed for an online setup where shifts occur gradually and continuously, necessitating real-time decisions without observing an unlabelled batch of examples first. Our methodology aims to detect harmful performance shifts on the fly as observations are received one by one and do not require access to the full production data to fit the statistic.\n\nAdditionally, applying offline methods in an online setting would be impractical and unfair, as these methods require learning a model or statistics from a batch of data. It would also be computationally expensive, as [1] requires learning a new model for each batch of production data. In this case, we would have to learn a number of models proportional to the size of the production data.\n\nHowever, we evaluated our method against the best method from [1] (Detectron) in a batch setting, increasing the size of production data or out-of-distribution (OOD) data. We generated the shifts similar to the setting described in Section 5.2 and ensured no shift in the first 1300 samples of the production data. We used NHANESI classification data as Detectron is only available for classification tasks. We replicated the experiments 50 times, resulting in a total of 10200 different shifts.\n\nOur results, summarized in the table below, show that for smaller sample sizes (100 and 1000), our method did not detect any shifts, which is expected since no shift occurred in the initial 1300 samples, while Detectron raised a significant number of alarms (1126 out of 1700 shifts for size 100 and 1493 for size 1000), all of which were false alarms. For larger sample sizes, while the method of [1] shows high power in detecting shifts, it also exhibits high false alarms. In contrast, our method demonstrates lower power but significantly better FDP control, aligning with our goal of minimizing false alarms.\n\nThese results validate that our method performs robustly in a batch setting, as well as working effectively in the online setting considered in the paper.\n\n| OOD Size | Power Detectron | FDP Detectron        | Power SHSD | FDP SHSD |\n|----------|-----------------|----------------------|------------|----------|\n| 100      | N/A             | 1                    | N/A        | 0        |\n| 1000     | N/A             | 1                    | N/A        | 0        |\n| 2000     | 0.96            | 0.61                 | 0.40       | 0.02     |\n| 3000     | 0.98            | 0.60                 | 0.63       | 0.02     |\n| 3500     | 0.98            | 0.60                 | 0.67       | 0.02     |\n| 8593     | 0.98            | 0.60                 | 0.74       | 0.04     |\n\nWe hope this clarifies the distinctions and the rationale behind our methodological choices. We will also clarify these differences and include additional experiments demonstrating them in the final version of the paper. Thank you again for your insightful feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkwdkpfpmd", "reviewer": "Reviewer_FDg9", "summary": "The paper introduces a sequential drift detector designed to identify drifts that may negatively impact model performance without requiring target labels in production. This is achieved by training an error proxy model on a calibration dataset, which can then be applied in a production scenario, thereby eliminating the need for target labels. While the practical benefits of this method are evident, its performance is initially worse than the reference method that uses labels. This issue is addressed by proposing a modification to the scoring metric, making the performance of both methods comparable. Although the method that uses labels detects drifts earlier, the label-less method now demonstrates similar performance. The methods are tested and compared on multiple synthetic and real-world datasets.", "questions": "### Line 7:\n\n- Is \"learnt\" the old variation of \"learned\" or is there a specialized meaning to it?\n\n### Line 46:\n\n- Does the predictor for the model error serve as an approximation of the conditional probability to produce a higher error in areas of the feature space with sparse or contradictory behavior?\n- Would the representation of this not be best captured by a Gaussian Process model?\n- You do not mention the model you used to produce your results. (Or do you always use the same model for the main prediction and the error prediction, if yes, why?)\n\n### Line 74:\n\n- In which domain do X and Y reside?\n- What are we trying to predict given what kind of information?\n- The current specification does not allow the assumption of a suitable bounded error function. Specifying this kind of error for regression problems proves to be very difficult and highly dependent on the application context, for example.\n\n### General Question on DDM and PHT:\n\n- The DDM (Gama, J., Medas, P., Castillo, G., Rodrigues, P. (2004). Learning with Drift Detection. In: Bazzan, A.L.C., Labidi, S. (eds) Advances in Artificial Intelligence – SBIA 2004. SBIA 2004.) and PHT (Page, E.S.: Continuous inspection schemes. Biometrika 41(1-2), 100–115 (1954). DOI 10.1093/biomet/41.1-2.100. and Gama, J., Sebastião, R. & Rodrigues, P.P. On evaluating stream learning algorithms. Mach Learn 90, 317–346 (2013). https://doi.org/10.1007/s10994-012-5320-9) drift detection tests are frequently cited and also track the development of the mean model error over time and restrict its allowed deviation based on sequential analysis. How does this compare to your work, and what are the differences?\n\n### Line 186:\n\n- Which type of R² score is used here? What is the reference model?\n- How is the R² score used in the production scenario, as it is batched based the application on sequential data points is not natively defined.\n- Against your assumption in section 2, the R² score is not bounded! Depending on the formulation, it can have large positive values but be bounded to larger than 0 or have large negative values but be always smaller than 1. R² is only bounded if it is viewed as a data descriptor that assumes that a linear model is used to describe the data, then it is a measure for the explainable variance. This specification confuses more than it help. The R²  formulation you use should be clearly specified in mathematical notation.\n\n### Equations 12 to 15:\n\n- Are you referring to joint probabilities or conditional probabilities? So, S(X)=1 and E>q or S(X)=1 given E>q?\n\n### Line 235:\n\n- If the Hoeffding interval is based on the Hoeffding Inequality, does your application fulfill all stated requirements?\n- With special focus on the R² Score you use as your “bounded” metric, Theorem B.1 seems to have unfulfilled assumptions.\n\n### Section 5.1:\n\n- What error metric is used for the stated experiment? Accuracy?", "limitations": "The limitations are mentioned and discussed.\n\nThe assumptions used for some proves are however not discussed into detail. Further Details in Questions and Weaknesses", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\n\n- **Novel Approach:** The introduction of a sequential drift detector that does not require target labels in production is a welcomed addiotion in the field of drift detection.\n- **Error Proxy Model:** The use of an error proxy model trained on a calibration dataset to identify drifts is a powerful contribution that addresses the challenge of label scarcity in production environments. This approach also opens up the possibility to use specialized error proxy models in the future to extend the approach.\n\n### Quality\n\n- **Thorough Evaluation:** The methods are rigorously tested and compared on multiple synthetic and real-world datasets, providing a comprehensive evaluation of the proposed approach.\n\n### Clarity\n\n- **Clear Explanation:** The paper clearly explains the methodology, including the training of the error proxy model and its application in production scenarios.\n- **Well designed mathematical Notation**: The provided notation is intuitive and provides detailed introspection into the newly proposed parts of the method.\n\n### Significance\n\n- **Practical Relevance:** The method's ability to function without target labels in production is highly relevant for real-world applications where obtaining labeled data is often challenging.\n- **Comparable Performance:** By achieving performance comparable to traditional methods that use labels, the proposed method offers a valuable alternative for drift detection in label-scarce environments.", "weaknesses": "### Line 17-19:\n\n- Domain knowledge regarding performance-based drift detection methods is not mentioned.\n- The claims regarding the properties of “traditional” drift detection methods are unfounded and can be seen as misinformation.\n- For further references the survey \"Learning under Concept Drift: A Review” of  Jie Lu, Fellow, IEEE, Anjin Liu, Member, IEEE, Fan Dong, Feng Gu, Jo˜ao Gama, and Guangquan Zhang provides a structured overview of different detection mechanisms and differences.\n- In addition, the use of a reference window is presented as a weakness, but the proposed method also uses a calibration set, which can be seen as a reference window.\n\n### Line 134:\n\n- Phrases like “If sufficient”, “Should align”, “would achieve” involve a lot of assumptions.\n- If the error estimator is a statistical model, it would open this expression up to further introspection.\n- While the statements proposed by the work seem solid, sections where these subjective claims are mentioned (e.g., here and the beginning of section 2) are very loose and are left unanswered.\n\n### Line 193:\n\n- The FDP plot has no blue in it; what does this sentence refer to?\n\n### Equations 10 and 11:\n\n- These equations confuse more than they clarify.\n- A new form is introduced just to show how it can be directly inferred from Equations 1 and 2. This, by itself, would be fine, but then the notation is changed again directly.\n- Why not go from Equations 1 and 2 directly to Assumption 4.1?\n\n### Experimental Reproducibility:\n\n- While the data is well explained, the configuration of the experiments is left up to interpretation.\n- Access to the scripts used to run the experiments would highly benefit reproducibility.\n- While the paper checklist mentions that the code will be provided, this is currently not realized, making it impossible to include it in the review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper and acknowledging our contributions.\n\n> The claims regarding the properties of “traditional” drift detection methods are unfounded and can be seen as misinformation.\n\nRegarding the issue raised about traditional shift methods (Lines 17-19), specifically their inability to distinguish harmful from benign shifts and their inflated false alarm rates, our assertions are based on the findings of Podkopaev and Ramdas (2022). Their analysis in Appendix A (\"Issues with existing tests for distribution shifts/drifts\") of their paper supports our claims. We will cite this appendix directly in the final version of our paper.\n\n> Phrases like “If sufficient”, “Should align”, “would achieve” involve a lot of assumptions.\n\nYes, we agree. In Line 134, we were describing the naive approach of using a plug-in error estimator. This served as a prelude to introducing our strategy, which aims to achieve good performance using an imperfect proxy for model performance.\n\n> If the error estimator is a statistical model, it would open this expression up to further introspection.\n\nWe agree. Additionally, as the reviewer highlighted, our approach allows for the use of any proxy, not only error estimation. For instance, in classification problems, we can use calibrated probabilities instead of the prediction of the error estimator as a proxy before our calibration step. We have tested this approach and found that it demonstrated similar performance to the error estimator.\n\n> The FDP plot has no blue in it; what does this sentence refer to?\n\nThe variance of the distribution was so small that it appeared as a line, making the blue colour not visible as in the production data case. We will clarify this in the final version.\n\n> Why not go from Equations 1 and 2 directly to Assumption 4.1?\n\nWe will simplify the presentation by going directly from Equations 1 and 2 to Assumption 4.1.\n\n> Is \"learnt\" the old variation of \"learned\" or is there a specialized meaning to it?\n\nYes, it is the British variant, and they both have the same meaning.\n\n> Does the predictor for the model error serve as an approximation of the conditional probability to produce a higher error in areas of the feature space with sparse or contradictory behavior? Would the representation of this not be best captured by a Gaussian Process model?\n\nYes, exactly. It is an implicit way of defining the regions where the model performs poorly. Particularly in the case of tabular data, we have considered tree-based models as they are well-known to work best. In addition, it opens the possibility of examining the leaves of the tree to define hyper-rectangular region. These regions can be represented as rules, providing valuable insights to users about which covariate spaces cause the model to perform poorly.\n\n> You do not mention the model you used to produce your results. (Or do you always use the same model for the main prediction and the error prediction, if yes, why?)\n\nWe use the same class of model and hyper-parameters by default to avoid biasing the results by explicitly fine-tuning the error estimator. However, the objective is to find the best possible error estimator, and users can choose any model that suits their problems.\n\n> In which domain do X and Y reside?\n\nIn our experiments, the domains of X and Y have been $\\mathbb{R}^p $ and $\\mathbb{R}$, with $p$ being the number of variables. The continuity or discreteness of the data is not important. It can be anything, as long as a bounded score value is attributed to each combination of prediction and true labels.\n\n> Which type of R² score is used here? What is the reference model? How is the R² score used in the production scenario, as it is batched based the application on sequential data points is not natively defined. Against your assumption in section 2, the R² score is not bounded! but be always smaller than 1. R² is only bounded if it is viewed as a data descriptor that assumes that a linear model is used to describe the data, then it is a measure for the explainable variance. This specification confuses more than it help. The R² formulation you use should be clearly specified in mathematical notation.\n\nWe use the classical R² score from the scikit-learn library, defined as $\\( R²(y, \\hat{y}) = 1 - \\frac{\\sum (y - \\hat{y})^2}{\\sum (y - \\bar{y})^2} \\)$. However, this is not what we monitor in our experiments. Throughout the paper, we only use the R² score to describe the general performance of the error estimator. In production, the loss we monitor and update sequentially is the absolute residual $\\( \\ell(X, Y) = |y - \\hat{y}| \\)$. We apologize for the confusion and will make it clearer that the monitored loss function is the absolute distance $\\( \\ell(X, Y) \\)$, computed with an output Y normalized between (0, 1).\n\n> If the Hoeffding interval is based on the Hoeffding Inequality, does your application fulfill all stated requirements?\n\nFor both confidence sequences and intervals, the only assumption we need is the boundedness of the loss. In all our experiments, we ensure the score is bounded by using the absolute distance with bounded output.\n\n> What error metric is used for the stated experiment? Accuracy?\n\nWe use the absolute distance between the predicted probability and the label.\n\n> General Question on DDM and PHT\n\nThe main differences between these methods and ours are that they require access to ground truth labels to monitor performance, whereas our method does not.\n\n> Are you referring to joint probabilities or conditional probabilities? So, S(X)=1 and E>q or S(X)=1 given E>q?\n\nWe were referring to the joint probability.\n\n> Domain knowledge regarding performance-based drift detection methods is not mentioned.\n\nWe will include it along with the suggested references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7ZcbTry4yT", "reviewer": "Reviewer_MiPF", "summary": "The authors introduce an approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. They solution substitute true errors with the predictions of a learnt error estimator.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work is the first to propose a principled method for detecting harmful distribution shifts without requiring true label.", "weaknesses": "It is worth noting that while I am familiar with distribution shifts, I do not know  continuous production environments is limited. I will be evaluating this paper from the perspective of an outsider in this field, and will rely more on the feedback of other reviewers.\n**************************************\nI encourage the AC to bring in reviewers who are familiar with this field.\n\nI encourage the AC to give less weight to my review comments.\n**************************************\nWeaknesses:\n\n1) I suggest that the authors provide a section on related work concerning distribution shifts and continuous production environments, to help readers clearly understand the motivation behind this paper. To my knowledge, the Out-of-Distribution (OOD) field’s research on Dataset Shift [1-4] also discusses distribution shifts; please include a discussion on this. The introduction does not well introduce the research questions of this paper, especially considering that this paper has not yet reached the length limit.\n\n2) My major concern about this paper is whether “requires no access to ground truth data labels” is a worthwhile background setting to study, especially since Podkopaev and Ramdas [2022] have already satisfactorily addressed more common scenarios. Considering that the method proposed by the authors is relatively straightforward: training an error estimation model to predict the performance of the primary model on production data.\n\n[1] Puli et al., Don’t Blame Dataset Shift! Shortcut Learning Due to Gradients and Cross Entropy, in NeurIPS 2023.\n\n[2] Huang et al., Harnessing Out-of-Distribution Examples via Augmenting Content and Style, in ICLR 2022.\n\n[3] Silva et al., The Value of Out-of-Distribution Data, in ICML 2023.\n\n[4] Yang et al., Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning, in NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper.\n\nWe appreciate the suggestion to include a dedicated section on related work concerning distribution shifts and continuous production environments. Regarding the question of whether \"requires no access to ground truth data labels\" is a worthwhile background setting to study, we believe it is highly relevant. In many real-world environments, ground truth labels are either unavailable, delayed, or expensive to obtain. Examples include:\n\n* **Healthcare:** (Raravian et al., 2015, Population-level prediction of type 2 diabetes from claims data and analysis of risk factors) \n* **Insurance:** (Noodhun et al., 2018, Risk prediction in the life insurance industry using supervised learning algorithms) \n* **Predicting Future Outcomes:** (Zhang et al., 2019, Time-aware adversarial networks for adapting disease progression modeling) \n* **High Collection Costs:** (Lieu et al., 2022, Deep unsupervised domain adaptation: A review of recent advances and perspectives)\n\nRegarding the remarks about our method, it is not solely about learning an error estimator. The cornerstone of our method is the calibration step that leverages even an imperfect estimator to achieve performance comparable to methods that have access to ground truth labels. This is particularly useful since it is often unrealistic to expect a highly accurate error estimator in many practical scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for detecting harmful shifts without having access to ground truth labels. The reviewers agree that the problem is well motivated, and the main contribution of the paper, allowing to handle the lack of access to ground truth labels, makes the contribution quite practical. The techniques are noted to be novel and insightful, as well as motivated from mathematical derivations. There were some issues raised regarding the comparison with existing works (see the baseline mentioned by KyAy and related works mentioned by 4vCN), but this was mitigated by the authors’ response, and the required modification to the paper is small enough to be achievable towards the final version. Concluding, the paper provides a novel solution to an important problem, and would be a welcome addition to Neurips.
Decision: Accept (poster)