Sequential Harmful Shift Detection Without Labels

Authors: salim i amoukou
tom  bewley
saumitra  mishra
freddy  lecue
daniele  magazzeni
manuela  veloso
j p morgan
a i research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 16:30:47

Abstract: We introduce a novel approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. It builds upon the work of Podkopaev and Ramdas [2022], who address scenarios where labels are available for tracking model errors over time. Our solution extends this framework to work in the absence of labels, by employing a proxy for the true error. This proxy is derived using the predictions of a trained error estimator. Experiments show that our method has high power and false alarm control under various distribution shifts, including covariate and label shifts and natural shifts over geography and time.

Main paper: 
 introduction: 
 When deploying a machine learning model in production, it is common to encounter changes in the data distribution, such as shifts in covariates [Shimodaira, 2000] , labels [Saerens et al., 2002 , Lipton et al., 2018] or concepts [Gonçalves Jr et al., 2014] . Many methods exist for detecting such distribution shifts. However, a distinct but equally important challenge is assessing whether a shift has a harmful impact on the prediction error of a given model, which may necessitate interventions such as ceasing production or retraining the model. Not all distribution shifts are harmful, but traditional methods for shift detection are unable to distinguish harmful and benign shifts.While some approaches address the specific issue of performance shift, most require access to ground truth data labels in the production environment [Gama et al., 2013 , 2014 , Bayram et al., 2022] . In scenarios where predictions concern future outcomes, such as medical diagnosis or credit scoring, immediate access to labels in production is not feasible. This work focuses on the challenge of detecting harmful distribution shifts -those that increase model error in production -without requiring access to labels. As Trivedi et al. [2023] note, current methods for harmful shift detection without labels rely on disparate heuristics, often lacking a solid theoretical foundation. Such methods include proxies based on aggregate dataset-level statistics [Deng and Zheng, 2021] , optimal transport mappings between training and production distributions [Koebler et al., 2023] , and model-specific metrics such as input margins [Mouton et al., 2023] , perturbation-sensitivity [Ng et al., 2023] , disagreement-metrics [Chen et al., 2023 , Ginsberg et al., 2022] , and prediction confidence [Guillory et al., 2021 , Garg et al., 2022] . While such methods may be practically effective in certain contexts, they rely on assumptions and correlations that do not hold universally, so can provide no guarantees.Furthermore, conventional methods rely on two-sample or batch testing, which involves comparing the statistical properties of a production dataset with those of a control sample. These methods have inherent limitations, as the sample size is prespecified. This is a problem because the necessary amount of data to detect any given shift is unknown beforehand. Furthermore, in real-world scenarios, data typically arrive sequentially over time and shifts may occur either suddenly or gradually. In such scenarios, it may be desirable to detect harmful shifts as early as possible. Batch testing is ill-suited to the sequential context [Maharaj et al., 2023] , as it does not accommodate the collection of additional data for retesting without adjusting for multiple testing, leading to diminished power.The most principled and relevant work to our problem is that of Podkopaev and Ramdas [2022] , which tackles the problem of sequential harmful shift detection with false alarm control but assumes the availability of ground truth labels in production. Our work builds on the foundation established by Podkopaev and Ramdas [2022] , extending the methodology to detect harmful shifts in unlabeled production data while effectively managing false alarms.Our approach leverages a secondary model to estimate the errors of the primary model. While learning such a model might seem challenging at first, consider a situation where the primary model performs well overall but struggles with specific data subgroups. Sagawa et al. [2019] demonstrate that this phenomenon can occur in natural distributions. In such cases, learning to predict "error given X" might be easier than the primary task of predicting "Y given X", because the error estimator only needs to identify those subgroups where the primary model struggles. This approach has shown promise in recent studies [Zrnic and Candès, 2024, Amoukou and Brunel, 2023] . More generally, Zrnic and Candès [2024] note that predicting the magnitude of the error, rather than its direction, is often easier. Furthermore, our approach is based on estimating the proportion of high-error observations over time. For this task, the error estimator does not need to be very accurate; it only needs to assign higher values to observations with higher errors. That is, the estimator only needs to correctly order most observations from low to high error, which is easier than precisely predicting the error itself. We demonstrate in Section 4.1 that even a relatively inaccurate error estimator can be effective at identifying high-error observations, and thus provides the functionality required by our framework. Although this paper uses a learned error estimator's predictions as a proxy for error, we note that any scalar function correlated with error could suffice to isolate high-error observations. For example, for a well-calibrated binary classification model, we could instead use that model's predicted probability, tracking observations with predictions near 0.5 to identify uncertain predictions. Left: calibrating an estimated error threshold to separate low/high true errors. Right: sequentially tracking production data exceeding the threshold and raising an alarm upon a significant increase.Figure 1 gives an overview of our approach. We first fit the secondary error estimator model to predict the error of the primary model, then use labeled data to calibrate an estimated error threshold (---) that separates observations with low (x) and high (x) true error as fully as possible. We run the error estimator on all observations encountered in production and continually monitor the proportion of observations whose estimated error falls above the threshold. We raise an alarm when this exceeds the rate of high-error observations (x) in the calibration set plus a tolerance threshold ϵ tol and correction terms to deal with the sequential setting and account for uncertainty in the estimates. In the example shown, this occurs at time t = 10.The rest of this paper is organized as follows. Section 2 outlines the problem definition and Section 3 provides an overview of the foundational work of Podkopaev and Ramdas [2022] as background. Section 4 is dedicated to the presentation and theoretical analysis of our sequential statistical test. Section 5 demonstrates the empirical efficacy of our method, showcasing its strong detection capabilities and controlled false alarm rates across various types of harmful shift. 
problem definition: 
 Let X and Y be input and label spaces, f : X → Y be a predictive model, and ℓ : Y 2 → E be a measurable and bounded error function that is selected for monitoring purposes. The model's error on a specific observation (X, Y ) ∈ X × Y, drawn from a joint distribution P (X,Y ) = P X P Y |X , is represented by the random variable E = ℓ(f (X), Y ). The probability distribution of the error is denoted by P E . As discussed above, our focus is not on detecting shifts in covariates or labels per se, but rather changes in the error distribution P E . Error changes can be caused by various types of shift in the underlying joint distribution, including changes in P X while the conditional label distribution P Y |X remains constant (covariate shift) or changes in P Y while P X|Y remains constant (label shift).We assume access to a dataset D n = {(X 0 i , Y 0 i )} n i=1 , sampled independently from a source distribution P 0 (X,Y ) . In addition, we have a sequence of data (X t , Y t ) t≥1 drawn independently from a time-varying distribution encountered by the model in production, P t (X,Y ) . We model the ocurrence of a shift in production by assuming this distribution is equal to the source before some time T ∈ N ∪ {∞} (i.e., P t (X,Y ) = P 0 (X,Y ) , ∀t < T ) and different thereafter (i.e., P t (X,Y ) ̸ = P 0 (X,Y ) , ∀t ≥ T ). Our goal whenever there is a shift, (i.e., T < ∞), is to decide if this shift is harmful to the model error.To formalize this, we introduce θ : P(E) → R + as a mapping from probability distributions on the error space E to a real-valued parameter. This mapping could, for instance, map the distribution to its mean or a certain quantile. We aim to construct a sequential test for the following pair of hypotheses:H 0 : ∀t ≥ 1, 1 t t k=1 θ(P k E ) ≤ θ(P 0 E ) + ϵ tol ;(1)EQUATIONwhere P k E denotes the error distribution at at time k, the running risk1 t t k=1 θ(P k E )is the average value of the error parameter up to time t, and ϵ tol ≥ 0 is a tolerance level. Intuitively, H 0 holds if the running risk remains below that of the source distribution (+ϵ tol ) for all time throughout production, and H 1 holds if this condition is violated.Objective. Construct a α-level sequential test, defined by an alarm function Φ : ∪ ∞ k=1 X k → {0, 1}, which at time t uses the first t observations X 1 , . . . , X t to output 0 (no harmful shift so far) or 1 (harmful shift; raise an alarm) with a controlled false alarm rate and high power, i.e., P H0 (∃t ≥ 1 : Φ(X 1 , . . . , X t ) = 1) ≤ α, andP H1 (∃t ≥ 1 : Φ(X 1 , . . . , X t ) = 1) ≈ 1. (3)We refer to this problem definition as sequential harmful shift detection (SHSD).Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: On lines 40-51, we discussed the potential difficulty of learning a second estimator and provided rationale and examples of when it is possible. We presented the assumptions under which our method should have false alarm control and discussed its validity in practice in appendix A.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
shsd with production labels: 
 A work closely related to ours is that of Podkopaev and Ramdas [2022] , which offers a solution for scenarios where the ground truth labels of the production data are available. This method leverages confidence sequences [Darling and Robbins, 1967 , Jennison and Turnbull, 1984 , Johari et al., 2015 , Jamieson and Jain, 2018] , which are time-uniform (i.e., valid for any time) confidence intervals, allowing for the ongoing monitoring of any bounded random variable. With access to labels, it is possible to calculate the true errors on the production data over time and monitor the running risk.Choosing the mean as the error parameter i.e. θ( Podkopaev and Ramdas [2022] use the empirical production errors E 1 = ℓ(f (X 1 ), Y 1 ), . . . , E t = ℓ(f (X t ), Y t ), to construct a confidence sequence lower bound L for the running risk, satisfying a chosen miscoverage level α prod ∈ (0, 1):P k E ) = E P k [E],EQUATIONThis equation guarantees that the lower bound remains valid over time with high probability. Furthermore, given the errors on the source dataE 0 1 = ℓ(f (X 0 1 ), Y 0 1 ), . . . , E 0 n = ℓ(f (X 0 n ), Y 0 n ), either another confidence sequence or a traditional confidence interval method [Howard et al., 2021, Waudby-Smith and Ramdas, 2020 ] can be used to construct a fixed-time upper confidence bound Û for the mean error θ(P 0 E ). For a miscoverage level α source ∈ (0, 1), Û satisfies the following condition:EQUATION)An alarm is raised when the lower bound of the running risk in production exceeds the upper bound of the source error plus a tolerance ϵ tol . Formally, this equates to defining the function Φ as follows:EQUATIONwhere the subscripted m denotes that the mean is the error parameter being tracked. This methodology provides uniform control over the false alarm rate across time, i.e.,EQUATIONIt also makes no assumptions about the data distribution or the type of shift. However, the reliance on immediate access to ground truth production labels at each time t limits the method's practical applicability. We now propose a solution that avoids the need for production labels.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss broader impacts in Section H. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
sequential harmful shift detection without production labels: 
 This section consists of two subsections, each detailing one of the two stages of our proposal. The initial stage consists of fitting an error estimator and calibrating it to identify high-error observations with few mistakes. Following this, we apply confidence sequence methods to track the proportion of high errors over time in production, and develop a test for raising an alarm based on this proportion.The primary drawback of the Podkopaev and Ramdas [2022] method is its reliance on having ground truth labels for the production data, which are often unavailable in real-world scenarios. A straightforward solution is to use a plug-in approach: replace the true error in production with an estimated error obtained from a secondary predictive model, denoted as r : X → E. This model trained to predict the true error of the primary model using any available labeled data. We can then reformulate the alarm function of Equation 6to deal with unlabeled production data as follows:Φm (X 1 , . . . , X t ) = 1 L(r(X 1 ), . . . , r(X t )) > Û (E 0 1 , . . . , E 0 n ) + ϵ tol (8)If r(•) is sufficiently accurate, the performance of this alarm mechanism should align closely with what would be achieved if ground truth labels were available. However, even if the estimator r exhibits strong performance on its training distribution, the absence of labels in production makes it difficult to conclusively determine the alarm's reliability in a shifting production environment.Our strategy to address this issue consists of using a calibration step to derive a more reliable statistic from the imperfect estimator r(•). Specifically, we propose to track the proportion of observations above a carefully-selected quantile of estimated error, rather than the mean value as in the original method of Podkopaev and Ramdas [2022] . The fundamental hypothesis here is that an estimator, even if not particularly accurate at predicting error magnitudes, may still effectively distinguish between the lowest and highest errors across a dataset, thereby preserving most ordinal relationships between observations. For example, if r(•) has correctly represented some underlying patterns to predict the errors, and if k-th and l-th ranked errors are significantly different, then it is highly probable that r(X (k) ) ≤ r(X (l) ). Focusing on the aggregate distinction of low and high errors rather than the prediction of specific magnitudes allows us to utilize an imperfect estimator r more effectively.Our proposed calibration process is as follows. Given the labeled source data D n and a trained error estimator r, we identify an empirical quantile of the true errors, q = Q(p, {E 0 i } n i=1 ), p ∈ [0.5, 1), and an empirical quantile for the estimated errors q = Q(p, {r(X 0 i ) : X 0 i ∈ D n )}, p ∈ (0, 1), such that the selector function S r,q (X) = 1{r(X) > q} reliably distinguishes between observations with true error below and above q. Specifically, we seek to balance the statistical power and false discovery proportion (FDP) of the selector, which are defined as follows: selector power for each (p, p) pair. Green dotted outline indicates the pair that maximises power subject to the FDP < 0.2 limit. Corresponding thresholds (q, q) shown as thick lines in left plot.EQUATIONWe search over a uniform grid of quantile pairs (p, p), compute the associated thresholds (q, q), and identify those that achieve an FDP below a maximum value. Among these qualifying pairs, we select the one that maximizes the power. Figure 2 illustrates this process for a toy example.In this case, thresholds are found that achieve a selector power of 0.72 while keeping FDP below the specified maximum of 0.2.We now present empirical evidence that it is possible to achieve high power and a controlled FDP in realistic settings, using the California house prices [Dua and Graff, 2017] , Bike sharing demand [Fanaee-T, 2013], HELOC [FICO, 2018] and Nhanesi [CDC, 1999 [CDC, -2022] ] datasets. We partition each dataset into training (60%), test (20%) and calibration (20%) sets and use the training data to train random forests (RFs) as the primary models. However, we first ablate the training data in various ways to ensure the models perform poorly on certain subgroups. The ablation is done on a per-feature basis.For continuous features, we exclude 80% of observations with values either above or below the median. For categorical features, we exclude data from one category. We then simulate production environments by gradually reintroducing these previously excluded observations alongside the test set. For each dataset, the number of distribution shifts studied equals the number of features times the number of splits: two for continuous features and the number of categories for discrete ones. We use half of the calibration sets to train RF regressors as the error estimators, then use the remainder to calibrate true and estimated error thresholds using the grid search process described above. In Figure 3 , we present the distribution of the FDP and power across all datasets and shifts, relative to the performance of the error estimator, as measured by the R-squared score on the source/calibration data. The R-squared score is binned into quantiles, with 10 bins used. We observe that the estimators are generally highly imperfect, with R-squared values consistently below 0.3. Despite these low predictive accuracies, we can still find threshold pairs that achieve an FDP below 0.2 in the source data (shown next to the red boxplot). The power ranges from 0.4 for the least accurate estimators to 0.9 for the most accurate. Crucially, when we apply the calibrated thresholds in the production environments, we achieve similarly low FDP values (shown in red), almost always below 0.25 (though some reach 0.4), while the power remains similar to the source data, ranging from 0.4 to 0.9. This consistency of the FDP/power even when error estimators are not particularly accurate is promising for shift detection.We can now state the specific objective of our sequential testing framework. During production, we propose to test if there is an increase in the proportion of observations exceeding the true error quantile q obtained in calibration. This is formalized in terms of the following hypotheses:EQUATIONEQUATIONwhere P P k denotes a probability taken under distribution P k . Note that this is a special case of the general test in Equations 1 and 2, with the probability θ(P k E ) = P P k (E > q) as the error parameter. Since we cannot observe production errors directly, we use the selector function S r,q (X) as a proxy for a check on the true error E > q. The effectiveness of the sequential test under this substitution depends on how well the selector's power and FDP properties generalize from the source distribution to the production environment. In particular, we can show that the method outlined below provably controls the false alarm rate given in Equation 3 if the following assumption holds:Assumption 4.1. ∀ t ≥ 1, 1 t t k=1 P P k (S r,q (X) = 1, E ≤ q) ≤ P P 0 (S r,q (X) = 1, E ≤ q).Referring back to the example in Figure 2 (left), this assumption implies that at all times during production, the proportion of data observed so far falling the quadrant above and to the left of the calibrated thresholds (---) does not exceed that observed under the source distribution. While we do not claim that this assumption always holds exactly, we find that it is only violated to a small extent in realistic settings (see Appendix A for more discussion and experimental analysis). If this is the case, and thresholds (q, q) have been found that yield a small number of false discoveries in calibration, P P 0 (S r,q (X) = 1, E ≤ q), then the number of false discoveries in production will also remain low. A substantial increase in false discoveries in production would require a shift specifically targeting those rare observations with low error but high estimated error.With this foundation established, we can now describe our testing methodology. Following a similar approach to that used by Podkopaev and Ramdas [2022] , we construct:1. A lower bound of 1 t t k=1 P P k (E > q) using a confidence sequence.2. An upper bound of P P 0 (E > q) using a traditional confidence interval.To construct the lower bound, we rewrite the target quantity as follows:EQUATION)The last inequality uses Assumption 4.1 to substitute the probability of a false discovery in production with the probability on the source. As we can empirically estimate both P P k (S r,q (X) = 1) and P P 0 (S r,q (X) = 1, E ≤ q) (via the labeled source data D n ), we can use a confidence sequence to construct a valid time-uniform lower bound of their sum. Specifically, we define the bound Lq asEQUATIONwhere w t and w n are the widths of the lower and upper bounds of 1 t t k=1 P P t (S r,q (X) = 1) and P P 0 (S r,q (X) = 1, E ≤ q) with miscoverage levels α 1 and α 2 respectively, such that for a total miscoverage level α prod = α 1 + α 2 ∈ (0, 1),EQUATION)The specific values of w t and w n used in our experiments are given in Appendix B. Respectively, these choices correspond to the predictably-mixed empirical-Bernstein (PM-EB) confidence sequence described by Podkopaev and Ramdas [2022] , and the classic Hoeffding interval.We similarly compute an upper bound Ûq for P P 0 (E > q) as follows:EQUATIONwhere w n is the same as above. This bound satisfies a miscoverage level α source ∈ (0, 1), such thatEQUATIONFinally, we define our sequential test using the following alarm function:EQUATIONwhere the subscripted q denotes that we are now detecting shifts in error across a particular quantile, rather than the mean. In Appendix C, we provide a proof of the following statement:Theorem 4.2. Under Assumption 4.1, Lq and Ûq satisfy Equations 17 and 19. Therefore, the function Φ q has false alarm control, i.e.,EQUATIONWhile a controlled false alarm rate is a desirable property, the power of Φ q may be limited if the degree of error change is not large. Noting that (1/t)t k=1 P P k (E > q) is lower-bounded by (1/t) t k=1 P P k (S r,q (X) = 1, E > q), detecting a change requires this probability to exceed P P 0 (E > q). Thus, we also propose to compare 1 t t k=1 P P k (S r,q (X) = 1, E > q) directly with P P 0 (S r,q (X) = 1, E > q). This leads to a second test with higher power. It uses an upper bound of P P 0 (S r,q (X) = 1, E > q), defined as:EQUATIONsatisfyingEQUATION)The alarm function for the second test is defined as:EQUATIONThrough an almost identical proof, we can similarly show that Φ 2 q also has false alarm control for comparing (1/t) t k=1 P P k (S r,q (X) = 1, E > q) with P P 0 (S r,q (X) = 1, E > q). 
experiments: 
 In this section, we compare the performance of the plug-in approach of Podkopaev and Ramdas [2022] 's method (Equation 8), which is designed to detect a change in the mean error, and our approach, which determines if an increasing number of observations fall beyond a certain quantile. We focus on the second test (Equation 24) to simplify the comparison with the mean detector and because it consistently outperforms the first statistics. Results for the first test are reported in Appendix E. We conduct three experiments using a variety of datasets and setups. The first experiment aims to illustrate the different approaches and demonstrate the applicability of our method to image data and deep learning models. The second experiment returns to the tabular datasets studied in Section 4.1, going into more detail by comparing the mean and quantile detection approaches in terms of power and FDP on the numerous generated shifts. The final experiment also consists of a large-scale evaluation of the approaches, in this case on natural shifts due to temporal and geographical changes.Although the focus of this paper is on the sequential or online setting, we provide an analysis using state-of-the-art methods in the batch setting in Appendix F.The first experiment replicates the setup of Saerens et al. [2002] using the CelebA dataset [Liu et al., 2015] . They demonstrate that a ResNet50 model [He et al., 2016] trained on this dataset performs poorly on "males with blond hair" due to spurious correlations. We split this dataset into a training set (60%), test set (20%) and calibration set (20%), and train a ResNet50 on the training set. Using half of the calibration set, we train another ResNet50 (with a regression head) as an error estimator. The remaining half is employed to determine the empirical quantiles p ∈ [0.5, 1), p ∈ (0, 1) at which we achieve maximum power while keeping the FDP below 0.2. We create a harmful shift in production as follows. For each time step up to t = 4990, we sample an observation uniform-randomly from the test set. Thereafter, we begin to oversample instances of males with blond hair, sampling such an observation with probability β t = 1/(1 + exp(-(t -4990))), and a random observation otherwise.The objective of this experiment is to visually observe how the methods can be used to monitor performance shift over time and to evaluate how each method compares to an idealised version with access to true production errors. Both Podkopaev and Ramdas [2022]'s method (mean detector) and our approach (quantile detector) involve comparing a lower bound to an upper bound. For both methods, Figure 4 displays the lower bound in blue and the version calculated with true production errors in gray. For the quantile detector, the blue line corresponds to Lq of Equation 16, which is the estimated lower bound of 1 t t k=1 P P k (E > q) with estimated production errors. The gray line represents the lower bound of this same quantity, except computed using the true errors. The blue lower bound of the plug-in approach of the mean detector is defined as L(r(X 1 ), . . . , r(X t )). The gray line represents L(E 1 , . . . , E t ), the lower bound of the original mean detector using true errors. The upper bound that needs to be surpassed for each method to raise an alarm is depicted in red. For the quantile detector, this is the second lower bound L2 q , and for the mean detector, it is Û (r(X 1 ), . . . , r(X t )). For the quantile detector, we also plot in pink the upper bound of the first statistic Φ q (Equation 20). The R-squared value of the error estimator on the source distribution is 0.35, which is not especially high. By analyzing the upper bounds for each method in Figure 4 , we observe that all bounds remain roughly constant before the shift starts. Unsurprisingly, we observe the mean detector using true production errors quickly detects the shift (gray line). In contrast, its plug-in version raises an alarm with a significantly delayed detection. For the quantile detector, there is a much smaller difference between the lower bound of the plug-in and the one using true production errors. This observation validates our expectation that the FDP remains relatively stable post-shift. Additionally, as expected, the plot shows that the lower bound of the quantile detector crosses the upper bound of the second statistic (red line) much earlier than that of the first statistic (pink line). This experiment suggests that in scenarios with a less accurate error estimator, targeting quantile changes is more effective for detecting harmful shifts than focusing on mean change. Additional experiments on image datasets confirming this observation can be found in Appendix D. Larger-scale analyses in the following subsections examine the advantages of the quantile detector in more depth.In this section, we conduct a large-scale experiment to evaluate the effectiveness of both methods in detecting harmful shifts while maintaining their ability to control false alarms. We also analyze how these metrics relate to the performance of the error estimator. We use two regression datasets (California house prices and bike sharing demand) as well as two classification datasets (HELOC and Nhanesi). We follow the feature-splitting setup of Section 4.1 to generate synthetic distribution shifts, excluding splits that result in subsets with fewer than 10 observations, and repeat each split 50 times with different random seeds. Table 1 shows the number of generated shifts and the number of harmful shifts detected by each method using the true errors (H-M for mean detector and H-Q for quantile detector).A shift is considered harmful by each method as soon as the lower bound exceeds the upper bound plus ϵ tol = 0.The left plot of Figure 5 displays the aggregated results across all distribution shifts for mean detection (red) and quantile detection (green) on the different datasets. The points labeled "all-[method]" represent the average results across the datasets. The quantile method achieves a significantly better power-FDP balance: (power 0.83, FDP 0.11) compared to the mean method: (power 0.67, FDP 0.41) across all experiments. An exception is observed for the Nhanesi dataset, where the mean detection shows slightly better power. However, overall, the quantile detection demonstrates a superior trade-off between power and false alarms. A similar trend is observed in the middle plot, which analyzes the absolute difference in detection time between each method using estimated errors and the same method with access to true errors. In the right plot, we compute how the power across datasets varies when we increase the threshold at which we consider the true shift as harmful (ϵ tol ). Across varying intensities of shift, the quantile detector consistently outperforms the mean detector, with false alarm rates at ϵ tol = 0 being 0.41 and 0.11, respectively. In Figure 6 , we further investigate the relationship between the power (top row) and FDP (bottom row) of each method and the error estimator's performance binned into 10 quantiles for each dataset. The error estimator performance, measured by R-squared values, is generally low across all experiments (0.10 -0.26). Notably, the quantile detector consistently maintains a lower FDP compared to the mean detector across all error estimator values. Regarding power, excluding the Nhanesi dataset, the quantile detector performs better than or equal to the mean detector.Figure 6 : Power and FDP by error across all datasets.In the last experiment, we conduct another large-scale evaluation of our approach on natural shifts within the Folktables dataset [Ding et al., 2021] . This dataset is derived from US Census data spanning all fifty states within the US (plus Puerto Rico), each with a unique data distribution. Furthermore, it includes data from multiple years (from 2014 to 2018), introducing a form of temporal distribution shift in addition to the variations between states. We select the income feature as the target label, specifically predicting whether income exceeds $50, 000. We first split the dataset of each state in the year 2014 into training (50%), and calibration (50%). Then, we train a separate RF classifier in each state in the year 2014, and an RF regressor to learn the error of the primary model on the calibration set. Subsequently, we evaluate the model's error on all the remaining 50 states over 5 years, effectively creating 250 production datasets. We consider a shift to be harmful if the model's error in production exceeds the error on the calibration dataset plus ϵ tol = 0. We introduce the shift in all datasets starting at time t = 3300.Table 2 summarizes the results for both methods, demonstrating that the quantile detector consistently outperforms the mean detector across all metrics. Figure 7 plots the sensitivity of each method relative to the shift harmfulness threshold. We observe that the quantile detector maintains superior performance across all threshold values.Overall, this experiment provides good evidence that our proposed method is effective under the kinds of natural shift encountered in realistic production environments. 
conclusion: 
 We have introduced an approach to identifying harmful distribution shifts in continuous production environments where ground truth labels are unavailable. Utilizing a plug-in strategy that substitutes true errors with estimated errors, alongside a threshold calibration step, our method effectively controls false alarms without relying on perfect error predictions. Experiments on real-world datasets demonstrate that our approach is effective in terms of detection power, false alarm control and detection time across various shifts, including covariate, label, and temporal shifts. In future work, we plan to apply interpretability techniques to the quantile detector to understand where and how the data are shifting in the input space, and to use this information to improve the primary model itself. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: [NA] Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates ("J.P. Morgan") and is not a product of the Research Department of J.P. Morgan. J.P. Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. arXiv preprint arXiv:2010.09686, 2020.Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.Tijana Zrnic and Emmanuel J Candès. Active statistical inference. arXiv preprint arXiv:2403.03208, 2024.To formalize the statement in the main body of the paper, we do not expect Assumption 4.1 to hold exactly, but we expect that in realistic settings, for all t ≥ 1, the inequality 1 t t k=1 P P k (S r,q (X) = 1, E ≤ q) ≤ P P 0 (S r,q (X) = 1, E ≤ q) + δ tol holds with a small δ tol . For instance, in Figure 8 , we compute the empirical distribution estimate of δ = 1 t t k=1 P P k (S r,q (X) = 1, E ≤ q) -P P 0 (S r,q (X) = 1, E ≤ q) with t equals to the total number of production data across the different distribution shifts and datasets of Section 5.2 and the natural distribution shifts of Section 5.3. We observe that Assumption 4.1 is valid approximately 50% of the time for both experimental setups, corresponding to the cases where δ is negative. In the other half of the cases where the assumption is not verified, we note that δ is very small, often less than 0.01.It should be noted that Assumption 4.1 allows for controlling false alarms when δ is zero or negative.To control false alarms when δ is positive, it is sufficient to always add δ to the lower bound Lq (Eq. 16) to have the false alarm guarantee. Specifically, under the assumption that for all t ≥ 1, 1 t t k=1 P P k (S r,q (X) = 1, E ≤ q) ≤ P P 0 (S r,q (X) = 1, E ≤ q) + δ we can show, similar to Theorem 4.2, that the corrected bounds Lcorr q = Lq -δ used in the following statistic:EQUATIONwill have false alarm control. The proof is identical to the proof of the Theorem 4.2, with Lq replaced by Lcorr q .However, in practice, we do not know the value of δ. Fortunately, in most realistic cases we have observed, δ is very small, especially compared to our maximum false alarm threshold of 0.2. Therefore, not adding this correction has very little impact on the statistics without correction (Equation 20).In our experiments, the confidence sequence bound w t is that of the Empirical Bernstein confidence sequence, as defined in the Theorem below. For a more detailed presentation of different confidence sequences, we refer the reader to Howard et al. [2021] .Theorem B.1. Let μt = 1 t t i=1 X i , and suppose X i are bounded a.s. for each i ≥ 1. Then, for each α ∈ (0, 1),C t = {θ t ± w t } forms a (1 -α)-level confidence sequence for E(μ t ), where w t = c α √ Vt log log Vt t , Vt = t i=1 (X i -μi-1) 2 denotes an empirical variance term and c α ≍ log(1/α).When we use a confidence interval, we use the classic Hoeffding interval:C n = {μ n ± w n } forms a (1 -α)-level confidence interval for E(μ n ),where w n = log(2/α) 2n .Theorem C.1. Under Assumption 4.1, Lq and Ûq satisfy Equations 17 and 19. Therefore, the function Φ q has false alarm control, i.e.,EQUATIONProof.P H 0 {∃t ≥ 1 : Φq(X1, . . . , Xt) = 1} = P H 0 ∃t ≥ 1 : Lq > Ûq + ϵtol = P H 0 ∃t ≥ 1 : Lq -(1/t) t k=1P P k (S r, q (X) = 1) -Ûq -P P 0 (S r, q (X) = 1)> ϵtol -(1/t) t k=1P P k (S r, q (X) = 1) -P P 0 (S r, q (X) = 1)≤ P H 0 ∃t ≥ 1 : Lq -(1/t) t k=1P P k (S r, q (X) = 1) -Ûq -P P 0 (S r, q (X) = 1) > 0≤ P H 0 ∃t ≥ 1 : Lq -(1/t) t k=1 P P k (S r, q (X) = 1) > 0 + P H 0 Ûq -P P 0 (S r, q (X) = 1) > 0 ≤ αsource + αprodThe last inequality is due to Equation 17 and 19.Here, we conduct two experiments using image datasets, specifically CIFAR-10 [ Krizhevsky et al.] and Fashion MNIST [Xiao et al., 2017] . Similar to previous experiments, we remove some part of the data during training phase, here 90% of the observations with label 3 for both datasets, and reintroduce them gradually during the production phase.In Figure 9 , we observe the same behavior as in Section 5.1. The quantile detector detects changes more quickly than the mean detector, and the performance of the former is closer to the true version than that of the latter. E Comparison Between Φ q and Φ 2 qIn this section, we will revisit the main experiments from sections 5.2 and 5.3, incorporating comparisons with the quantile detector using the first statistic Φ q . As expected, in figure 10 , we consistently observe that the first statistic Φ q achieves a better FDP than the second statistic Φ 2 q at the cost of a much smaller power. In addition, Φ q fails to detect any shift in the California dataset and has a much higher detection time. In Figure 11 , we have also computed the power relative to the harmfulness threshold ϵ tol of the Folktables data from Section 5.3. The second statistic performs much better than the first in terms of power, although the FDP of the latter is smaller (0.004) compared to 0.019 for the former. In this section, we compare our approach with a leading method, Detectron, proposed by Ginsberg et al. [2022] , in a batch setting. Directly comparing our method (SHSD) to those in Ginsberg et al. [2022] presents challenges due to fundamental differences in their design. Their methods are tailored for an offline batch setup, which requires a complete batch of production data to compute statistics and trigger alarms. In contrast, our approach is optimized for an online setting where shifts may occur gradually and continuously, necessitating real-time decisions without the ability to observe an entire unlabelled batch upfront. Our methodology is designed to detect harmful performance shifts on the fly, processing each observation sequentially without requiring access to the full production dataset.Applying offline methods like Detectron in an online setting would be both impractical and unfair, as these methods rely on training a model or computing statistics from a batch of data. Additionally, it would be computationally expensive since Detectron requires training a new model for each batch of production data. Consequently, deploying this approach online would entail training a number of models proportional to the production data size.To provide a meaningful comparison, we evaluated our method alongside Detectron in a batch setting, progressively increasing the size of the production or out-of-distribution (OOD) data. We generated shifts in line with the setup described in Section 5.2, ensuring no shift within the first 1300 samples of production data. We utilized the NHANESI classification dataset, as Detectron is specifically designed for classification tasks. Our experiments were replicated 50 times, yielding a total of 10,200 shift instances. The results, summarized in Table 3 , demonstrate that for smaller sample sizes (100 and 1000), our method did not detect any shifts, as expected given the lack of shifts in the initial 1300 samples. However, Detectron raised a significant number of alarms (1126 out of 1700 for sample size 100 and 1493 for sample size 1000), all of which were false alarms. For larger sample sizes, while Detectron shows high power in detecting shifts, it also produces a high false discovery proportion (FDP). In contrast, our method exhibits lower power but significantly better control over false alarms, consistent with our objective of minimizing false positives.These results confirm that our method performs robustly in both batch and online settings, effectively maintaining low false alarm rates while detecting harmful shifts as they arise.In this section, we highlight some potential limitations of disagreement-based detectors, such as Detectron, which may limit their effectiveness in certain contexts. The primary concept behind Detectron is to train a disagreement classifier that performs comparably to the original model on the training distribution while disagreeing with the original model's predictions on production data. This approach is highly sensitive to the base model's performance, the choice of function class, and the size and nature of the production data. Although Detectron shows high power in detecting harmful shifts (as evidenced by our experiments), it may raise false alarms when the shift is benign.In Figure 12 , we illustrate a failure case for the disagreement-based detector. In this example, training data points are represented in red and green, with the ground truth shaded accordingly. The solid black line represents the decision boundary of a base model, which we assume to be a perfect classifier.The data has shifted to the right, resulting in unlabeled production data that is still correctly classified by the base model.We've also depicted the potential learnable classifier as a dashed line, representing the boundary of all possible functions, which depends on the model type, complexity used for the disagreement classifier, and the nature and size of the data. We have shown a potential disagreement classifier in orange that performs similarly to the original model on training data but disagrees with the predictions of the base classifier in the production data. As shown, even with a benign shift, we can still find a disagreement classifier that performs well on training data but disagrees significantly in production, raising a false alarm.We run all our experiments on an Amazon EC2 instance (c5.4xlarge) that consists of 16 vCPUs and 32 GB of RAM.This research, focusing on developing algorithms to detect harmful distribution shifts in machine learning models, has significant and diverse practical impacts. It offers a solution to a key challenge in the safe deployment of AI across various industries by detecting shifts without needing labeled data. For instance, in healthcare, the ability to identify harmful shifts in predictive models enhances the accuracy and reliability of diagnostic tools, which is especially vital as patient data continuously changes due to new diseases or demographic shifts. In finance, the algorithms can detect market trends or consumer behaviour changes that might negatively impact forecasting models, leading to more adaptive and resilient economic models, improved risk management, and better-informed decision-making processes.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: This work's main contributions can be found in the abstract and introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper introduces a method for detecting harmful shifts without having access to ground truth labels. The reviewers agree that the problem is well motivated, and the main contribution of the paper, allowing to handle the lack of access to ground truth labels, makes the contribution quite practical. The techniques are noted to be novel and insightful, as well as motivated from mathematical derivations. There were some issues raised regarding the comparison with existing works (see the baseline mentioned by KyAy and related works mentioned by 4vCN), but this was mitigated by the authors’ response, and the required modification to the paper is small enough to be achievable towards the final version. Concluding, the paper provides a novel solution to an important problem, and would be a welcome addition to Neurips.

Reviews: review: paper_summary: paper_summary: This work is an extension of Podkopaev and Ramdas. They propose a framework to detect the harmful distribution shift without accessing the true labels during detection. To do that, the authors introduce an error estimator model to measure the error scores. Besides, the authors propose a strategy to manage false positives by leveraging these imperfect error predictions. Experiments show that the method effectively controls false positives and performs well under various distribution shifts.
strength_weakness: strength_weakness: - In the experiments section, the authors only compare with a baseline scheme and ignore other existing distribution shift detection algorithms (e.g., [1-2]), although they may not claim to only detect harmful distribution shifts.

### Reference
1. Hinder F, Artelt A, Hammer B. Towards non-parametric drift detection via dynamic adapting window independence drift detection (dawidd)[C]//International Conference on Machine Learning. PMLR, 2020: 4249-4259.
2. Frias-Blanco I, del Campo-Ávila J, Ramos-Jimenez G, et al. Online and non-parametric drift detection methods based on Hoeffding’s bounds[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 27(3): 810-823.
questions: questions: - In lines 117-118, could the authors explain how they connect the upper bound of the training dataset to the lower bound of the test dataset to derive this probability inequality?
- The proposed method does not rely on distribution assumptions and has very low accuracy requirements for error estimation (In section 4.1). Additionally,  there are no hyper-parameters that need to be preset (in section 4.2). However, there is no free lunch. Despite these excellent properties, does the method will face any other challenges?
limitations: limitations: - The proposed approach is to detect harmful distribution shifts in production environments, which is a very interesting problem setting. However, there are many out-of-distribution detection and robust learning algorithms that can cover or solve the harmful distribution shift problem, or even solve it directly, which makes the proposed method not so necessary in practical applications.
paper_summary: paper_summary: This paper proposes a new method for identifying harmful distribution shifts when no labels are available at test time. This work is a good contribution to the field of ML and has practical importance.
strength_weakness: strength_weakness: - In my view, there are two main weaknesses in this paper:
  1. The error prediction function is a function of the covariates X. That implies that this method only works in detecting harmful shifts that involve shifting the distribution of X (e.g., covariate shifts). If the shifts happen only in P_{Y|X}, instead, the method would have low power. However, this limitation is not discussed by the authors (Please let me know if I have any misunderstanding here). If this is true, the authors should compare their work with [1], because it's not clear how their work is different in nature from that work. If it turns out that both works have a similar nature, the new method would have to be superior in terms of performance;
  2. In the experiments, the authors do not compare their method with any other competitors. You could, for example, compare your method with covariate shift detection methods [1,2] (which do not rely on the presence of labels).

References:

[1] Ginsberg, Tom, Zhongyuan Liang, and Rahul G. Krishnan. "A learning based hypothesis test for harmful covariate shift." arXiv preprint arXiv:2212.02742 (2022).

[2] Polo, Felipe Maia, Rafael Izbicki, Evanildo Gomes Lacerda Jr, Juan Pablo Ibieta-Jimenez, and Renato Vicente. "A unified framework for dataset shift diagnostics." Information Sciences 649 (2023): 119612.
questions: questions: The questions I have pertain to the first point I mentioned in the weaknesses section. Could you clarify that point for me? I would be willing to increase my score if I understand how your work builds on the previous research by Ginsberg et al. and recognize a significant difference.
limitations: limitations: Limitations are discussed.
paper_summary: paper_summary: The paper introduces a sequential drift detector designed to identify drifts that may negatively impact model performance without requiring target labels in production. This is achieved by training an error proxy model on a calibration dataset, which can then be applied in a production scenario, thereby eliminating the need for target labels. While the practical benefits of this method are evident, its performance is initially worse than the reference method that uses labels. This issue is addressed by proposing a modification to the scoring metric, making the performance of both methods comparable. Although the method that uses labels detects drifts earlier, the label-less method now demonstrates similar performance. The methods are tested and compared on multiple synthetic and real-world datasets.
strength_weakness: strength_weakness: ### Line 17-19:

- Domain knowledge regarding performance-based drift detection methods is not mentioned.
- The claims regarding the properties of “traditional” drift detection methods are unfounded and can be seen as misinformation.
- For further references the survey "Learning under Concept Drift: A Review” of  Jie Lu, Fellow, IEEE, Anjin Liu, Member, IEEE, Fan Dong, Feng Gu, Jo˜ao Gama, and Guangquan Zhang provides a structured overview of different detection mechanisms and differences.
- In addition, the use of a reference window is presented as a weakness, but the proposed method also uses a calibration set, which can be seen as a reference window.

### Line 134:

- Phrases like “If sufficient”, “Should align”, “would achieve” involve a lot of assumptions.
- If the error estimator is a statistical model, it would open this expression up to further introspection.
- While the statements proposed by the work seem solid, sections where these subjective claims are mentioned (e.g., here and the beginning of section 2) are very loose and are left unanswered.

### Line 193:

- The FDP plot has no blue in it; what does this sentence refer to?

### Equations 10 and 11:

- These equations confuse more than they clarify.
- A new form is introduced just to show how it can be directly inferred from Equations 1 and 2. This, by itself, would be fine, but then the notation is changed again directly.
- Why not go from Equations 1 and 2 directly to Assumption 4.1?

### Experimental Reproducibility:

- While the data is well explained, the configuration of the experiments is left up to interpretation.
- Access to the scripts used to run the experiments would highly benefit reproducibility.
- While the paper checklist mentions that the code will be provided, this is currently not realized, making it impossible to include it in the review.
questions: questions: ### Line 7:

- Is "learnt" the old variation of "learned" or is there a specialized meaning to it?

### Line 46:

- Does the predictor for the model error serve as an approximation of the conditional probability to produce a higher error in areas of the feature space with sparse or contradictory behavior?
- Would the representation of this not be best captured by a Gaussian Process model?
- You do not mention the model you used to produce your results. (Or do you always use the same model for the main prediction and the error prediction, if yes, why?)

### Line 74:

- In which domain do X and Y reside?
- What are we trying to predict given what kind of information?
- The current specification does not allow the assumption of a suitable bounded error function. Specifying this kind of error for regression problems proves to be very difficult and highly dependent on the application context, for example.

### General Question on DDM and PHT:

- The DDM (Gama, J., Medas, P., Castillo, G., Rodrigues, P. (2004). Learning with Drift Detection. In: Bazzan, A.L.C., Labidi, S. (eds) Advances in Artificial Intelligence – SBIA 2004. SBIA 2004.) and PHT (Page, E.S.: Continuous inspection schemes. Biometrika 41(1-2), 100–115 (1954). DOI 10.1093/biomet/41.1-2.100. and Gama, J., Sebastião, R. & Rodrigues, P.P. On evaluating stream learning algorithms. Mach Learn 90, 317–346 (2013). https://doi.org/10.1007/s10994-012-5320-9) drift detection tests are frequently cited and also track the development of the mean model error over time and restrict its allowed deviation based on sequential analysis. How does this compare to your work, and what are the differences?

### Line 186:

- Which type of R² score is used here? What is the reference model?
- How is the R² score used in the production scenario, as it is batched based the application on sequential data points is not natively defined.
- Against your assumption in section 2, the R² score is not bounded! Depending on the formulation, it can have large positive values but be bounded to larger than 0 or have large negative values but be always smaller than 1. R² is only bounded if it is viewed as a data descriptor that assumes that a linear model is used to describe the data, then it is a measure for the explainable variance. This specification confuses more than it help. The R²  formulation you use should be clearly specified in mathematical notation.

### Equations 12 to 15:

- Are you referring to joint probabilities or conditional probabilities? So, S(X)=1 and E>q or S(X)=1 given E>q?

### Line 235:

- If the Hoeffding interval is based on the Hoeffding Inequality, does your application fulfill all stated requirements?
- With special focus on the R² Score you use as your “bounded” metric, Theorem B.1 seems to have unfulfilled assumptions.

### Section 5.1:

- What error metric is used for the stated experiment? Accuracy?
limitations: limitations: The limitations are mentioned and discussed.

The assumptions used for some proves are however not discussed into detail. Further Details in Questions and Weaknesses
paper_summary: paper_summary: The authors introduce an approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. They solution substitute true errors with the predictions of a learnt error estimator.
strength_weakness: strength_weakness: It is worth noting that while I am familiar with distribution shifts, I do not know  continuous production environments is limited. I will be evaluating this paper from the perspective of an outsider in this field, and will rely more on the feedback of other reviewers.
**************************************
I encourage the AC to bring in reviewers who are familiar with this field.

I encourage the AC to give less weight to my review comments.
**************************************
Weaknesses:

1) I suggest that the authors provide a section on related work concerning distribution shifts and continuous production environments, to help readers clearly understand the motivation behind this paper. To my knowledge, the Out-of-Distribution (OOD) field’s research on Dataset Shift [1-4] also discusses distribution shifts; please include a discussion on this. The introduction does not well introduce the research questions of this paper, especially considering that this paper has not yet reached the length limit.

2) My major concern about this paper is whether “requires no access to ground truth data labels” is a worthwhile background setting to study, especially since Podkopaev and Ramdas [2022] have already satisfactorily addressed more common scenarios. Considering that the method proposed by the authors is relatively straightforward: training an error estimation model to predict the performance of the primary model on production data.

[1] Puli et al., Don’t Blame Dataset Shift! Shortcut Learning Due to Gradients and Cross Entropy, in NeurIPS 2023.

[2] Huang et al., Harnessing Out-of-Distribution Examples via Augmenting Content and Style, in ICLR 2022.

[3] Silva et al., The Value of Out-of-Distribution Data, in ICML 2023.

[4] Yang et al., Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning, in NeurIPS 2023.
questions: questions: N/A
limitations: limitations: N/A

Comments: comment: Thank you for addressing the question, it is very clear to me.
Thank you for your valuable feedback and for raising your score. We appreciate your insights and will incorporate our discussion into the final version of the paper.
I have gone through the responses. 

**I encourage AC to completely ignore my comments and score.**

Good luck :)
Thank you for your response. Please find below our answers to the remaining points you raised:

> Please verify that this does not violate the independence assumptions of Hoeffding's Inequality. Based on my experience, having attempted to use Hoeffding's Inequality in regression for non-stationary environments, I can confirm that Hoeffding's Inequality requires both bounded and INDEPENDENT random variables. By normalizing your data targets, you may introduce interdependencies among them, which could be problematic.
 
You’re correct that Hoeffding’s Inequality, as well as the inequalities used for confidence sequences, require both bounded and independent random variables. Regarding the normalization, we are only rescaling the output before constructing or splitting the data into production, calibration, and training sets using min-max scaling with the same minimum and maximum values.  To the best of our knowledge, this rescaling process should not violate the assumption of independence.
 
 
> I agree with your point, but your reference specifically mentions tests based on distributions. There is an entire class of performance-based drift detectors that do not have these issues but do require target labels. Please clarify this distinction in your discussion.
 
In the second paragraph of the Introduction (L 25-32), where we describe methods that might detect harmful shifts without labels, we will also include techniques that do require labels in the final version of the paper, including the ones you suggested.
 
>In addition, the use of a reference window is presented as a weakness, but the proposed method also uses a calibration set, which can be seen as a reference window.
 
We agree that the calibration set can be seen as a window, but our comment was about inference time. Our method, being completely sequential, does not require defining a window for the production sample size, as it processes the data as it arrives, one by one. In Lines 17-20, we will specify that the prespecified sample set refers to the production set.
 
> You do not mention the model you used to produce your results. While you mentioned that the model can be changed, what specific model did you use for your evaluations? This is important for reproducibility.
 
For the image dataset, as stated in Lines 268-272, we used a ResNet-50 as the base model, modifying only the head for error estimation. For the tabular data, we consistently used a Random Forest regressor/classifier with the default parameters from the scikit-learn library (num_trees=100, max_depth=None, boostrap=True). We will clarify this further in the paper.
 
 
>General Question on DDM and PHT. I understand that DDM and PHT require labels, while your approach does not. However, my question was about what happens afterward. After using your proxy model to predict the error, you still need to determine whether there is a drift. This decision-making process seemed quite similar to DDM to me. Could you elaborate on the differences, if any, or clarify if your method was inspired by DDM?
 
Our paper is not inspired by DDM but by the work of Ramdas et al. (2022). While there may be some similarities in the decision-making process, as DDM computes an approximate confidence interval in the reference set (called the concept set) and compares it with a threshold value, there is a substantial difference. DDM is designed for contexts where the model also changes dynamically. Unlike DDM, the method proposed by Ramdas et al. (and by extension, our methods) uses a calibration step and defines the warning threshold based on statistical tools that provide provable false alarm guarantees in finite samples with minimal assumptions, such as independence and bounded loss.
Thank you for your detailed rebuttal! I'll briefly outline my remaining questions and concerns:

```
computed with an output Y normalized between (0, 1).
```

**MOST IMPORTANT QUESTION**: Please verify that this does not violate the independence assumptions of Hoeffding's Inequality. Based on my experience, having attempted to use Hoeffding's Inequality in regression for non-stationary environments, I can confirm that Hoeffding's Inequality requires both bounded and INDEPENDENT random variables. By normalizing your data targets, you may introduce interdependencies among them, which could be problematic.

``` 
The claims regarding the properties of “traditional” drift detection methods
```
I agree with your point, but your reference specifically mentions tests based on distributions. There is an entire class of performance-based drift detectors that do not have these issues but do require target labels. Please clarify this distinction in your discussion.

```
In addition, the use of a reference window is presented as a weakness, but the proposed method also uses a calibration set, which can be seen as a reference window.
```

Unfortunately, this point was not addressed in your rebuttal.

```
You do not mention the model you used to produce your results.
```

While you mentioned that the model can be changed, what specific model did you use for your evaluations? This is important for reproducibility.


```
General Question on DDM and PHT
```
I understand that DDM and PHT require labels, while your approach does not. However, my question was about what happens afterward. After using your proxy model to predict the error, you still need to determine whether there is a drift. This decision-making process seemed quite similar to DDM to me. Could you elaborate on the differences, if any, or clarify if your method was inspired by DDM?
Thank you for the detailed explanation! Please include our discussion in your final version of the paper.

I have raised my score.
Thank you for your response.

> As far as I understand, in a practical sense, the main difference is that one method is designed for online settings while the other is designed for offline settings. Is that true? I did not understand if your method is designed to detect harmful shifts of X. Is that the case?

Yes, the major difference is that our method is designed for online use, while the other is for offline. Regarding whether our method is designed to detect harmful "shifts of X" (i.e. covariate shifts), our central assumption is Assumption 4.1, i.e. that the selector function generalizes in terms of FDP from calibration to production. This is more likely to hold in the case of pure covariate shift, where the relationship Y | X is invariant. However, it may still be effective if this relationship changes, provided the model tends to have high error in the same regions of the input space before and after the shift. Note that in our empirical experiments, we study shifts in natural datasets, where the Y | X relationship is very likely to be at least somewhat non-invariant. Our method is shown be effective in these cases.

> Do you know why the Detectron has such a bad FDP in your experiment?

Detectron’s main idea is to learn a Disagreement Classifier that performs as well as the original model on the training distribution while disagreeing with the original/base model's predictions on the production data. This method is highly sensitive to the performance of the base classifier, the function class used, and the size and nature of the production data. While it effectively detects harmful shifts (as shown by its high power in our experiments), it may fail when the shift is benign.

Below is a simple code example that generates a plot illustrating a failure case. In this example, the training data points is represented in red and green, with the ground truth shaded in corresponding colors. The base model is shown as a solid black line. For simplicity, we assume the model is a perfect classifier. The data has shifted to the right, creating unlabeled production data, all correctly classified by the base classifier. 

We’ve also depicted the potential learnable classifier as a dashed line, representing the boundary of all possible functions, which depends on the model type, complexity used for the disagreement classifier, and the nature and size of the data. We have shown a potential disagreement classifier in orange that performs similarly to the original model on training data but disagrees on the predictions of the base classifier in the production data. As shown, even with a benign shift, we can still find a disagreement classifier that performs well on training data but disagrees significantly in production, raising a false alarm.

```{python}
import matplotlib.pyplot as plt
import numpy as np

# Generating sample data
x = np.linspace(-3, 3, 500)
base_classifier = 2 * np.sin(x) 
shaded_area = 2 * np.sin(x)
boundary_upper = 5 * np.sin(x) 
boundary_lower = 0.5 * np.sin(x)
disagreement_classifier = 3.5 * np.sin(x) * (x <= 0) + 4.5 * np.sin(x) * (x > 0)

np.random.seed(0)
x_class1 = np.random.uniform(-2, -1, 10)  
y_class1 = np.random.uniform(-2, 1, 10) + 3

x_class2 = np.random.uniform(-2, -1, 10)
y_class2 = np.random.uniform(-5, -2, 10) - 3

x_class1_shifted = np.random.uniform(-2, -1, 20) + 3
y_class1_shifted = np.random.uniform(-2, 2, 20) + 4.5
x_class2_shifted = np.random.uniform(-2, -1, 10) + 3
y_class2_shifted = np.random.uniform(-5, -2, 10) + 2

x_unlabelled = np.concatenate([x_class1_shifted, x_class2_shifted])
y_unlabelled = np.concatenate([y_class1_shifted, y_class2_shifted])

# Plotting
plt.figure(dpi=200, figsize=(8, 5))

plt.fill_between(x, shaded_area, 12, color='green', alpha=0.1, edgecolor='none', label='Ground Truth (Class 1)')
plt.fill_between(x, -12, shaded_area, color='red', alpha=0.1, edgecolor='none', label='Ground Truth (Class 2)')

plt.scatter(x_class1, y_class1, c='green', marker='+', label='Training Data (Class 1)')
plt.scatter(x_class2, y_class2, c='red', marker='+', label='Training Data (Class 2)')

plt.scatter(x_unlabelled, y_unlabelled, facecolors='none', edgecolors='black', label='Unlabelled Production Data')

plt.plot(x, boundary_upper, 'gray', linestyle='--', label='Set of Potential Learnable Classifiers')
plt.plot(x, boundary_lower, 'gray', linestyle='--')
plt.plot(x, base_classifier, 'k-', label='Base Classifier')
plt.plot(x, disagreement_classifier, 'tab:orange', linestyle='-', label='Disagreement Classifier')

plt.legend(loc='upper left', bbox_to_anchor=(1, 1), fontsize='small', frameon=False)
plt.grid(False)
plt.tight_layout()
plt.axis('off')
plt.show()
```
Thank you for your reply!

I wanted to clarify some points:
1. As far as I understand, in a practical sense, the main difference is that one method is designed for online settings while the other is designed for offline settings. Is that true?
2. I did not understand if your method is designed to detect harmful shifts of X. Is that the case?
3. Do you know why the Detectron has such a bad FDP in your experiment?