Incremental Uncertainty-aware Performance Monitoring with Labeling Intervention

Authors: alexander  koebler
thomas  decker
ingo  thon
volker  tresp
florian  buettner
Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-05 13:31:38

Abstract: We study the problem of monitoring machine learning models under temporal distribution shifts, where circumstances change gradually over time, often leading to unnoticed yet significant declines in accuracy. We propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates model performance by modeling time-dependent shifts using optimal transport. IUPM also quantifies uncertainty in performance estimates and introduces an active labeling strategy to reduce this uncertainty. We further showcase the benefits of IUPM on different datasets and simulated temporal shifts over existing baselines.

Main paper: 
 introduction: 
 Deployed machine learning models often face the critical challenge of distribution shifts, where the data encountered in production deviates from the data used during training. Many relevant shift scenarios involve changes over time, which are often gradual and continuous [26, 25] . These shifts are characterized by the fact that the statistical properties of the data or the environment change progressively rather than abruptly. This gradual nature can make time-dependent shifts more insidious, as they may not be immediately apparent but can still lead to substantial degradation in prediction quality over time [12] . Therefore, anticipating and understanding temporal performance changes is essential for ensuring the reliability and effectiveness of a machine learning model in dynamic environments [7] . However, directly monitoring the performance during the deployment is challenging as labeled data is often unavailable in production. Moreover, obtaining labels can be cumbersome, time-consuming, and costly, leading to delays in the assessment. Therefore, an increasing number of label-free estimation methods have been proposed that aim to anticipate the model performance purely based on unlabeled data available at runtime. While this task is provably impossible in general [6] , it can be approached by leveraging prior knowledge about the structure of the shift [13] . Existing techniques leverage strategies based on feature statistics [9] , importance weighting [23, 3] , model confidence [14, 13] , disagreement between models [16, 1] , or model differences after retraining [27] . However, only a few works explicitly try to incorporate the additional structure arising from the temporal nature of shifts into the estimation process [2, 28] . Furthermore, existing techniques are unable to quantify any uncertainty related to the performance estimate, cannot anticipate model degradation for arbitrary loss functions, and provide no information on how to best aid estimation with limited labeled data. To address these challenges we make the following contributions:• We propose Incremental Uncertainty-aware Performance Monitoring (IUPM), as a novel label-free performance estimation method tailored to temporal distribution shifts (see Figure 1 ). Optimal Transport Optimal Transport (OT) aims at finding the cost-minimizing way to transform one probability measure into another [22] . Consider having n 0 samples from a domainΩ 0 = {x i 0 } n0 i=1and n 1 samples from another domainΩ 1 = {x i 1 } n1 i=1 with corresponding empirical distributions p0 = x0∈Ω0 1 n 0 δ x0 p1 = x1∈Ω1 1 n 1 δ x1where δ x denotes the Dirac measure. For a cost function c : Ω 0 × Ω 1 → R + , the transformation of p0 into p1 can be formalized by a coupling γ which represents a valid distributions over (Ω 0 × Ω 1 ) with marginals corresponding to p0 and p1 . Identifying the cost-optimal coupling reads:γ = arg min γ∈Γ x0∈Ω0 x1∈Ω1 c(x 0 , x 1 )γ(x 0 , x 1 ) with Γ = {γ ∈ R n0×n1 | γ1 n1 = p0 , γ T 1 n0 = p1 }which can be solved using different algorithmic approaches [22] . In the discrete sample case the obtained γ ∈ R n0×n1 simply is a matrix with entries γ(x 0 , x 1 ). Moreover, the conditional couplingγ(X 0 = x 0 |X 1 = x 1 ) = γ(x 0 , x 1 )x0∈Ω0 γ(x 0 , x 1 ) is a left-stochastic matrix whose entries can be interpreted as transition probabilities when moving from samples of X 1 to samples of X 0 following the most cost-efficient path.Incremental Performance Estimation using OT Consider a machine learning model f that has been trained with labeled data {(X 0 , Y 0 )} from the distribution P 0 (X 0 , Y 0 ). Suppose f is deployed in order to make predictions over time t > 0 with respect to data {(X t , Y t )} T t=1 each distributed with P t (X t , Y t ). During runtime (t > 0) we only have access to unlabeled data from X t and our goal is to estimate only based on this information how well the model performs over time. To do so we proceed as follows: Let γ t (X t-1 |X t ) be the conditional coupling linking data from X t to data from X t-1 in a cost-efficient way. Further, we define Ψ t (X 0 |X t ) = t i=1 γ i (X i-1 |X i ) describing the transition matrix obtained from composing all incremental transition matrices γ i (X i-1 |X i ) via matrix multiplication. It expresses the overall transition probabilities of going back to the labeled data available at t = 0 by connecting samples of two subsequent time points incrementally using an individual optimal transport coupling. Based on this we propose the following strategy to estimate missing labels for performance evaluation over time.P (Y t |X t ) = E Ψt(X0|Xt) [P (Y 0 |X 0 )]This means that our label estimate P (Y t |X t ) arises as mixture distribution [10] combining labeled data in X 0 according to the accumulated incremental coupling results. This strategy is explicitly reasonable for temporal shifts as we leverage their gradual nature by modeling subsequent distribution shifts incrementally using Optimal Transport. It implicitly assumes that the true decision boundary around data points from X t is similar to the ones of data points from X t-1 that are linked via the cost-efficient coupling, which again suits in particular gradual shifts over time. Given an arbitrary loss function L to measure model performance and a set of samples Ω t from X t , the resulting performance estimate for IUPM at time t, denoted by LIUPM t , is given by:LIUPM t = E P (Xt) E P (Yt|Xt) [L(f (X t ), Y t )] = 1 n t xt∈Ωt E P (Yt|Xt=xt) [L(f (x t ), Y t )]Note that P (Y t |X t ) is an actual predictive distribution that also internalized uncertainty for cases where linked samples have contradicting labels. Thus, we can use it to quantify the uncertainty of the anticipated performance using the expected standard deviation (SD) of the sample-wise loss estimates:U( LIUPM t ) = E P (Xt) SD P (Yt|Xt) [L(f (X t ), Y t )]Labeling Intervention In addition to providing a means for users to consolidate their trust in the performance estimate, the quantified uncertainty U( LIUPM t ) can also be used to automatically trigger efficient relabeling when the uncertainty exceeds an acceptable level. To make the most efficient use of a limited labeling budget allowing to label only m samples, we propose a Uncertainty Intervention (UI) strategy only querying ground truth labels for critical samples x t that contribute the largest uncertainty to the overall performance estimate:arg top-m xt∈Ωt SD P (Yt|Xt=xt) [L(f (x t ), Y t )]where arg top-m denotes the operator selecting the top m elements maximizing the objective. The new labels are used to update Ψ t (X 0 |X t ) such that P (Y t |X t = x t ) assigns a fixed label removing the accumulated uncertainty for sample x t . 
experiments: 
 We evaluate our proposed IUPM approach on two different settings and data modalities. First, to assess the general functionality of IUPM, we present results based on synthetic toy examples with continuous shifts in two-dimensional space. Second, to emphasize the connection to real-world scenarios, we perform experiments based on MNIST [18] and continuous image perturbations, e.g., resembling a gradual camera degradation. Throughout the experiments, we compare our approach to four existing performance estimation methods described in [13] . Average Confidence (AC) simply estimates the prediction accuracy as the expectation of the confidence for the predicted class across the data set. The more sophisticated Difference Of Confidence (DOC) [14] uses the discrepancy between the model confidence on the source and target data sets as an estimate of performance degradation. Average Threshold Confidence (ATC) [13] learns a threshold for model confidence on the initialization data set Ω 0 and estimates accuracy on the current set as the fraction of examples where model confidence exceeds this threshold. Lastly, we consider Importance reweighting as proposed by [3] . Additionally, we evaluate the use of the direct mapping γ(X 0 |X t ) for label transport and performance estimation inspired by [8] . We call this approach Non-Incremental Performance Estimation (NIPM). Across all experiments, we consider the model accuracy as the loss criteria L to evaluate performance. For more details, we also refer to Appendix A.1 and A.2. We use three two-dimensional toy data sets (Fig. 2 ) provided by [21] . For all three datasets, we train a Random Forest (RF), XGBoost (XGB), and a Multilayer Perceptron (MLP) classifier in the initial source distribution. After training at t = 0, all data sets are shifted for t = 1, . . . , 100 steps to simulate gradual changes over time. For the "Clusters" and "Moons" data sets, this shift results from rotating both classes by 2°per step. The 1 show that IUPM consistently outperforms all baselines across all datasets and models. The error in the accuracy estimate can be further reduced by intervening on the labels. In this case, we relabel the top 50% of Ω t according to our introduced UI strategy once U( LIUPM t ) > 0.1. SD P (Yt|Xt=xt) [L(f (x t ), Y t )]with randomly selected m samples from Ω t . The results in Table 2 show a considerable benefit of targeted sampling for the majority of scenarios, demonstrating that it can improve the efficiency of the labeling intervention under a limited labeling budget. This supports the hypothesis that correcting the labels of examples with the highest instance-wise uncertainty also has an increased benefit to the overall performance estimation.Monitoring Performance Degradation due to Image Perturbations To assess more complex shifts, we monitor a model classifying handwritten digits [18] that experience a shift caused by common image perturbations [20] such as rotation, used in a related context in [24] , translation, and scaling of the digits. For this experiment, we trained a LeNet [19] . Further, we perform the matching based on representations of the second classification layer of this LeNet, reducing the dimensional of the matched samples from 784 to 84 and demonstrating a generalizing approach to deal with higher dimensional data also used in [5] . All shifts are evaluated for 20 steps, where Figure 3 illustrates the performance estimation for a 180°rotation. Even without intervention, IUPM can best estimate the performance. By allowing label intervention with the same settings as in the previous experiment, the performance estimation error can be reduced and the associated uncertainty can be limited to the predefined value U( LIUPM t ) < 0.1. The observations are further supported by the results on two additional shifts in Table 3 .  
conclusion: 
 We have introduced a novel method tailored to monitor a deployed machine learning model facing gradual distribution shifts over time. Our IUPM approach takes a step towards achieving a more reliable assessment of the quality of a model's output at run time by explicitly taking the uncertainty of the performance estimate into account, which current methods lack. This allows to simultaneous increase the user's confidence in the estimate and to perform a targeted label intervention to efficiently restore a sufficient trustworthiness of the system only when needed. We have shown that over two different data modalities and five different shifts, IUPM outperforms a number of existing performance estimation approaches. Incorporating additional insights from related fields such as gradual domain adaption [15] or active testing [17] as well as scaling to more complex data sets with real-world shifts, are interesting avenues for future research. 
 
 Appendix: In this section we provide some additional information to the used baselines. For the four existing baseline methods we relied on the implementations provided by [13] . Average Confidence (AC) simply estimates the prediction accuracy as the expectation of the confidence for the predicted class across the data set in step t as AC Ωt = E x∼Ωt [max j∈Y f j (x)]. Difference Of Confidence (DOC) [14] uses the discrepancy between the model confidence on the source and target data sets as an estimate of performance degradation. To obtain an approximation of the performance in step t, the degradation is subtracted from the performance on the initialization data set t = 0DOC Ωt = E x,y∼Ω0 [arg max j∈Y f j (x) ̸ = y] + E x∼Ωt [max j∈Y f j (x)] -E x∼Ω0 [max j∈Y f j (x)].This section provides additional implementation details for the experiments. For our IUPM implementation we rely on the entropic regularization optimal transport implementation with logarithmic Sinkhorn by [11] . As for the synthetic two-dimensional datasets, we used the data generator functionality provided by [21] . The "Clusters" data set is generated using the make_blobs function with a distance parameter of 1.0. The "Moons" and "Circles" data sets are generated using the corresponding functions with a noise parameter of 0.2 and a circle factor of 0.3. For the training and initialization step, a training set of 800 samples is generated, from which a validation and initialization set Ω 0 of 200 samples is partitioned. In each consecutive step, a set Ω k with a different random seed is generated. We then apply a shift to the set corresponding to the step k, i.e. k • 2 • for rotation and k • 0.02 for translation. For the synthetic data, we use a Random Forest Classifier (RF) and a XGBoost Classifier (XGB) [4] with 50 estimators and a maximum depth of 5 as well as a Multilayer Perceptron (MLP) with a single hidden layer of size 128. For optimal transport matching, we use a regularization parameter of 10 -4 . Concerning the MNIST experiments, we adapt the image perturbation implementation introduced in [20] to the continuous setting. The used LeNet model has been trained for 100 epochs with early stopping based on a patience of 10 epochs and PyTorch's Adam optimizer with a batch size of 16 and a learning rate of 1e -3. For the optimal transport matching, we use a regularization parameter of 1. In this experiment, Ω 0 also consists of 200 validation samples. 



Reviews: review: title: title: A novel approach to label-free performance estimation over temporal shifts
main_review: main_review: This work deals with an important challenge faced by real-world ML models. It proposes a label-free method for monitoring the performance of ML models after deployment when data distribution gradually shift from the training set over time. The use of optimal transport appears to be innovative and cost-efficient. Overall, it is an interesting paper but I am unsure of the real-world implications given the assumption of gradual shift and the simple datasets used in the experiments. See below for some details.

1. It would be useful to show to what extent IUPM works with abrupt changes in the dataset rather than a gradual shift, and what the boundary cases are (i.e. quantifying how "gradual" the shift needs to be for IUPM to work above baseline).

2. Does it work with higher dimensional data? The authors show an experiment with MNIST but real-world problems are much more complex. What are the computational complexity and scalability of IUMP? It would be useful to see more discussions on this.

3. Lines 129-132, "For this experiment, we perform the matching based on representations of the second classification layer of the LeNet, reducing the dimensional of the matched samples from 784 to 84 and demonstrating a generalizing approach to deal with higher dimensional data also used in [5]." Would this also work if matching is done on a later layer? It might be interesting to see a more comprehensive experiment on the trade-offs between matching on different layers and real-world guidance on choosing a layer.
title: title: Temporal Shift Monitoring , with room for sensitivity checks and broader application
main_review: main_review: __Summary:__ The authors propose a way to monitor machine learning model performance over time, without needing a lot of labeled data. This method addresses the challenge of performance monitoring of deployed models, under temporal distribution shifts. Their system, Incremental Uncertainty-aware Performance Monitoring (IUPM), quantifies uncertainty of and queries labels only when absolutely needed.

__Quality:__
The proposed approach  IUPM is technically sound. The experiments are well-executed, with clear baselines, but could be expanded for more diverse real-world datasets. In particular, the method seems to rely heavily on the relationship between the original training data and the shifted data. If the original model was not trained on a representative sample, the system’s ability to estimate future performance could be compromised.

__Clarity:__
The paper is mostly clear but can be dense in places, particularly where technical details like OT and incremental performance estimation are discussed.  

__Originality:__
The authors leverage optimal transport in an innovative way. The method is novel in its combination of performance estimation and uncertainty quantification for temporal shifts. 

__Significance:__
The approach addresses performance degradation over time (a common and often overlooked issue in deployed models). The idea of selectively triggering label interventions based on uncertainty is also a practical contribution. IUPM has the potential to make an impact in fields where retraining or labeling is expensive, though its real-world implications could be explored further.  The method is designed for cases where shifts happen over time. It may not be well-suited to non-temporal shifts or contexts where the primary changes are driven by other factors (i.e., spatial, categorical…). 

__Pros:__
- Innovative use of optimal transport
- Combines uncertainty quantification with selective labeling to reduce monitoring costs.
- Empirical results show improvements over existing baselines.
- Tackles a practical problem in model deployment.

__Cons:__
- The method assumes gradual distribution shifts, which may limit its application in scenarios with abrupt or unpredictable shifts 
- Limited exploration of real-world, high-dimensional datasets beyond MNIST.  There is limited evidence of how the system performs in more complex, high-dimensional, real-world data with unpredictable shifts.
- Potential computational overhead for large-scale or real-time applications.

__High-level Feedback:__
The authors have developed a creative and promising approach to an important challenge in machine learning. The combination of performance estimation with uncertainty quantification and selective label intervention is both novel and practical. With some refinement in presentation and broader testing, this work could have a meaningful impact on the field. Expanding real-world use cases and simplifying some technical explanations will further enhance the paper’s accessibility and significance.

__Suggested Edits:__
Add more discussion on the implications of the experimental results. Why does IUPM outperform existing methods? Are there particular scenarios where it struggles? This kind of analysis will strengthen the contribution of the paper.