Title: Reliably detecting model failures in deployment without labels
Venue: NeurIPS 2025
Accepted: yes
Keywords: ML for Healthcare

Guardrails for AI

AI Safety

Deterioration Monitoring

Trustworthy ML
Abstract: The distribution of data changes over time; models  operating in dynamic environments need retraining. But knowing when to retrain, without access to labels, is an open challenge since some, but not all shifts degrade model performance. This paper formalizes and addresses the problem of post-deployment deterioration (PDD) monitoring. We propose D3M, a practical and efficient monitoring algorithm based on the disagreement of predictive models, achieving low false positive rates under non-deteriorating shifts and provides sample complexity bounds for high true positive rates under deteriorating shifts. Empirical results on both standard benchmark and a real-world large-scale internal medicine dataset demonstrate the effectiveness of the framework and highlight its viability as an alert mechanism for high-stakes machine learning pipelines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to determine whether a model has suffered from \"post-deployment deterioration\" (PDD). That is, the joint distribution of the covariates $x$ and labels $y$ have shifted such that the model's performance is worse. The paper distinguishes this from out of distribution (OOD) detection in that OOD detection does not ask whether or not the performance of the model has become worse. The method takes inspiration from the model disagreement literature, which checks if multiple models make different predictions on a test point. Here, the \"multiple models\" come from adding a \"Variational Bayesian Last Layers\" to an existing model's output layer (i.e., the model now outputs a distribution over logits). This distribution over logits is sampled from, passed through a softmax, and then the resulting probability distribution is sampled from with an increased temperature to get a prediction. The number of these sampled predictions that disagree with the base-model's argmax prediction is the \"model disagreement\". This process is repeated for a number of training datapoints to get a sense for typical model disagreement for in-distribution data. Model disagreement is computed the same way for input test data, and if the disagreement is higher than typical for in-distribution data, the test data are flagged. In experiments, the paper shows the proposed method to have often higher true positive rate (TPR) than similar methods with the same false positive rate (FPR); however, the TPR is fairly noisy across different random seeds. In the appendix, an \"idealized\" version of the algorithm is proposed and its theoretical properties (e.g., FPR and TPR related to its input parameters) are studied.\n\nSTRENGTHS AND WEAKNESSES:\n# Weaknesses\n\nI've broken down comments here into various sections below.\n\n## What is the proposed method doing?\n\nI overall think it's not clear what this algorithm is really doing and why it should work. And I think it's really important when proposing new methods to give some kind of scientific understanding of the methods (unless the empirical results are so phenomenal that the research community just needs to know about this new method without really knowing anything about it, but I don't think that's the case here). I don't think the paper is currently meeting this standard, which is the main influence on my score. A few specific comments on this below:\n\n1. In the main text of the paper, I didn't see any discussion of why readers should think the proposed method would work. It has the flavor of existing OOD algorithms, but it's supposed to (1) detect OOD points, and (2) detect if those points are going to give bad performance from the model. I didn't really see why we would expect (2) to hold. Overall, I think there needs to be a lot more discussion of the intuition behind the algorithm in the paper.\n\n2. The theoretical discussion in the supplement has some intuition of why the proposed algorithm will work. This development proves conditions under which an \"idealized\" version of the algorithm will work well. I think this is really great, but could use some expansion. First, the idealized version of the algorithm is pretty idealized, and it's not clear how connected it is to the real version of the algorithm. The idealized version assumes access to all other models that have training error within $\\epsilon$ of the base model. I think the paper is arguing that this is approximated by some of the Bayesian sampling and increased temperature. But why does this approximate this model set well? The paper briefly implies (line 1010) that the Bayesian sampling and temperature will *over-approximate* the size of this model class, but it's not clear to me that this is true. I think careful exploration of the connection between the idealized and practical algorithm is needed to finish the story of this paper. Second, I really think some of the theoretical development needs to be moved to the main text, as it's what ties the entire paper together.\n\n3. There are no experients that get at the heart of what's going on with the algorithm. I think synthetic experiments are an excellent opportunity to really probe what's going on behind the scenes of an algorithm -- when should we expect the algorithm to fail or succeed? How well does the intuition behind the algorithm hold up empirically? Carefully designed synthetic experiments give the opportunity to answer all of these questions, so I would really recommend thinking about how that could be done for the current paper.\n\n## Theoretical results\n\nI thought that two potentially impractical assumptions were made in the theory.\n\n**Assumptions**\n- The theoretical results hinge on assuming the model class is correctly specified, which is a major assumption that is rarely true in practice.\n- Assumes that $m$ datapoints are available at once to make a decision of prediction on. Again, it's not clear to me how practical this setting is.\n\n\n**Clarity**\nThere were a few places towards the beginning of the theory where I found things either underdefined or just a little unclear. These are mostly towards the beginning of the theoretical development, as I started skimming towards the end as these built up.\n\n- In Lemma A.1: $h$ was not defined. Additionally, the proof of the lemma states that $err(f, Q_g) > err(f, P_g)$, where $Q$ is the deployment distribution and $P$ is the training distribution. Why must this be true given the assumptions of the theorem? What if $Q$ is a point-mass at a point that $f$ happens to get correct? And below line 970, a sup is taken over $A$ (I think sets $A$?) without saying what $A$ varies over. Finally, the proof starts by stating it is going to show that D-PDD implies PDD, but it never shows the reverse direction, despite the statement of the Lemma saying that PDD also implies D-PPD.\n- Definition 4 states that the algorithm detects when $\\epsilon_q - \\epsilon_p > 0$, then Definition 5 states that the algorithm detects when $\\xi \\geq 0$, and then the line after Definition 5 states that $\\xi \\geq \\epsilon_q - \\epsilon_p$. So these quantities aren't the same, so how can it be that the algorithm is simultaneously thresholding both?\n- Definition 5 states that $\\xi \\geq 0$. But why? Does this assume $f$ is the minimizer of $err(f; P_g)$? I don't think this was stated.\n- Eq (6) uses the notation $Q_{1-f}$, but I don't think the $1-f$ notation was ever defined.\n- Thm A.3 proves a bound on the FPR of the algorithm. The discussion around it implies that it bounds the FPR by $\\alpha$, the \"significance\" level of the algorithm. But I don't think the theorem proves this. In particular, it states that the FPR can be bounded by $\\alpha + (1-\\alpha)e^{-n\\epsilon_0^2 + d}$, where $d$ is the VC dimension of the hypothesis class. If $d$ is huge, then I'm pretty sure this bound can be arbitrarily larger than 1, whereas $\\alpha < 1$.\n- Towards better connecting the theory to the main paper: Theorem A.3 gives a required batch size $m$ to achieve a bounded FPR. It would be really interesting to evaluate $m$ for typical ML models and see if the result is practical. Is this telling us we need to batch up 94303 samples before making an effective decision about domain shift? Or is it saying we only need 20?\n\n\n## Experiments\n1.  The experiments are run in \"{10,20,50}-shot scenarios\". I wasn't totally clear on what this means. Does this mean that the model gets to see a batch of, e.g., 10 test datapoints before guessing whether there's model drift? It seems like this runs counter to the run in real time idea of the paper. In practice, I'd imagine that we want to make predictions one at a time. And then we have to make a \"domain shift yes/no\" guess when each datapoint is observed. How does the current experimental setup fit into this picture? Or are there really scenarios where we can wait to make a prediction until we've seen 50 test points?\n\n2. In the entire experiment depicted in Figs 2 and 3, it seems like the existing BSSD method is outperforming the proposed method. This doesn't seem to be discussed in the paper -- so what is this experiment showing that's good or promising about the proposed method?\n\n3. Figure 2b seems to be showing really bad performance of the proposed method, as it's true positive rate is nearly zero across the board. The text seems to indicate that this is a good thing because there's actually not much domain shift happening here. But in that case shouldn't there be no true positives at all? How are other methods getting high true positive rates if there is nothing to correctly flag?\n\n4. The x-axis in Fig 3 isn't described. I'm guessing this is an amount of domain shift, but it's not clear to me what the numbers mean.\n\n\n\n## Overall Comments\n1. Some references to specific Appendicies seem to be pointing to the wrong place (e.g., saying that baselines are described in B.4, but they're actually described in B.1). Please check the appendix references to make sure they're all pointing to the right place.\n2. \"D3m does not trade off prediction accuracy\" -- it does require the addition of a Bayesian head to the model, which I would assume could affect the accuracy.\n3. $\\tau$ was never defined in the main text\n4. The introduction makes it sound like there are few restrictions on the model class, but the paper seems to be entirely about classification models.\n5. I think the claim that \"we answer all desiderata\" it misleading, as one of the disiderata is that the method has a low FPR. I strongly suspect it is impossible to guarantee this in practice, but this claim makes it sound like the proposed method completely solves this issue.\n\n# Strengths\n\nI think with a properly motivated algorithm (along the lines of the theory in the appendix), this paper has the potential to be very strong. Proving conditions under which not only OOD detection is possible but a practically meaningful form of OOD (when the model performance is actually worse) is very impactful for applications, and I think this paper has the possibility to not only do that, but also give an explicit algorithm for doing so. I think it's especially strong that the algorithm has a mechanism to explicitly control its false positive rate, which I think will be important in practical settings; if I had to guess, I would say that hospital staff are going to be pretty susceptible to alarm fatigue from a black box ML system, so setting the FPR to be guaranteed low is important.\n\nQUESTIONS:\n1. How does the idealized algorithm that the theory is about relate to the actual algorithm? How much should users trust the theoretical results to apply to the real algorithm?\n\n2. When will the algorithm fail? Are there any indicators or situations that users should look out for?\n\n3. How can users reduce the variability of the proposed method that's seen in, e.g., Table 2? What could lower this variance and why?\n\n4. The paper seems to propose waiting to make PDD predictions until a large batch of test points has been accumulated. Why is this realistic? How would this work in time-sensitive situations?\n\nThank you!\n\nLIMITATIONS:\nI don't think the main text included much discussion of the proposed method's limitations besides pointing out the fact that its performance can be noisy -- the \"Conclusions and Limitations\" section actually mostly summarizes the strengths of the proposed method. As noted above, I would suggest really digging into the intuition behind the proposed method so that its failure case(s) can be better understood by readers.\n\nFurther, I think it's important to stress that the PDD task is fundamentally impossible without labels. If the distribution of $(x,y)$ changes such that $y\\mid x$ is totally different but the distribution of $x$ is the same, there's no way to detect this from labels. So I think it's really important to make the assumptions this paper is making more clear and emphasize how reality might come into conflict with those assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes methodology, D3M, for monitoring distribution shifts that may lead to performance degradation of deployed classifiers. \n\nThe central claim is that D3M addresses two desiderata that are not satisfied by any single prior method: (1) no need to access training data at deployment time and (2) only flag *deterioriating* shifts. \n\nThe method works by leveraging uncertainty estimates from a Bayesian neural network and monitoring for high variance/disagreement in the predictions. \n\nThe D3M method is competitive with Detectron (which requires training data at deployment time), though trails by substantial margins on some datasets. They also claim in writing that D3M has a number of efficiency benefits over alternative approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The problem addressed in this paper (monitoring for distribution shifts) is very important in safety critical settings like medicine. The paper evaluates on interesting and realistic benchmarks from the medical domain.\n- The proposed methodology has at least one advantage over prior work like Detectron: it does not requiring training data at deployment time.\n- The method is competitive with prior work like Detectron.\n\n**Weaknesses**\n\n- In Section 2.3, the authors make a number of claims about D3M without concrete evidence or experiments:\n    - “D3M does not trade-off prediction accuracy.” To support this claim, the authors should provide some experiments demonstrating that the prediction accuracy with D3M is indeed equivalent to or better than the prediction of accuracy of other classifiers. However, no such experiments or results are included in the paper.\n    - “Potential for edge deployment.” The authors claim that D3M is particularly well-suited for edge deployments but do not provide any analysis of edge hardware and how their method will map onto it. Without a concrete analysis, this claim is speculative.\n    - “While VBLL allows more efficient sampling compared to to a fully Bayesian network, the price to pay is the lack of diversity of the sampled predictions.” To support this claim, I would expect to see a comparison of the diversity of predictions when using VBLL and a fully Bayesian network. Without it, I cannot judge the importance of the proposed remedies (*e.g.* temperate scaling).\n- In Section 3.2, the authors claim that “D3M is significantly more scalable and practical for real-world deployment [than Detectron]”. While it is a clear advantage that D3M doesn’t require training data, the authors do not provide any concrete evidence of the scalability of D3M and Detectron. I would recommend that the authors include some quantitative measure of cost (*e.g.* FLOPs) that demonstrates that D3M is indeed more scalable than Detectron.\n- The presentation of the results on the IM dataset [Temporal Shift] could be improved.\n    - I’m quite confused by the Y-axis in Figure 2b. The label says TPR@5%, but this is in a setting with only a non-deteriorating shifts, so the true positive rate is undefined?\n    - The legend in Figure 2 and Figure 3 could be improved. I’m not sure what some of the abbreviations correspond to. For example, is DC the abbreviation for detectron? **\n\nQUESTIONS:\n- I’m confused by the columns in Table 1. How can D3M be both deteriorating and non-deteriorating?\n- How does the use of VBLL compare with other Bayesian approaches?\n- Other typos and clarifications:\n    - There is a duplicated “operating operating” in the first sentence of the abstract.\n    - “Underlines in our results indicate that D3M is close to the best baseline.” What is “close” here? It seems somewhat arbitrary given the differences in performance and nearly 10 points in some cases.\n    - Please, clarify what exactly an N-shot scenario is. Also explain that the {10, 20, 50} in Table 2 correspond to shots in the caption.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a methodology to address post-deployment deterioration of model performance due to domain shift in the data. The authors propose a train, calibrate, and deploy strategy where, with the base model, there is a variational bayesian last layer (VBLL) to model the posterior predictive distribution over the target classes. The VBLL layer helps to identify deployment time deterioration of the model based on the disagreement rate over K drawn samples from the variational posteriors. The method is evaluated on publicly available datasets (i.e., vision, healthcare) and shows comparable performance with the baselines in detecting domain shifts in data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The codebase is available, structured well, and documented for review.\n2) The introduction of the paper is well placed and motivates the approach.\n3) The algorithm is clearly explained with appropriate figures and descriptions\n4) Section 2.3: Algorithmic insights is well explained\n5) Experiments are conducted on publicly available datasets\n6) The related work is properly structured.\n\nWeakness:\n1) The abstract has two “operating”. Needs to proofread the manuscript.\n2) Evaluation on the UCI Heart Disease, CIFAR 10.1, and Camelyon 17 datasets can be more thorough with holdout participants or a dataset subset. \n3) Temporal shifts are only shown for the UCI IM dataset with an artificial shift in the test data (i.e., it’s not evident in the original data). This compromises the motivation of the problem. I strongly suggest evaluating on another temporal dataset with an inherent temporal shift.\n4) The impact on the target classification due to the VBLL layer is not clearly discussed\n5) Please provide an ablation study on the parameters: temperature and alpha.\n\nQUESTIONS:\n1) How does the VBLL layer impact classification accuracy?\n2) How can the VBLL layer be extended to a regression problem?\n\nLIMITATIONS:\n1) Evaluation can be more thorough. Please select a new temporal dataset for evaluating the method\n2) Need to proofread the manuscript", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of knowing when a machine learning model’s performance is getting worse after deployment especially in real-world settings where data changes over time and labels are not available. The authors introduce D3M, a method that checks for model disagreement to detect harmful changes in data. Unlike typical drift detectors, D3M aims to avoid false alarms when the model is still performing well and to reliably catch cases where performance is truly dropping. The method is supported by theoretical guarantees and shows strong results on both standard benchmarks and a large internal medicine dataset, making it promising for use in critical applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths - The writing is very clear and the paper is well-structured. The paper offers theoretical justification and broad empirical evaluation.\n\nWeaknesses - See questions\n\nQUESTIONS:\n1. Line 1: operating repeated twice\n2. Can the authors provide the definitions of deteriorating and non-deteriorating shifts directly in the introduction? \n3. Can the authors explain as to why measuring disagreement via a variational approach that too only for the last layer is a useful method for measuring disagreement? In my opinion, wouldn't there be limited diversity or limited choice of hypothesis class exploration if only the last layer is trained in that fashion?\n4. How do the results change (qualitative explanation is sufficient) if there are multiple classification heads instead of the variational training?\n5. After T rounds, can you provide a graph of how the base model (ID) threshold changes? Is it more or less very similar to each other?\n6. In addition to metrics such as TPR, FPR, can the authors also evaluate a unifying metric such as Matthew's correlation coefficient in order to holistically characterize performance?\n7. Can the authors also qualitatively compare their work to conformal prediction and risk control? Conformal prediction can also be an easy to adopt strategy to explain model failures and even identify distribution shifts.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}