WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales
Authors: Drew Prinster
Xing Han
Anqi Liu
Suchi Saria
Venue: ICML 2025 poster (2025)
Keywords: AI Safety Monitoring
Conformal Martingale
Conformal Prediction
Changepoint Detection
Sequential Testing
Adaptive Testing
Anytime-Valid Inference
E-values
E-processes
Distribution Shift
Primary area: general_machine_learning->evaluation
TL;DR: For monitoring the post-deployment risks of AI/ML models (to adapt to mild shifts and detect severe ones), we propose and evaluate a weighted generalization of conformal test martingales

Abstract: Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but also continual, post-deployment monitoring to quickly detect and address any unsafe behavior. Methods for nonparametric sequential testing---especially conformal test martingales (CTMs) and anytime-valid inference---offer promising tools for this monitoring task. However, existing approaches are restricted to monitoring limited hypothesis classes or ``alarm criteria'' (e.g., detecting data shifts that violate certain exchangeability or IID assumptions), do not allow for online adaptation in response to shifts, and/or cannot diagnose the cause of degradation or alarm. In this paper, we address these limitations by proposing a weighted generalization of conformal test martingales (WCTMs), which lay a theoretical foundation for online monitoring for any unexpected changepoints in the data distribution while controlling false-alarms. For practical applications, we propose specific WCTM algorithms that adapt online to mild covariate shifts (in the marginal input distribution), quickly detect harmful shifts, and diagnose those harmful shifts as concept shifts (in the conditional label distribution) or extreme (out-of-support) covariate shifts that cannot be easily adapted to. On real-world datasets, we demonstrate improved performance relative to state-of-the-art baselines.

WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales
Drew Prinster' Xing Han
Anqi Liu'
Suchi Saria
Abstract
ditions is essential for responsible deployment. However, even meticulously trained models with apparent reliability guarantees can fail abruptly when shifts in the data distri- bution or operational environment violate the underlying conditions they were designed for (Amodei et al., 2016), That is, unforeseen deployment conditions can make it im- possible guarantee reliability for all situations in advance. Consequently, there is growing recognition for the need to continuously monitor deployed Al systems for determining when model updates are required to mitigate downstream harm (Vovk et al., 2021; Podkopaev and Ramdas, 2021 b; Feng et al., 2022; 2025). In this work, we moreover argue that such monitoring methods should ideally perform at least three key functions: (I) maintain end-user reliability and minimize unnecessary alarms by adapting online to mild or benign data shifts; (2) rapidly detect more extreme or harmful shifts that necessitate updates; and (3) identify the root-cause degradation to inform appropriate recovery.
Responsibly deploying artificial intelligence /Al) machine learning (MI.) systems in high-stakes set- tings arguably requires not only proof of system reliability, but also continual, post-deployment monitoring quickly detect and address any un- safe behavior. Methods for nonparametric se- conformal test mar-
quential lesting-especinng tingales (CTMs) and anytime-valid inference- offer promising tools for this monitoring task. However, existing approaches are restricted to monitoring limited hypothesis classes or "alarm criteria" (e.g.. detecting data shifts that violate cer- tain exchangeability or IID assumptions), do not allow for online adaptation in response to shifts, and/or cannot diagnose the cause of degradation or alarm. In this paper, we address these limita- tions by proposing a weighted generalization of conformal test martingales (WCTMs), which lay theoretical foundation for online monitoring for any unexpected changepoints in the data distribu- tion while controlling false-alarms. For practical applications, we propose specific WCTM algo- rithms that adapt online to mild covariate shifts (in the marginal input distribution), quickly detect harmful shifts, and diagnose those harmful shifts as concept shifts (in the conditional label distribu- tion) or extreme (out-of-support) covariate shifts that cannot be casily adapted to. On real-world datasets, we demonstrate improved performance relative to state-of-te oo-tth---tttttttt baselines.
For example, consider a healthcare use case where the goal is to predict the risk of sepsis, life- rife-threatenin infection, Y from electronic health record inputs X (e.g., vital signs, lab tests, medical history, demographies) using an AI/M system (e.g., Adams et al. (2022)). Various clinical data shifts pose challenges to monitoring in practice (Finlayson et al., 2021). Figure illustrates hypothetical synthetic-datt shifts that can be interpreted through this sepsis example, where for simplicity X only represents a patient's age. An example of benign shift is mild shift in patient demo- graphics primarily toward young adults; Figure la shows corresponding covariate shift in the marginal X distribution. Such a mild shift to younger population would be benign, as younger individuals are well-represented the training data and also tend to have lower, less variable (casier to predict) sepsis risk- so, there no need for an alarm. On the other hand, harmful shift examples include if the Al tool were deployed with much older population than observed in the training data (e.g., Figure 1b) or if a new microbial strain were to arise that was especially severe in children (e.g., Figure Ic). In any such harmful shift case, swift de- tection and root-cause analysis is essential to initiate and inform retraining, to ultimately minimize harm.
1. Introduction
As AI/ML systems become integral to real-world applica- tions, ensuring their safety and utility under evolving con-
'Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA. Correspondence to: Drew Prinster drew- prinster comm" Xing Han <aaronhan22366 (a233@@mmallccoo com'.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Recent advances in anytime-valid inference (Ramdas et al.,

---

WATCH: Adaptive Monitoring for Al Deployments via Weighted-Conformal Martingales
(a)
Benign covariate shift
(b)
Harmful covariate shift
(c)
Harmful concept shift
(Mild shift in X)
(Extreme shift in X)
(Fundamental shift in YJX)
10
10
Y (eg. sepsis risk)
Pro- Pra-Changepoint Data
Y (eg. sepsis risk)
Pre-Changapoint Data
Y Yegg sepsis risk)
Pre-Changepoint Data
Post: wwwe--hhaneepinttooo Data
Post Changepeint
Post Changepoint Data
Change in YJX
relation
4
2
20
40
60
20
40
80
X (eg, age)
X (eg, age)
(eg, age)
X-CTM (Adaptation Criterion)
..... XCTM Adaptation Threshold
WCTM (Alarm Criterion)
WCTM Alarm Threshald
Changepoint
Moan Martingale Vels
Mean Martingale Vals
Mean Martingale
WATCH Notice: Mild covariate shift detected, adapting accordingly...
10a
(!) WATCH Alarm (9) Fundamental concept shift detected, retrain model!
10
10
10'
(1) WATCH Alam Extreme covartate shift detected. retrain modell
101
101
101
10-2
10-2
10-2
200
400
600
000
1000 1200 1400 1600
400 600 800 1000 1200 1400 1600 Test (Deplayment) Datapoint Index
200
0
200 400 600 800 1000 1200 1400 1600 Test (Deployment) Datapoint Index
Test (Deplayment) Datapoint Index t
Figure 1: Each column represents data shift scenario: the top row is a simulated shift example and the bottom row shows WATCH's s response, averaged over random seeds. WATCH raises an alarm to retrain the AI/ML once the WCTM (blue) exceeds its alarm threshold; meanwhile, an X-CTM (gray)-a standard CTM that only depends on inputs X, and thus only detects covariate shifts- shifts-dynamins dynamically initiates the WCTM's 's adaptation phase and aids in root-cause analysis. In (a), the X-CTM starts the WCTM's I' adaptation phase, which allows the WCTM to avoid raising unnecessary alarm. In (b), the extreme covariate shift causes the WCTM to raise alarm, indicating that the covariate shift too severe to be adapted to. (c), the illustrated concept shift causes WATCH to raise an alarm, but without the X-CTM detecting a shift in covariates X this allows WATCH to diagnose the root-cause of the alarm as a concept shift in Y X.
2023) and especially conformal test martingales (CTMs) (Volkhonskiy et al., 2017; Vovk, 2021) offer promising tools for AI monitoring with sequential, nonparametric guaran- tees. However, existing CTM monitoring methods (e.g.. Vovk et al. (2021)) all rely on some form of exchangeability (e.g., IID) assumption in their null hypotheses informally, meaning that the data distribution is the same across time or data batches-ae and result, standard CTMs can raise necessary alarms even when shift is mild or benign (c.g., Figure la). Meanwhile, existing comparable monitoring methods for directly tracking the risk of a deployed Al (e.g., Podkopacv and Ramdas (2021b)) tend to be less efficient than CTMs in their speed of detecting harmful shifts, data usage, and/or computational complexity (Sec. 4.3).
alarms), while quickly detecting harmful shifts (e.g.. Figure 1b & c) and enabling root-cause analysis.
2. Background
Notation: Assume an initial dataset Z = {Z where each datapoint is a feature-label pair, Z; := (Xi, Yi) X x 2. For simplicity, further assume that an AI/L model ğ is pretrained on a separate dataset.' After deploying the AI/ML model, the test points are ob- served sequentially at each time = 1,... T (batch data can be given random ordering). though for simpler expo- sition, we initially focus on = 1. We abbreviate indices [m] := {1, ..., m). Random variables are denoted with cap- ital letters (e.g., 7,) and observed values with lowercase (e.g., 2)). For Z := Z1:(n+t). we let Fz denote the joint distribution function, Jz the joint density function. For a set of distributions Fz, we write Z1, Zz,... ~ Fz to mean the sequence has some unknown distribution Fz E Fz.
Our paper's contributions can be summarized as follows:
Our main theoretical contribution is to propose weighted-conformal test martingales (WCTMs), con- structed from sequences of online weighted-conformal p-values, which generalize their standard conformal precursors. WCTMs lay a theoretical foundation for sequential and continual testing of broad class of null hypotheses beyond exchangeability, such shifts that one aims to model and adapt to.
2.1. Conformal Prediction and Conformal p- Values
Standard conformal prediction (CP) (Vovk et al., 2022) is an approach to predictive inference: the task of converting a black-box AI/ML prediction, fu( ((Xn+1), into a predictive confidence interval/set, Cin (X,+1), that should contain the true label with a user-specified rate, 1 - a E (0, 1) (e.g..
For practical applications, we propose WATCH: Weighted Adaptive Testing for Changepoint Hypotheses, a framework for AI monitoring using WCTMs. WATCH continuously adapts to mild or benign distribution shifts (c.g., Figure la) to maintain end-user safety and utility (and avoid unnecessary
'This corresponds to split conformal setting (Papadopoulos, 2008). but our theory also extends to full conformal (Vovk al., 2022). which avoids splitting data at heavy computational cost.
2

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
90%). This objective is called valid (marginal) coverage?
Standard conformal p-values (Eq. (2)) are valid or "bona- fide" p-values, in the usual statistical testing sense, for the exchangeablity null hypothesis (Vovk et al., 2022). That is, sid
P(Yn+1 E (X 2 1 a.
(1)
assuming HD" then Pa+1 ~ Unif[O, 1] (Vovk, 2021) and
Constructing the CP set, (X,,)())))) requires a labeled calibration dataset, 3 Zi:. and a "nonconformity" scor fune- tion, S : X' x 3 + IR, which generally uses the prefit ML predictor to quantify how "strange" a point (x, y) is relative to the training data. common example is the absolute-value residual score, S(x,y) y) = 1 -xxx((
Pxg (P S a) Sa a Aa (0,
(5)
Thus, observing an extreme value of pn+1 is evidence against Hy (Eq. (4)), or evidence for distribution shift,
2.3. Standard Conformal Test Martingales: Testing Exchangeability Online via Betting
Though not always framed as such, the CP set (Xn),.((nn-)) can be computed via a conformal p-value, which places the "strangeness" of the test point's score in context of the calibration-date scores. That is, with e;- == S(z;, 3i) for all i E [n + 1), Vovk et al. (2022) defines the conformal p-value for test point Zn+1 relative to calibration data Zı.. as the fraction of the n + 1 scores, U1:(n+1). that are at least as large as t1 with ties broken uniformly at random:
Standard conformal test martingales (CTMs) (Volkhonskiy et al., 2017; Vovk, 2021; Vovk et al., 2021) continually ag- gregate information from sequence of conformal p-values (Eq. (2)) to perform online testing of the exchangeability null, HD They can be constructed on top of any standard CP method, and thus on top of any conformalized AI/ML model, and used to monitor for deviations from HD. We will describe CTMs from a game-theoretic teestin-bby-betting interpretation (Shafer and Vovk, 2019; Shafer, 2021).
Mi : v > Un+13l + un Iii e4= -0++1 H
Pu+1
(2)
n 1+11 1
where Un+1 O0, 1}; that is, Mh+1 is obtained from an independent standard uniform distribution." 4 The CP set In (X.+1) can then be understood as the subset of labels y E y that would not result in too "extreme" of a test-pou rest-point p- p-value, P (Xx+1.,)) y) (this is explicit notation for Pa+1 to emphasize its dependence on the candidate label y):
Under H46 stochastic process Mu. M1, ..... Mc, ... is
CTM if it is a nonnegative martingale- -.e..l.eeeee............. M. 2 0 for all t and Exg [Me MM..... M,-1] = M construccee from a corresponding sequence of conformal p-values P........... ... via an appropriately defined "betting process." " In the game-theoretic interpretation, a bettor has initial wealth Mo (usually set to Mo 1, for simplicity). Then, at each time t = 0, 1, ..., she may use wealth M, to bet on the value of p to be observed next; once P++1 is revealed, she receives her reward and/or pays her losses, resulting in a new total wealth Mz+1. The bettor bets against the null hypothesis Hg if 7g8s is true, then P. she cannot expect to outperform random guessing; so, if the bettor grows her wealth by large factor M,/M, >> 1, this is evidence that the bettor "knows" an alternative hypothesis more accurate than H (Vovk, 2021). M /o can be taken as an "anytime-valid evidence against HC. In particular, Mt / Mo is also an "e-process" for H$55 (see Vovk and Wang (2021); Ramdas et al. (2023)).
(n)aa(Xnt1) = {y E y: (x+11(X+1,,)) > a}.
(3)
2.2. Exchangeability Underlies Standard CP Validity
For the standard CP set in Eq. (3), the coverage guaran- tee in Eq. (1) holds assuming that the calibration data Z1.. and test point Za+1 are all exchangeable- -itiungeabbe--iuiiiellll.y this means that the data distribution is invariant over time (e.g., independent and identically distributed or "IID" data). Formally, exchangcability means that the joint distribu- tion, F2, invariant to reorderings or permutations a:: that is, Fz (=o(1): ... To(n+1))
sid Unif(O, 1] and
F2(21,-.. ... Zn-1) )) for all 21, Zn+1 Z and all a [n + - In 1]. With F denoting the set of exchangeable joint distributions, we can write the assumption of exchangeability as (composite and nonparametric) null hypothesis:
More formally, define a betting function as function h [0. 1] + [0, oo] that integrates to one, i.e., fo' h(w)du = 1. We follow Vovk (2021) and assume the betting function
HSx : Z1, Zz,,... Zut1 ~ F
(4)
h (p) = 1 + e(p- 0.5),
(6)
Heere, marginal means on average over the draw of calibration and test data; see, e.g., Foygel Barber et al. (2021) for more details. FFor t >>> 1, the calibration data might be kept fixed (i.e.. Zi..), or may include past test points (i.e.. (1(()))));; for now, we focus on = to avoid this distinction and simplify exposition. *Conservative conformal yaluue set uh+1 := 1; the random variable p corresponding to Pn+1 is called p-variable, hut will often refer to both as p-values for more common terminology.
where E E= := 1-,, (E is selected somewhat arbi- trarily). The intuition is that 0 corresponds to betting on smaller p-values, >> 0 corresponds to betting on larger p-values, and e = 0 represents not betting. A CTM can be constructed by selecting a betting strategy he, at each time that may depend only on the past p-value observations Pi.... 11 this paper uses the "composite jumper martin-
3

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
gale" strategy described in Vovk et al. (2022). Then, con- formal (test) martingale Mf [0, 1]" [0, oo), 0, 1, ,, is constructed by accumulating the "wealth" attained by the bets on the previous conformal p-values, P1.P2. Pt-1- That is, for a single bet 6 E E at each time i, the product Ih he, (p:) is a CTM that can be interpreted as the factor by which the bettor has multiplied her wealthy by time I. More broadly, CTMs may accommodate more flexible bet- ting strategies (while only depending on past p-values), v, that may distribute bets across E at each time, as
framing conformal prediction, while in our paper (sequen- tial) hypothesis testing is the primary focus; the conformal p-values in that paper are deterministic and attain conserva- tive validity, whereas we incorporate randomness to achieve exact validity; lastly, whereas that paper focuses on a batch- data setting where all the data are observed simultancously, we focus on an online monitoring setting where each p-value is determined only by the current and past data, and we im- portantly present new theory for when a sequence of WCP p-values can be distributed uniformly and independently, which is important for monitoring with WCTMs.
f M, : / hha ((laa A t.
3. Theory and Methods: Weighted-
Conformal p- Values and Test Martingales
For example, in our implementations we take v to be the composite jumper Markov chain described by Vovk et al. (2022) to determine how the "wealth" values are spread across the betting space E at each timepoint.
Our main theory and method contribution is to introduce weighted-conformal test martingales (WCTMs), constructed from a generalized version of weighted- cooforrmal p- g-values, which expand the scope of their standard conformal analogs. WCTMs enable more customizable and informative alarm criteria than standard CTMs, and our specific variants enable adaptation to mild shifts, while in response severe shifts they raise alarms and enable root-cause analysis.
A standard CTM (Eq. (7)) can be used to test the exchange- ability null 748 (Eq. (4)) online either by using M./ Mo as an anytime-valid evidence metric, or by raising an alarm when Mr/M, exceeds some user-defined threshold c. That is, by Ville's incquality for martingales (Ville, 1939), CTMs achieve anytime-valid control over the false alarm rate (i.e., over the probability of an alarm despite Hg' being true).
3.1. Generalized Weighted-Conforme p- p-Values
In this section we present a general version of weighted- conformal p-values. Sequences of these generalized weighted-conforma p-values will lay a theoretical foun- dation for online testing of a broad range of null hypotheses beyond exchangeability. To begin, observe that standard conformal p- p-values (Eq. (2)) can be equivalently written as
Pag (3 :: Mr/Mo 2 c) S 1/c,
which is sometimes referred to strong validity due to its control over ever raising false alarm (Vovk, 2021).
2.4. Weighted Conformal Prediction for Adapting to Distribution Shifts
n+1 Σ 1=1
He > enti3 + un+1 He, = =
Whereas Standard CP computes valid predictive confidence sets assuming exchangeable data, weighted conformal pre- diction (WCP) (e.g., Tibshirani et al. (2019); Podkopaev and Ramdas (2021a); Barber et al. (2023); Prinster et al. (2024); Barber and Tibshirani (2025)) generalizes standard CP to attain valid coverage (Eq. (1)) even under various distribution shifts. WCP methods adapting to distribution shift, that is by computing CP sets on reweighted version of the empirical calibration set's scores, which in effect can modulate the size of the predic- tion sets to maintain coverage.
Pn+1
n
where the purpose of this notation is to isolate where the comparison of Un+1 to each i-th score is given uni- form weight 1 n+I. For some arbitrary weight vector ő = (wi,..., 1x+1) E [o, 1] where En+l w. = 1, it is then straightforward to define weighted-coofformal p-values as
thus an approach to
n+1
w H > Until + unt1 1(; = =
P
1=1
(9)
As we will see in the next section, WCP methods are asso- ciated with weighted-conformal p-values, special-case of which was introduced in Jin and Candès (2023) for standard covariate shifts, based on Tibshirani et al. (2019). Concur- rently with our paper, Barber and Tibshirani (2025) also leverage weighted-conforme p-values to unify various the- ories of conformal prediction. However, there are several key differences: regarding motivation, that work discusses connections to (one-shot) hypothesis testing as a means of
Note that existing (split or full) weighted CP methods (e.g.. Tibshirani et al. (2019); Podkopaev and Ramdas (2021a); Prinster et al. (2024) and certain methods in Barber et al. (2023)) can be also be defined by weighted-conformal p- values by plugging p pi (X y) in for Pa+1 P (X,+1, y) in
'Here, "weighted" refers to weights the score distribution prior to computing a p-value, not to weighting the p-value itself.
4

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Eq. (3), where to is an appropriately chosen weight vector. 6 To understand the meaning of the weights w for our hy- pothesis testing purpose, we draw from the general view of weighted CP described in Prinster et al. (2024), which expounded analysis from Tibshirani et al. (2019): For setup, let Eg denote the event {Z1,.., 1 = {E1,.... 1n meaning that the empirical distribution of the datapoints has been observed, but we do not know whether Z,- = 2, and so Then, the oracle weights To would be given by entries
Theorem 3.1. (Independence and exact validity for HolP) of online WCP p-values.) For any ' E 14, let Holf) denote the set of all assumptions on the joint PDF. f, including that any uppoximations used to estimate T (Eq. (10)) are eactly accurate. Let v == T (f) be the corresponding esti- mated weights and ..,,..., OCT E (0, 1) some user-defined significance levels. If Ha(f) holds for the true f. then Pfe P . ... are IID uniform on [0.1]:
P P Sa,...., Pf S <aT) 01 ar. (12)
/ = P[V++1 = 1/3 E,}
(10)
Proof Sketch: We defer a full proof to Appendix B. The proof builds on those for standard CTMs in Vovk (2002) and Vovk et al. (2003), which leverage the idea of "reversing time," while drawing on ideas from Tibshirani et al. (2019) and Prinster et al. (2024). We imagine that the sequence of data observations (21, 2T) is generated in two steps: first, the unordered bag of data observations, {21,...,,y)) is generated from some probability distribution; then-here generalizing Vovk (2002) and Vovk et al. (2003) by weight- ing permutations according to their likelihood all possible permutations o of the values {z1....zy) each possible sequence (=a(1)) (T)) is chosen with proba- bility proportional to f(zo(1)) .... Zo(r)), where f := fz is the bebbaabilityddnsiii function? for the distribution Fz. Roughly (ignoring borderline effects), the second step en- sures that, conditionally on knowing (=1.-.,, (21....,,7) 2r. (and there- fore unconditionally). that Pf has a standard uniform distribution; when Zí = Zx(T) is observed, this settles 10 (T) and conditionally on knowing (=1,-.,, 2 and ZT (T) (and therefore, after relabeling indices, on knowing {2), ..., 11) that Pf also has a standard uniform distribution, and so on.
E Eoro (m+ (20(1)) .... to(n+))
Eo (20(1)) (o(11))
where f is the joint probability density function (PDF) and a is a permutation of [n + 1]. In words, the oracle weight wi we would ideally use for w; is the probability that the test score Vn- took on the value 'y conditioned on the em- pirical distribution Eg. For further exposition, see Prinster et al. (2024). While for arbitrary S, computing Eq. (10) is intractable due to requiring knowledge of f and the factorial complexity, we next turn to how simplifying or approximat- ing v. can be useful in practical hypothesis testing.
3.2. Expanded Hypothesis Testing with Weighted-Conformall p-aioorall--aluuue p-Vauees
We now look at how the weighted-conforman p-values intro- duced in Eq. (9) can be used to sequentially test a variety of nonparametric null hypotheses beyond exchangeability. Let us begin by using Hu(f) to denote any set of assump- tions on the true f some approximations on that we wish assume are true in our null hypothesis (e.g., which may enable testing for evidence of inaccurate estimation). Hereon let w :- w (f)) denote the weight vector com- puted with this approximation. For example, if we assume exchangeability as our null hypothesis (Ho(f) = H((A)) ) then f (2a(1). .... 1)) = f(ă1, .... 1n+1) and Eq. (10) reduces to W n+T. which recovers standard conformal
the value of Pf" =
3.3. Main Practical Testing Objective: Testing for {Concept Shift or Unanticipated Covariate Shift)
We now provide a worked example for how weighted- conformal p-values can be used to test common assump- tion in the robust ML literature: namely, the covariate shift (Shimodaira, 2000; Sugiyama et al., 2007) assumption, where the marginal input distribution Fx may shift to some other distribution Gx at test time, but the conditional label distribution F is assumed to remain invariant. In the standard covariate shift setting considered in Tibshirani et al. (2019), the oracle weights uं in Eq. (10) are proportional to density-ratio weights gx (X,) 333/f(XXX [Xx)). A common goal is to adapt to covariate shift by learning density-ratin function ((x)) R gx (x)) (z) for reweight- ing the data (e.g., Sugiyama et al. (2007); Tibshirani et al. (2019); Yang et al. (2024); Zhang et al. (2024)).
p-values. More generally, (f) may denote some other set of invariance assumptions on if or density-ratio estimates as- sumed to be accurate (see Sec. 3.3 and App. C for worked example for our main testing objective). Then, p n+1 is valid and exact paluu for the null hypothesis (f)
P Ho(l) (Po+1 S ax) =
(11)
approximate
Moreover, when a sequence of weighted-conforme p-values computed online. then they are not only distributed uni- formly, but also independently, at each timestep. We next state this formally.
However, whereas robustness results often rely on the density-ratio estimator being reasonably accurate, i.e.
*Conservatively valid weighted CP methods would need to further set M = 1 Eq. (9) to reduce appropriately.
MMoee generally, the Radon-Nikodym derivative.
5

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
m(x) ~ gx (x(x)/fx(x) (z), there is little work on testing for whether this reliability criterion is achieved in practice. Let k denote the set of distributions for X with density-ratio function (ar) with respect to the true Fx E Fx. Then, we aim test the following joint null hypothesis:
1966; Shiryaev, 1963) while achieving a weaker form of validity. Consider the Shiryaev-Roherts procedure applied to WCTMs M,, whose k-th stage alarm time is
t-1
Mr
The == min Tk-1 Tk-1:
Σ
2 KEN. (15) M,
H60 = (x)
rid F x Ex, E In I tad - 1]
Tk-1
(Xnte, Yu+t)
rid ~
Fx,x x G()) 1 12 lad,
This Shiryaev- procedure based on standard CTMs controls the average run length (ARL) under Hf(f), where the procedure is expected to be reset after timesteps.
(13)
which can be roughly read as assuming that Y X is in- variant and that ((x) = 9x (z)/ (z). Thus, observing an extreme value of p would convey evidence against 7 meaning that there has been a concept shift in Y | [ or that the density- adaptation ((z) is inaccurate.
Proposition 3.3. (WCTM-based Shiryaev-Robert ARL con- trol) Let denote the Robbers stopping time (Eq. (15)) based on a WCTM Mo. MI..., Mr... constructed for the null hypothesis Ho(f). Then, assuming Ho(f),
E Encch P [T]] 2c.
(16)
3.4. Weighted-Conforee Test Martingales for Continual Monitoring
3.6. WCTM Implementation in WATCH: Dynamic Online Adaptation and Root-Cause Analysis
With Theorem 3.1 at hand, weighted-conformall test mar- tingales (WCTMs) can be constructed from a sequence of weighted-conformaal p-values to enable continual, anytime- valid monitoring of a customizcable null hypothesis Ho (f). Just as in the construction of standard CTMs (Section 2.3), WCTMs require betting function (we assume the betting function in Eq. (6)) and a strategy for placing bets et at each time & that can only depend on past p-value obser- vations (we use the composite jumper strategy from Vovk et al. (2022)). Then, a WCTM is constructed by feeding a sequence of weighted-conformall p-values p. , ...,,P.:.. ................................ .... into Eq. (7) (i.e., by setting P. == p,"" in Eq. (7)). Due to the P. being IID uniform on [0.1] under Hf) (Theorem 3),,, the stochastic process Mo. Mi..., M, is nonnegative test martingale for (f). and by Ville's incquality (Ville, 1939) it achieves the anytime-valid control over false alarms in the following proposition (proof in Appendix B).
All our implementations and experiments in this paper focus on WCTMs that continuously adapt to mild covariate shifts (e.g., Fig la) while raising alarms in response either extreme covariate shifts (e.g., Fig 1b) concept shifts in Y|X (e.g., Fig Ic). These goals correspond to monitoring the H. null hypothesis given in Eq. (13). Our adaptation procedure performs online density ratio estimation with an online probabilistic classifier (e.g., a multilayer perceptron or logistic regression model similar to in Zhang et al. (2024)) to update the estimation of ((+( (x) at each timestep t.
Online Adaptation with Dynamic Initialization For the initial adaptation stage, a key question is when to begin the adaptation procedure, or when to begin estimating (1 (x). Our methods make this determination automatically and dynamically by running secondary standard CTM that is restricted to only monitoring for changepoints in the marginal X distribution via a nearest- nonconfor- mity score (Vovk et al., 2021). At deployment, the main WCTM method is initially a standard CTM (with uniform weights); then, once the secondary " "X-TMM" method ex- ceeds pre-determined adaptation threshold (see gray paths in Fig 1) at some time lad, this triggers the adaptation of the main WCTM monitoring method (blue paths in Fig 1) and corresponding weighted CP intervals. For efficiency purposes, after the adaptation time fad. the CP calibration set [n + tadl is treated as fixed (no longer adding test points to it online), meaning the saighted-confomme p-values are computed summing only over [n + ta 1] U {n + t).
Proposition 3.2. (WCTM anytime-valid false-alarm con- trol) Let Mo,M,.... Mi.... M.,... be WCTM constructed from the sequence of weighted-conformal p-values 5 s ... pl. pi.... Then, assuming Ho, pf p2,
P PHO(f (3t: BBt:M//M, 2 c) S 1/c.
(14)
3.5. Scheduled or Multistage WCTM-Based Monitoring
The anytime-valid guarantee in Eq. (14) controls the proba- bility of raise false alarm at any time in an infinite sequence of observations, but this can be overly strong, especially when one has set time horizon in mind for monitoring. For such "scheduled" or "multistage" monitoring, we im- prove the efficiency (speed in detecting true shifts) by fol- lowing Vovk et al. (2021); Vovk (2021) and augmenting WCTMs with standard changepoint detection metrics such as CUSUM (Page, 1954) and Shiryaev-Robert (Roberts,
p
Σ
((t))
1f > ente) +un+t 11 (w; = matt))
vEln+ew-
(17)
where here :iن0) x (Xx).) See Appendix C for details.
6

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Medical Expenditure Panel Survey (MEPS)
LIhOA Nitty Predicser Mimaistmmaa
Retogamned DMorkednn Inputti Chanpee
Mattinek Nuto derygamm-- Parbaring Coteaton
Eotyan doderty
(altni (92)
100
Teat Deplymman Coapehn Frcec
Superconductivity
Interd Waih
SrrdicMorr Servay!
palan Changepars Altr Crapgess
Me Mortarin Citatin)
hoa dlll Ri22
Bike Sharing
Coverage
marval Widee
Hatogramm: of 3-lauum
D4aritdtng imputst
1eoyttene-a 14 Borka Creedton
Coustbble Schekut Martirting battor
100
300
130 720 14d (2gp an Eitaeettiiee
1od (hiplly vand Uirana malle
Figure 2: Tabular data results for the henign covariate shift setting to evaluate the adaptation ability of proposed WCTM methods (bluc); all values are averaged over 200 random seeds. Training and calibration sets were sampled uniformly at random (with 1/3 of the total data used for training and calibration each), while post-changepoint test-set datapoints were sampled from the remaining holdout data with probability proportional to exp(A {(r)). The maitt-magn scalar À for each dataset was set as Xeep = 5.0, Aup = 2.5, and Asa 5.0. The function/ it was selected simuluate realistic shifts as described in the main text. Error regions represent standard errors for coverage, martingale paths, ans Shiryaev-
paths, and interquartile range for interval widths. Whereas standard CTMs (orange) raise unnccessary alarms with their anytime-valii valid and scheduled monitoring criteria, WCTMs avoid doing so by adapting online to the shift. That is, WCTMs maintain target coverage, adapt by increasing interval sharpness, and avoid unneeded alarms.
Root-Cause Analysis with Parallelized WCTM and X- CTM The parallel implementation of both the primary WCTM and the secondary X-CTM method furthermore enables root-cause analysis, that is to diagnose whether performance degradation was due to a harmful covariate shift (e.g., Fig 1b) or a fundamental concept shift (e.g.. Fig Ic). That is, both the WCTM and the X-CTM have detected changepoints, then harmful shift can be diagnosed as an extreme covariate shift (c.g.. Fig 1b); on the other hand, if the X-CTM does not detect changepoint, this suggests that no covariate shift is present, and the WCTM alarm was instead due to a concept shift in Y |X (Figure 1).
be found in Appendix E. Code to reproduce all experi- ments is available at the following repository: https: //github. com/ aaronhan222 watch.
Baselines We compare the proposed WCTM methods that constitute WATCH against standard CTMs (Vovk et al., 2021) in all experiments. To evaluate detection speed on true harmful shifts (sec 4.3), we also compare against methods for directly performing sequential hypothesis testing and changepoint detection on the set-prediction miscoverage risk, proposed by Podkopaev and Ramdas (2021b).
In all experiments and for all baselines, the underlying ML predictor being monitored was a neural network. On the tab- ular data, we used the scikit-lear (Pedregosa et al., 201 1) MeeReeressor (with L-BFGS solver and logistic activa- tion); for the image data, we used a 3-layer MLP with ReLU activations the MNIST datasets and ResNet-32 (He et al., 2016) on CIFAR-10 10 datasets. For weight estimation, we use a 3-layer MLP with ReLU activations to distinguish between source and target distributions.
4. Experiments
We conduct a comprehensive empirical analysis of the WATCH framework on real-world datasets with various distribution shifts. Our results show that WATCH adapts effectively to benign shifts (Section 4.1) and triggers alarms when it fails to adapt (Section 4.2), while also quickly de- tecting harmful shifts with little delay (Section 4.3). De- tails on the datasets, models, and additional results, can
7

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformll Martingales
No Corruption
Minor Berign Corruption
Benign Corruption
Extreme Corruption
(Anytime-Valid Monitoring Criterion)
(Arytime-Valid Moritoring Criterion)
(Anytime-Vaidd Moritoring Criterion)
JAnytime-Vllidd Monitoring Criterion)
Mertingsle Vake 166.0
WC
prrpposed
WC jn (propooed
M peoposed
Fobuc at 2022)
La
CTM (tevi at 20221
CTM Drani al. 2023)
20
WITH jpropmedd
Changeraant
Chii cepoort
300
CTN (braket
2022)
23"
Ehreahold
Alaim Braahou
Nam Shranndd
Partingale
750 25
die sie
L50 26 330 (Deplyymnt Datapoint
300
I6 20 325 (Deployment) Datapaint
401
316
1oo 266 IDeploymeni Datapoint
300 255 20 405
Test (Dsploymett Datapeint C
(b)
(c)
(d)
Figure 3:
Example martingale trajectories of WCTM and CTM on CIFAR-IO-C with increasing levels of corruption. WCTM adapt to milder shifts to avoid unnecessary alarms while still detecting severe shifts. All plots are averaged from 5 random experiments.
Table 1: Average detection delay (ADD) computed over 200 random seeds for each monitoring method and dataset (error bars are standard errors). Anytime-valid monitoring methods WCTMs (proposed), CTMs (Vovk. 2021). and sequential testing from Podkopaev and Ramdas (2021b) (PR-ST); multistage monitoring methods are Shiryacv-Rober Roberts (SR) procedure applied WCTMs (proposed), SR applied to CTMs (Vovk, 2021), changepoint detection on the miscoverage risk from Podkopacy and Ramdas (2021b) in both online (R-CCD-nlline) and minibatched (PR-CD-50) variants. Results corresponding to our method are in blue. The best ADD are bolded.
Monitoring Criterion (+)
Anytime Valid Criterion
Scheduled, Multistage Criterion
Monitoring Method (-)
(WICTM
PR-ST
SR via (W)CTM
PR-CD-online
PR-CD-50
Dataset (L)
CP Type (4)
ADD (+: SE
ADD (+: SE)
ADD (+: SE)
ADD (# SE)
ADD (4: SE)
Weighted
115.8 (1338)
536.6 (+23.9)
102.1 (13.2)
111.3 (#3.8)
196.5 (=5.4)
MEPS
Standard
114.8 (+3.7)
475.2 (+18.8)
101.8 (+3.2)
111.0 (+39
194.5 (=5.3)
Weighted
190.0 (#:8.)
602.5 (=24.7)
155.6 (+6.6)
154.3 (=5.9)
241.2 (=6,5)
Bike Sharing
Standard
192.4 (18.9)
607.7 (+24.7)
157.4 (16.6)
157.4 (15.9)
244.3 (=6.6)
Weighted
173.2 (+-62)
802.9 (+30.)
142.8 (+5.0)
192.9 (+6.0)
289.1 (-6.1)
Superconduct
Standard
169.7 (+5.8)
813.1 (9.9)
142.3 (+5.0)
187.6 (+5.8
287.9 (66.1
4.1. WCTMs Adapt to Mild and Benign Shifts to Avoid Unnecessary Alarms
4.2. From Mild to Extreme Covariate Shifts: WCTMs Raise Alarms if Unable to Adapt
WCTMs Adapt to Benign Covariate Shifts Figure com- pares the average performance (across 200 random seeds) of WCTMs to standard CTMs across 3000 sequentially- observed test datapoints, where the true changepoint shift is induced after the 500th test point. From the conformal cov- erage plots in the first column, it is clear that all the shifts are benign in that coverage for the corresponding (standard or weighted) CP methods-a a metric of prediction safety does not degrade below the target level (0.9) after the change- point. In fact, for the baseline method (orange), coverage increases, which could be considered "beneficial" shift; nonetheless, the standard CTM raises unnecessary alarms across all datasets, for both its anytime-valid (fourth column) and scheduled (fifth column) monitoring criteria. In contrast, the proposed WCTM (blue) avoids these unnecessary alarms across all datasets and both monitoring criteria. That is, the WCTM adapts to the benign covariate shift by decreasing its interval widths (second column)-indicatiinn more in- formative predictions-witt while maintaining target coverage. The relatively uniform distribution of the postchangepoint weighted p-values (third column, blue) is empirical evidence validating Theorem 3.1; meanwhile, the martingale (fourth column) and Shiryaev-Robers (fifth column) WCTM paths avoiding alarms supports Theorems 3.2 and 3.3 respectively.
We demonstrate such a property of WCTM using image corruption experiments. Shift severity was controlled by increasing the proportion of corrupted samples in the test set relative to training and calibration. Figure 3 illustrates the behavior of WCTM in comparison to CTM on CIFAR-10 un- der varying levels of brightness corruption. When the target distribution is completely clean (a). neither method triggers false alarm. We then introduce level-1 brightness corrup- tion while retaining over 50% clean samples (b), creating very mild, benign shift. In this scenario, CTM quickly raises unnecessary alarm, whereas WCTM successfully avoids it. Next, we increase the corruption to level-3 and reduce the clean-sample ratio to 40% (c). Although WCTM reacts to this shift, it still does not trigger an alarm. Finally, at corruption level-5, with the clean-sample ratio reduced to 30% (d), WCTM does raise the alarm. See Appendix D.6 for further experiments on corrupted image data and D.2 for further analysis and explanation of what characterizes mild, moderate, or extreme covariate shifts.
4.3. WCTMs Detect Harmful Concept Shifts Faster than Sequentially Tracking Loss Metrics
Because concept shifts fundamentally change the Y|X rela- tionship, they are generally harmful, and monitoring meth-
8

---

WATCH: Adaptive Monitoring for Al Deployments via Weighted-Conformal Martingales
(Adapt Criterto))
X-CTM Adapt Threshold
WCTM (Alam Criterion)
Alamm Threshold
Changepaint
Benign/mild covariate shift
Harmfu//extreme covariate shift
Fundamental concept shift
Moan Martingai Values
10'
Moan Martingale Velues
10'
Martingate Values
10'
10a
10a
102
10a
10
10-1
10-1
0
500
1000
1500
2000
500
1000
1500
2000
0
500
1000
1500
2000
Test (Deplayment) Datapuint Index
Test (Deployment Datapoint Indext
Test (Deployment) Datapoint Index
Figure 4: Results for root-cause analysis with WCTM (blue) and a secondary X-CTM (gray) on the MEPS dataset, averaged over 200 random seeds. The WCTM and X-CTM together are able diagnose different types of shifts.
ods should detect them quickly possible. Our periments compare the average detection delay (ADD) of WCTM methods relative to comparable CTM methods, as well as against methods for sequentially tracking the loss metries directly with comparable false-alarm control. That is, the sequential testing procedures proposed in Podkopaev and Ramdas (2021b) (PR-ST) have anytime-valid control comparable to (W)CTMs, while the changepoint detection procedure in Podkopaev and Ramdas (2021b) (PR-CD) has weerage-un-lennghe control comparable to running the Shiryaev-Roberts procedure (15) on top of (W)CTMs. Both PR-ST and PR-CD methods can be used to monitor the miscoverage risks of either weighted CP or standard CP methods, though they are less data efficient (requiring an extra holdout set for computing concentration inequalities). We compared to the balting-based -process in Podkopaev and Ramdas (2021b) (with Gel 0) because those meth- ods were reported to perform the best and are more com- parable to (W)CTMs, which are also betting-basse based. We compared WCTMs to sequential testing variant with stan- dardized anytime-falss alarm rate of 0.01; for SR-WCTMs we we compared vs changepoint detection variant with common average-run-length (under the null) of 20,000.
such with the X-CTM achieving large value while the WCTM adapts to the shift, avoiding an alarm. The middle plot corresponds to an extreme covariate shift setting, and WATCH diagnoses it by the X-CTM identifying covariate shift and the WCTM raising an alarm due to potential harm. Lastly, the right plot corresponds to harmful concept shift, and WATCH identifies it as such by the WCTM raising alarm, but with the X-CTM maintaining lower values, suggesting that no covariate shift has occurred
5. Summary and Future Directions
In this paper we introduced the novel methods of weighted- conformal test martingales (WCTMs), which are con- structed from sequences of weighted-conformal p-values. We presented new theory for how online weighted- conformal p-values can be used in sequential hypothesis testing, and we demonstrated how WCTMs enable contin- ual monitoring of deplyed AI/ML models. The proposed approach, WATCH, achieves three main goals: (1) adap- tation to benign shifts avoid unnecessary alarms and improve the utility of CP prediction sets for end-users; (2) fast detection of harmful shifts; and (3) root-cause analysis to identify the cause of any performance degradation as a harmful covariate shift or a fundamental concept shift.
Table 1 reports the average detection delay (ADD) results for monitoring methods grouped by false-alarm control type, with proposed methods in blue. Among the anytime-valid monitoring methods, WCTMs and CTMs achieve compa- rably fast ADD; however, relative comparable PR-ST methods, (W)CTMs are over three times faster. Mean- while, among the stagewise monitoring methods, WCTM and CTM-based SR procedures are comparable, but with significantly faster ADD than PR-CD run either online or in minibatches. PR-CD-online has lower ADD than in mini- batches, but its wall-clock runtime is far slower (Appendix D.3) due to the method's (t²) time complexity, whereas the proposed WCTM and SR-WCTM methods are O()).
Our contribution opens up several promising directions for future work. Future theory directions include developing specific WCTM algorithms for testing other nonparamet- ric null hypotheses, exploring connections to conditional permutation tests (e.g.. Berrett et al. (2020)), analyzing WCTMs' efficiency with respect to certain alternative hy- potheses, developing robustness results for approximate weights, and Ener-grained root-cause analysis. For applica- tions, implementations of WATCH to monitor the risks of Al systems in real practical settings such as healthcare and ex- tensions to monitor foundation models and AI agents would be valuable. We hope that WATCH encourages greater at- tention on post-deployment monitoring for safer Al systems, with a focus on adaptive monitoring, fast detection of harm- ful shifts, and diagnostics to inform recovery.
4.4. WCTMs Diagnose Root Cause of Harmful Shifts
The left plot corresponds benign covariate shift setting (same setting as in Figure 2), and WATCH diagnoses it as

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Impact Statement
formal e-values. Advances in Neural Information Pro-
cessing Systems, 36:65585-65596.
This paper presents research aimed at propelling advance- ments in the broad domain of machine learning. The im- plications of our findings are wide-ranging, with potential high-stake applications in sectors including healthcare, tonomous driving, and e-commerce. Based on our current understanding, this research does not warrant an ethics re- view, and detailed discussion of the potential societal impacts is not required at the current stage.
Bates, S., Candès, E.,, Lei, L.,, Romano, Y., and Sesia, M. (2023). Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149-178.
Berrett, T. B., Wang, Y., Barber, R. F., and Samworth, R. J. (2020), The conditional permutation test for indepen- dence while controlling for confounders. Journal of the Royal Statistical Society Series B: Statistical Methodol- ogy, 82(1):175-197.
Acknowledgements
Cohen, J. W. Cohen, S. B., and Banthin, J. S. (2009). The medical expenditure panel survey: a national information resource to support healthcare cost re- search and inform policy and practice. Medical care, 47(7 Supplement 1):S44-550.
D.P. was funded by PhD fellowship from the Amazon Initiative for Interactive AI (AI2AI); D.P., X.H., A.L., and S.S. were partially funded by the Gordon and Betty Moore Foundation grant #12128; D.P., X.H., and S.S. were par- tially funded by National Science Foundation (NSF) grant #1840088. A.L. is also supported by an Amazon Research Award. The authors would like thank Yaniv Romano for helpful discussions
Eliades, C. and Papadopoulos, H. (2022). A betting function for addressing concept drift with conformal martingales. In Conformal and Probabilistic Prediction with Applica- tions, pages 219- PMLR.
draft of this paper; Jwala Dhamala and Jie Ding for helpful discussions throughout the JHU A12AI collaboration; and the anonymous ICML reviewers for informative feedback.
Eliades, C. and Papadopoulos, H. (2023). A conformal
martingales ensemble approach for addressing concept drift. In Conformal and Probabilistic Prediction with Applications, pages 328-346. PMLR.
References
Adams, R., Henry, K. F., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., et al. (2022). Prospective, multi-site study of patient outcomes after implementation of the trews machine learning-based early warning system for sepsis. Nature medicine, 28(7): 1455-1460.
Fanaee-T, H. (2013). Bike Sharing. UCI Machine Learning Repository. DOI: http:://oi..ng//0223333////
Fannjiang, C., Bates, S., Angelopoulos, A. N., Listgarten, J., and Jordan, M. I. (2022). Conformal prediction under feedback covariate shift for biomolecular de- sign. Proceedings of the National Academy of Sciences, 1 119443))e204566119
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Mané, D. (2016). Concrete problems in ai safety. arXiv preprint arXiv: 1e0..66666.6..
Farinhas, A., Zerva, C., Ulmer, D., and Martins, A. F. (2023). Non-exchangeable conformal risk control. arXiv preprint arXhv:2310.0126..
Angelopoulos, A. N., Barber, R. F. and Bates, S. (2024). Theoretical foundations of conformal prediction. arXiv preprint arXiv:241 1824.
Fedorova, V., Gammerman, A., Nouretdinov, I., and Vovk, V. (2012). Plug-in in martingales for testing exchangeability on-line. arXiv preprint arXiv: /244.225..
Bar, Y., Shaer, S., and Romano, (2024). Protected test- time adaptation via online entropy matching: A betting approach. arXiv preprint arXiv:2408.07511
Feldman, S. and Romano, (2024). Robust conformal prediction using privileged information. arXiv preprint arXiv:2406.05400
Barber, R. F., Candes, E. J., Ramdas, A., and Tibshirani, J. (2023). Conformal prediction beyond exchangeability. The Annals of Statistics, 5 51(2):816-845.
Feng. J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, E., Celi, L. A., and Pirracchio, R. (2022). Clinical arti- ficial intelligence quality improvement: towards continual monitoring and updating of ai algorithms in healthcare.
Barber, R. F. and Tibshirani, R. J. (2025). Unifying dif- ferent theories of conformal prediction. arXiv preprint arXiv:2504.02292.
NPJ digital medicine, 5(1):66.
Bashari, M., Epstein, A., Romano, Y., and Sesia, M. (2023). Derandomized novelty detection with fdr control via con-
Feng, J., Xia, F., Singh, K., and Pirracchio, R. (2025). Not all clinical ai monitoring systems are created
10

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
equal: Review and recommendations. 2(2):Ala2400657.
NEJM AI,
Lee, Y. and Ren, Z. (2025). Selection from hierar- chical data with conformal e-values. arXiv preprint arXiv:250.00214.
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., and Saria, S. (2021). The clinician and dataset shift in artificial intelligence. New England Journal of Medicine, 385(3):283-286
Mu, N. and Gilmer, J. (2019). Mnist-c: A robust- ness benchmark for computer vision. arXiv preprint arXiv: 1etei9996..237
Fischer, L. and Ramdas, A. (2025). Sequential monte carlo testing by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, page qkaf014.
Nair, Y. and Janson, L. (2023). Randomization tests for adap- tively collected data. arXiv preprint arXiv:2301
Page, E. S. (1954). Continuous inspection schemes.
Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshi- rani, R. J. (2021). The limits of distribution-feed condi- tional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-433
Biometrika, 41(1/2):100-155
Papadopoulos, H. (2008). Inductive conformal prediction: Theory and application to neural networks. In Tools in artificial intelligence. Citeseer.
Gauthier, E., Bach, F., and Jordan, M. I. (2025). E-values expand the scope of conformal prediction. arXiv preprint arXiv:2503. 13050.
Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, A. (2002). Inductive confidence machines for regression. Machine learning: ECML 2002: 13th European con- ference on machine learning Helsinki, Finland, August 19-23, 2002 proceedings 13, pages 345-356 Springer.
Hamidich, K. (2018).
Superconductivty Data. UCI Machine Learning Repository. DOI:
https//cbi.ogg//.44422cc5..
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Ma- chine learning in python. the Journal of machine Learn- ing research, 12:2825-2833
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Hendryeks, D. and Dietterich, T. (2019). Benchmarking neural network robustmess to common corruptions and perturbations. In International Conference on Learning Representations.
Podkopaev, A. and Ramdas, A. (2021a). Distribution-Fie uncertainty quantification for classification under label shift. In Uncertainty in artificial intelligence, pages 844- 853. PMLR.
Hindy, A., Luo, R., Banerjee, S., Kuck, J., Schmerling, E., and Pavone, M. (2024). Diagnostic runtime monitoring with martingales. arXiv preprint arXiv:2407.21748.
Podkopaev, A. and Ramdas, A. (2021b). Tracking the risk of deployed model and detecting harmful distribution shifts. arXiv preprint arXiv: 10.06177.
Ho, S.-S. (2005). martingale framework for concept change detection in time-varying data streams. In Pro- ceedings of the 22nd international conference on Machine learning, pages 321-327.
Podkopaev, A. and Ramdas, A. (2023). Sequential predic- tive two-sample and independence testing. Advances in neural information processing systems, 36:53275-53307.
Prinster, D., Liu, A., and Saria, S. (2022). Jaws: Auditing predictive uncertainty under covariate shift. Advances Neural Information Processing Systems, 35:35907- 35920.
Ho, S.-S., Schofield, M., Sun, B., Snouffer, J., and Kirschner, (2019). martingale- based approach for flight behavior anomaly detection. In 2019 20th IEEE International Conference Mobile Data Management (MDM), pages 43-52. IEEE
Prinster, D., Saria, S., and Liu, A. (2023). Jaws-x: address- ing efficiency bottlenecks of conformal prediction under standard and feedback covariate shift. In International Conference on Machine Learning, pages 28167-28190 PMLR.
I Amoukou, S., Bewley, T., Mishra, S., Lecue, F., Maga- zzeni, D., and Veloso, M. (2024). Sequential harmful shift detection without labels, Advances Neural Information Processing Systems, 37:129279-129300
Prinster, D., Stanton, S. D., Liu, A., and Saria, S. (2024). Conformal validity guarantees exist for any data distri- bution (and how to find them). Forty-first International Conference on Machine Learning.
Jin, Y. and Candès, E. J. (2023). Model-free selective in- ference under covariate shift via weighted conformal p- values, arXiv preprint arXiv:2307.09229

---

WATCH: Adaptive Monitoring for Al Deployments via Weighted-Conformal Martingales
Ramdas, A., Grünwald, P., Vovk, V., and Shafer, G. (2023). Game-theoretic statisties and safe anytime-valid infer- ence. Statistical Science, 3844).57--601
Shiryaev, A. N. (2016). Probability-I, volume 95. Springer.
Stanton, S., Maddox, W., and Wilson, A. G. (2023),
Bayesian optimization with conformal prediction sets. In International Conference on Artificial Intelligence and Statistics, pages 959-986. PMLR.
Ramdas, A., Ruf, J., Larsson, M., and Koolen, W. M. (2022). Testing exchangeability: Fork-convexity, supermartin- gales and e-processes. International Journal of Approxi- mate Reasoning, 141:83-109.
Sugiyama, M.. Krauledat, M., and Müller, K.-R. (2007).
Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5).
Roberts, S. (1966). A comparison of some control chart procedures. Technometrics, 8(3):411-433
Tartakovsky, A., Nikiforov, I., and Basseville, M. (2014),
Romano, Y., Sesia, M., and Candes, E. (2020). Classifica- tion with valid and adaptive coverage, Advances in neural information processing systems, 33:35.1-3591.
Sequential analysis: Hypothesis testing and changepoint detection. CRC press.
Teneggi, J. and Sulam, J. (2024). Testing semantic im- portance via betting. Advances in Neural Information Processing Systems, 37:76450-76499.
Saha, A. and Ramdas, A. (2024). Testing exchangeability by pairwise betting. In International Conference on Artificial Intelligence and Statistics, pages 4915-4923. PMLR.
Tibshirani, J.J., Foygel Barber, R., Candes, E., and Ramdas, A. (2019). Conformal prediction under covariate shift. Advances in neural information processing systems, 32.
Shaer, S., Maman, G., and Romano, Y. (2023). Model-x sequential testing for conditional independence via testing by betting. In International Conference on Artificial Intelligence and Statistics, pages 2054-2086. PMLR.
Veeravalli, V. V. and Banerjee, T. (2014). Quickest change detection. In Academic press library in signal processing, volume 3, pages 209-255. Elsevier.
Shafer, G. (2021). Testing by betting: A strategy for sta- tistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society,
Ville, J. (1939). Etude critique de la notion de collectif. Gauthier- Shithier-Villars Paris.
184(2):407-431
Shafer, G. and Vovk, V. (2019). Game-theoretic fame-thhoreticcouuuttion foundations for probability and finance, volume 455. John Wiley & Sons.
Volkhonskiy, D.,, Burnaev, E.,, Nouretdinov, I., Gammerman, A., and Vovk, V. (2017). Inductive conformal martingales for change-point detection. In Conformal and Probabilis- tic Prediction and Applications, pages 132-153. PMLR.
Shekhar, S. and Ramdas, A. (2023a). Nonparametric two- sample testing by betting. IEEE Transactions on Infor- mation Theory, (002))1178-122)
Vovk, V. (2002). On-line confidence machines are well- calibrated. In The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings., pages 187-196. IEEE.
Shekhar, S. and Ramdas, A. (2023b). Reducing sequential change detection to sequential estimation. arXiv preprint arXiv:2309.099 09111.
Vovk, V. (2003). Well-calibrated predictions from on- line compression models. In Algorithmic Learning The- ory: 14th International Conference, ALT 2003, Supporo, Japan, October 17-19, 2003. Proceedings 14, pages 268- 282. Springer.
Shekhar, S. and Ramdas, A. (2023c). Sequential change- point detection via backward confidence sequences. In International Conference on Machine Learning, pages 30908-3000 309300 PMLR.
Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244.
Vovk, V. (2020). Testing for concept shift online. arXiv preprint arXiv:2012.14246.
Vovk, V. (2021). Testing randomness online. Statistical Science, 36(4):595-611
Shin, J.. Ramdas, A., and Rinaldo, A. (2022). E-detectors: A nonparametric framework for sequential change detection. arXiv preprint arXiv:2003.03532.
Vovk, V. (2023). The power of forgetting in statistical hypothesis testing. In Conformal and Probabilistic Pre- diction with Applications, pages 347-366. PMLR.
Shiryaev, A. N. (1963). On optimum methods in quickest detection problems. Theory of Probability & Its Applica- tions, 8(1):22 46.
Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorith- mic learning a random world, volume 29, Springer.
12

---

WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformee Martingales
Vovk, V., Gammerman, A., and Shafer, G. (2022). Algorith- mic Learning in Random World. Springer Nature.
Vovk, V., Nouretdinov, I., and Gammerman, A. (2003). Test- ing exchangeability on-line. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 768-775.
Vovk, V., Petej, I., Nouretdinov, L,, Ahlberg, E., Carlsson, L., and Gammerman, A. (2021). Retrain or not retrain: Conformal test martingales for Change-point detection. In Conformal and Probabilistic Prediction and Applications, pages 191-210. PMLR.
Vovk, V. and Wang, R. (2021). E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736-1754.
Vovk, V. and Wang, R. (2023). Confidence and discoveries with e-values. Statistical Science, 38(2):329-354.
Wald, A. (1945). Sequential tests of statistical hypotheses. In Breakthroughs in statistics: Foundations and basic theory, pages 117-186. Springer.
Wasserman, L., Ramdas, A., and Balakrishnan, S. (2020). Universal inference. Proceedings of the National Academy of Sciences, 11729))66880-16890
Waudby-Smith and Ramdas, A. (2024). Estimating means of bounded random variables by betting. Jour- nal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1-27.
Xie, L., Zou, S., Xie, Y., and Veeravalli, V. V. (2021). Se- quential (quickest) change detection: Classical results and new directions. IEEE Journal on Selected Areas in Information Theory, 2(2):494-514.
Xu, C. and Xie. (2021). Conformal prediction interval for dynamic time-series. In International Conference Machine Learning, pages 11559-11569, PMLR.
Yang. Y., Kuchibhotla, A. K., and Tchetgen Tchetgen, E. (2024). Doubly robust calibration of prediction sets under covariate shift. Journal of the Royal Statistical Society Series B: Statistical Methodology, page qkae009.
Zhang, Y.-.., Zhang, Z.-Y., Zhao, P., and Sugiyama, M. (2024). Adapting to continuous covariate shift via online density ratio estimation. Advances in Neural Information Processing Systems, 36.
13

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Appendix for "WATCH: Adaptive Monitoring for AI Deployments via Seighted-Conforme Martingales"
A. Related Works
This work is motivated by developing methods for monitoring AI/ML deployments that perform three key functions: (I) online adaptation to mild or benign data shifts; (2) rapid detection of extreme harmful shifts that necessitate updates; and (3) identifying the root-cause of degradation to inform appropriate recovery. Although these monitoring goals could be viewed from many perspectives, in discussing related work we primarily focus on methods in anytime-valid inference, sequential testing of nonparametric null hypotheses, and especially conformal test martingales, which most closely related to our own.
Sequential testing for changepoints in the data distribution: Sequential hypothesis testing to detect changes in the data distribution is an old and widely-studied problem, dating at least to Wald's sequential probability ratio test (Wald, 1945). However, classic sequential changepoint detection methods often required a prespecified stopping time and were primarily designed for testing simple, often parametriically-speciii null hypotheses-foo for an overview of classic parametric methods, we defer to a relevant textbook Tartakovsky et al. (2014) and review articles (Veeravalli and Banerjee, 2014; Xie et al., 2021), In contrast, our weighted-confonee anytime-valid inference (SAVI,, which allow for arbitrary stopping times, and specifically our work is situated in the recent SAVI literature on testing composite and nonparametric null hypotheses. We refer readers to Ramdas et al. (2023) for a recent review of the SAVI literature, as well as the textbook by Shafer and Vovk (2019) on the closely related topic of testing-by-hetting
test martingale (WCTM) methods belong to a more recent literature called sequential
Standard conformal test martingales: Within the SAVI literature, our work is most closely related to conformal test martingales (CTMs), which are martingales constructed from a sequence of standard conformal p-values for continually testing the assumption that the data are exchangeability or independent and identically distributed (IID). Our WCTM methods generalize standard CTMs for testing a broader range of null hypotheses, including those where one wishes to accommodate or adapt to certain anticipated changes the data distribution (e.g. adapting to mild covariate shifts). Standard CTMs were initially introduced by Vovk al. (2003), while drawing on theory for the calibration of online conformal prediction methods developed in Vovk (2002). Since then, various works have further developed standard CTMs, such by introducing new betting and score functions for more efficient changepoint detection, ensembling CTMs, and demonstrating their performance on real-world applications (Ho, 2005; Fedorova et al., 2012; Volkhonskiy et al., 2017; Ho et al., 2019; Vovk et al., 2021; Vovk, 2021; Eliades and Papadopoulos, 2022; 2023). CTMs are also discussed in textbooks on conformal prediction (Vovk et al., 2005; Angelopoulos et al., 2024). The online form of conformal prediction that CTMs are based on is one form of online compression model, as defined and discussed in Vovk (2003); Vovk et al. (2005); Vovk (2023).
In particular, Volkhonskiy al. (2017) developed inductive CTMs (i.e., CTMs based on inductive or split conformal (Papadopoulos et al., 2002; Papadopoulos, 2008)) with score and betting functions specifically taylored to fast changepoint detection; Vovk et al. (2021) proposed an approach to using standard CTMs to monitor for when a deployed AI/MI system should be retrained; and Vovk (2021) provided a comprehensive and detailed review of CTMS as methods for testing the IID or exchangeability assumptions. Bar et al. (2024) implement methods based on CTMs for test time-adaptation of a classifier's point prediction and Hindy et al. (2024) leverage multiple CTMs over different feature spaces to aid in diagnostic runtime monitoring. Several works have developed or used CTMs for testing for concept shift (i.e., shift in Y X) (Ho, 2005; Eliades and Papadopoulos, 2022: 2023; Vovk, 2020), but all of these have focused on a classification setting with limited number of classes (e.g., to implement label-conditional CTMs, or using standard CTMs that lack ability to disambiguate between covariate and concept shifts). In contrast, our specific WCTMs implemented in this paper are able to test for concept shift in both regression and classification settings while also being able to disambiguate between concept shifts and extreme covariate shifts with the aid of an additional X-CTM.
Confidence sequences, e-processes, and other multiple-hypothesis testing methods: to other SAVI and lesting-by-betting methods outside of CTMs such as confidence sequences and e-processes- we refer to the review paper of Ramdas al. (2023) for full exposition of these topics. In particular, in the main paper we empirically compared our WCTM methods against the betting- based sequential testing and changepoint detection methods in Podkopaev and Ramdas (2021b) (the theory for which was developed in Waudby-Smith and Ramdas (2024)) because this paper is the
Our work also relates more broadly
14

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
closest ours in its motivation of monitoring deployed AI/L deployments among (non-CTM) SAVI papers. As we note in the main paper, for monitoring the risk of a set-valued (e.g., conformalized) AI/M predictor, our WCTM methods generally more data-efficient (not requiring separate datasets for conformalization and testing, as Podkopaev and Ramdas (2021b) does): our WCTM-based Shiryaev-Roberts procedure is more computationally efficient than the compareable changepoint detection method in Podkopaev and Ramdas (2021b) (i.e., (1)) versus ((²)); and, empirically we found that our methods often detect harmful concept shifts faster than comparable methods in Podkopaev and Ramdas (2021b). More recently, Amoukou et al. (2024) built on Podkopaev and Ramdas (2021b) by developing similar methods for when ground-truti labels do not become available and need be estimated.
A key target of the SAVI literature testing composite or nonparametric null hypotheses has been developing new approaches for testing the IID or exchangeability assumptions. Other than CTMs, the other main nontrivial approach to sequentially testing exchangeability was developed in Ramdas et al. (2022) based on the ideas of universal inference (Wasserman et al., 2020); more recently, other approaches have emerged based on pairwise betting (Saha and Ramdas, 2024) and sequential Monte Carlo testing (Fischer and Ramdas, 2025), the latter of which can be viewed as a special case of CTMs (Vovk, 2021) streamlined for a particular alternative hypothesis. Otherwise, there are various and proliferating other methods for sequential nonparametric changepoint detection (e.g., Shin al. (2022); Shekhar and Ramdas (2023a;b;c;; Podkopaev and Ramdas (2023)). SAVI and testing-by by betting methods are also being leveraged for wide variety of applications including interpretability (Teneggi and Sulam, 2024), conditional independence testing (Shaer et al., 2023), applications in finance (Shafer and Vovk, 2019), and more.
serting-by-bettingg which is fundamental to SAVI (Ramdas et al., 2023), can be understood as one approach to multiple hypothesis testing that is especially advantagcous in sequential settings, Other related works that leverage conformal prediction for hypothesis testing while accounting for multiple-testing corrections, but in batch settings, include Bates et al. (2023); Bashari al. (2023); Vovk and Wang (2023); Gauthier al. (2025); Lee and Ren (2025).
Weighted conformal prediction and adapting to distribution shifts: Weighted conformal prediction (c.g.. Tibshirani et al. (2019); Podkopaev and Ramdas (2021a); Xu and Xie (2021); Fannjiang et al. (2022); Prinster et al. (2022; 2023); Stanton et al. (2023); Barber et al. (2023); Farinhas et al. (2023); Nair and Janson (2023); Yang et al. (2024); Feldman and Romano (2024); Barber and Tibshirani (2025)) is broadly an approach to proactively adapting the validity of conformal predictive sets to distribution shift by reweighting nonconformity scores using either knowledge or estimates of the distribution shift. Any weighted CP prediction set associated with a weighted-conformat p- valuues, as described in the main paper; accordingly. a WCTM can be constructed top of any weighted CP method deployed on a sequence of data observations to continually monitor the assumptions approximations underlying that WCP method's implementation. There are of course many other approaches to adapting to distribution shifts at test time; for example, one work that is similar to ours with regard to this motivation, but that does not fall under weighted CP, is Bar et al. (2024), which leverages standard CTMs to guide the test-time adaptation of a classifier's point prediction by entropy matching.
B. Proof of Theorems
B.1. Proof for Theorem 3.1 (Weighted Conformal p-value Validity)
The proof can be viewed generalization of the proofs for Theorem in Vovk et al. (2003) and Theorem 2 in Vovk (2002), while drawing on analysis from Tibshirani et al. (2019) and exposition and discussion from Prinster et al. (2024). The key difference relative to Vovk (2002) and Vovk et al. (2003) is that, whereas those papers use the assumption of exchangeability to place equal weight on every permutation of the data observations, here we avoid this assumption by first proving a general result for an arbitrary (potentially non-exchangeable) joint distribution. We then describe how this implies that the validity of more specific and tractable methods is premised on the assumptions or approximations used for practical implementation.
The basic idea for the proof begins with setup from Vovk (2002), for "reversing time." In particular, we imagine that the sequence of data observations (21, (21, 2r) is generated in two steps: first, the unordered bag or multiset of data observations, {zi,..., ). is generated from some probability distribution (that is, the image of Fz under the mapping (#1, .2.... + {2I,..., 2r h); then-here generalizing Vovk (2002) and Vovk et al. (2003) by weighting permutations according to their lkelihood-frrm all possible permutations a of the values {z1, }. J. each possible sequence (Zo(1), ...,, > is chosen with probability proportional to (2)(1): (T) where f := f2 is the probability-dens
function" for the distribution
Fz. Roughly (ignoring borderline effects), the second step ensures that, conditionally on knowing (21,.., 27) (and therefore
Moore generally, the Eadon-Nikodie derivative.
15

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted- Martingales
unconditionally), that Pft has standard uniform distribution; when ZT (T( is observed, this settles the value of PT -Po and conditionally on knowing {z1,..., Er} and Zr = ² (and therefore, after relabeling indices, on knowing {Z1,..., 2T-1)), that P also has standard uniform distribution, and so on.
Lemma B.1. for any trial and any confidence level (0,1), P(P:" Sa E a.
(18)
Proof of Lemma B.I. We begin by conditioning the event {Z1.... Ze} = (21,...,.0 21), which denote as E and we consider drawing any particular ordering or permutation of the data values with probability according to f. that is with probability proportional to (1) .... (o))
Note that for any E [d]. if a permutation is drawn such that (tt) = i, this means that Z, = 2x(1) = 2;; moreover, because score function S bijective, this further implies that V, = vo(()) = uy. Thus, given the bag data E! recall that for each [f), the probability of drawing such permutation given by the "oracle weights"
f(E()) Za(t))
T. == P(V = | P = |
(19)
Eo (Eo((I)..... kr(())
which we assume to be well-defined, which will generally be the case in practice, where at least f(1 2T) > 0 for the true (identity permutation) ordering of the data observations (z1,..., 2T).
This implies that the distribution of Vi Et() the conditional distribution of the test-poin score given the bag of data values E!! is given by
Vd
E
W 8og
For any i E [t], define a conservative WCP p-value, P.
and an anticonservative WCP p-value, P. p."" as
P
03
1(1 vi)
j=1
Σ o6 160 > vn).
j=1
It is worth noting that Pt + is valid p-vuuue for f: P(P. a E sa =
P(P. S a) S a. However, the
lemma claims exact validity for pe conditional on (!()) which we will now proceed to show.
Observe that for all [4], p. _ < P + + and
p
63 1[0; v.).
Moreover, observe that as in the proof for Lemma in Vovk (2002), the semi-closed intervals 1,,"" .P; +) either coincide
are disjoint, and Up!""""" .P. 2255+) [o, 1).
Similarly as in the proof for Lemma in Vovk (2002), for E (0, 1), let us partition the indices as follows, where we say that an index is
"strange" if 1, - < a,
> a. "ordinary" if p _--
and "borderline" if 66- < a < P 3+
16

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Let i' denote the index of any borderline example, and denote p0*+ : Pe + and p" = Pp 00 Then, the probability
(conditional on E drawing each permutation with probabilities according to f) that the last index o is strange
is yda : the probability that (4) is ordinary is 1 - p and, the probability that (()) is borderline is pd + p Moreover, observe that if (t) is strange, then
1 p Pr(() I a (by definition) and if (t) is borderline then the event that Po(t) S a
determined by the independent uniform ue, and thus the probability of this event is
That is,
E P(P Sa| E = P(P Sa E (t) is strange} P(a((t) is strange
+ P(Pp Sa E (() is ordinary) P(a(() is ordinary | E + p(pp sal F (1) is borderline) (1) is borderline F
p pm= + +++ (p"*+ pm -)²-)) a - -000
a-p00-
pun+t p
a.
With Lemma 1 in hand, proceed with the proof for the main theorem. Temporarily fix positive integer T; following the strategy of Vovk et al. (2003), we will first prove by induction that for any / = I,..., T and any o1, Og E 1j, that
P(PP" S City prd S Sai [ = O't .....
(20)
For = Eq. (20) immediately follows from Lemma 1. For t > 1, by the law of total probability over (tt) (i.e., over the last index value after drawing permutation a from E the fundamental bridge between probability and expectation, and properties of the indicator function, we have
P(P. I aat....
PUP
Sa }
Σ P(Ppe Pr. Sai E Ze = =0(1)) = za(1) E
=(1=1
Σ E[1[Pre S a..... PiP sai E,D,, Z. =0(1)]] -PZz Za(tt) E
ott==
Σ
E[ <ar 1(P S sa-1,... Pp" S <n E!!) Z. zo10 PfZ 2a(1) E
-()=1
Next, observe that conditioning on E and Z, = (o) in the expectation term settles the value of 1 (Pte" S ar) to be (p S ac): that is, noting that the score function S is bijective, (E!)) Z 2o(1)} = {E!", V₁ vo(0) =
Pee Poty So, 1(P
aa²) 1 < { can be pulled out from the expectation to obtain
P(pi S Sa,,.., Pre" Sai }
Σ 1P S ar) [[ S a....... Pre S sail ELO, Z, = 20(0] PfZ. = 2a(1) E
a(t)=l
Now, observe that conditioning on F and 2, = Zo(() implies that (,,,.., 2 = (.I,..,,, That is,
observing the bag of t data values (i.e., observing E!!) and the value taken on by the t-th random variable (Z, = 2o(1))
implies that each of 7, ..,, Zt- takes value in {21, ..., 2,3 2 (2)
17

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Without loss of generality, we can relabel the indices the data values as {Z1, 1 <- {-1, ze) (2)())) and so
1) The WCP p-variable PT is determined by drawing sequence For(1); ......................... (t-1 (i.e., determined by permutation o: [t- - [1- from the bag of data values (=1.-.,, =t } ...,
denote this event {Z1. .... Z{-1} {z1, .... 1 } as _ (((1))
with probability proportional to f......22_1 (1): ... (t-(t))) and then applying the the nonconformity score function S, and so F is a "sufficient statistic" for pi= (for a known (2.....Z.-1)) ). In other words, we can substitute the conditioning on the event F and Z = (t) with conditioning on F
P(PPt" I Ci.... pi S a1 }
Σ He S a+) PiPpi S at-b ... P Sai | PPZz za(1) |
[t)=1
Using the inductive assumption and Lemma B.1, this becomes
P(P."" Sa,,... Pi" S an E =
1P:0() S sar) 10-100011.0 .(Zz = 20(1)) {
o(t==
= PPPp S a, { a0-1
= .11
which proves Eq. (20). Note that Eq. (20) is conditional result for any I,... T: marginalizing over the event ES? and taking& ( = T implies
P(P S Ser..... P S <a = RT
(21)
We have proven that PT P2 ... pa lid Unif[O, 1J""; this implies the analogous result for the infinite sequence, that is, vid pre , P. .... ~ Unif[o, o (Shiryaev, 2016; Vovk et al., 2003).
Note that Eq. (21) holds in theory for an arbitrary joint PDF f. but it is an abstract statement because it relies on the oracle weights in Eq. (19), which are intractable due to requiring knowledge of f and factorial complexity. Thus, if the oracle weights in Eq. (19) are simplified using some assumptions or approximations on f (e.g., conditional independence or invariance assumptions, density-ratio estimations) that we denote by Ho(f), and denoting the resulting approximated weights as w. then the p. are IID uniformly distributed, assuming Ho(f):
P Pnolh {Pt S aT,,.., ... P S <a1} ar..aa.
(22)
Eq. (22) implies that PIù, P..... P4 Ho(f) Hd Unif 1|". Similarly as before, this result for fixed number of points T' implies the corresponding result for infinite sequences (Shiryaev, 2016; Vovk et al., 2003), that is, p ,i.... Ho(s) iid Unif 1
B.2. Proofs for Proposition 3.2 (WCTM Anytime False-Alarm Control) and Proposition 3.3 (Average-Run-Length Control for WCTM-based Shiryaev-Roberts Procedure)
The proofs for Proposition 3.2 and Proposition 3.3 follow from Theorem 3.1 and Vovk (2021)'s results for standard conformal test martingales. By Theorem 3.1, weighted conformal p-values have IID uniform distribution [0.1] premised on some assumptions Hf) Thus, sequence of weighted conformal p-values is sequence of III) random variables drawn from 1n] assuming that the null hypothesis Hf( is true. Because the construction of weighted conformal test martingale (e.g., Eq. (7)) is only allowed to depend on the data via the sequence of weighted conformal p-values, we can draw from Vovk (2021)'s results that construct betting martingales that only assume a sequence of IID uniform [0.1] random variables. That is, by construction, weighted conformal test martingales are a betting martingale (Vovk, 2021) whose validity is premised on the assumptions (f) (used to compute the weights). Then, Proposition 3.2 follows by Ville's 's inequality (Ville, 1939) and Proposition 3.3 follows from Proposition 4.2 in Vovk (2021).
18

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
C. Deriving Specifie WCTM Algorithms for Main Practical Testing Objective: Testing for {Concept Shift or Unanticipated Covariate Shift}
In this section we describe the derivation for the specific WCTM algorithms implemented in this paper for our primary testing objective, of detecting either concept shift (in Y X) or unanticipated covariate shift (extreme shift in X). Because WCTM is constructed on top of a corresponding weighted CP method by a betting process on the associated sequence of weighted CP p-values, this will initially parallel the procedure described Prinster et al. (2024) for deriving weighted CP methods. The resulting weighted CP method (that the WCTM algorthims are built on top of) will be similar to that used in Tibshirami al. (2019), except whereas that paper focuses on CP for shift between two batches of data, we focus on an online monitoring setting where shift may occur at unknown post-deployment time and density-ratio weights may be estimated online.
Step 1: List assumptions/ull hypothesis of interest
We begin by restating the null hypothesis (i.e., the nonparametric assumptions on what aspects of the data distribution are expected to stay invariant versus change) that is of primary practical interest in this paper. That is, as in Eq. (13) in the main paper, our null hypothesis P6fss) assumes that the conditional label distribution Y X remains invariant and that the marginal X distribution can shift, but that at some post-deployment time lad (the time at which to begin adaptation to covariate shift), the covariates X shift in distribution in a manner described by the (estimated) density-ratio function ((x).
That is, we want to sequentially test the null hypothesis
F FYxx Fx, i In + Lad - 1]
16000
(23)
(Xnte; Yntc) ~ FYix (())
2 tad-
where boldface is used to denote a set of distributions, and in particular Gx denotes the set of distributions for X with density-ratio function ((x) (with respect to the true but unknown source covariate distribution Fx E Fx):
= =(xx (z) ((x) if dFx(x)>00 =0]
x
The density-ratio function ((x) can be estimated with offline data and/or continually updated each test point is observed.
Step 2: Factorize joint density using nall hypothesis assumptions
We now use null hypothesis assumptions H(ss) to factorize the joint probability density function. See Prinster et al. (2024)'s Appendix B.1 for a more detailed derivation for a more general setting.
n+t
((21,,,2244) fx(x1) fx (x 9x (x(tee))): ... (xx+)))) II
(f)()))))))
n+w--1
n+t
fx((,)
II j=ntla
[ex(2))
II [fr)(())))))) ==1
I fx(x)) 15 fx()))
multiplying by 1
we have
nt
n+f
gx(x,)) fx(2,)
n
|21 knte)
[x(2;)]
II
II
Sy [fri(()l=)))
and approximating the density-ratio function
(x,)
with our estimate ((Ix), this becomes (xy)
19

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
n+t II j=n+fe
a+t
(041))
((3))
II Frix(())))) fx (17))
(12))
Il f2(3))
(24)
Time deneedeenddnn factors
Time invariant factur
Step 3: Compute or estimate weights and weighted-conformal p-values
Recalling the definition of the CP weights from the main paper (Fqq. (10)) and plugging in the factorization from Eq. (24):
Pl = [ Eoro(ntt)=: (2 Zo(4t))
E. (2o(1) Fo(n+()
Ensa(nt)i=i j-ntle ((Fo())
I fz((0())
E. [lsmate ((Fo(t)
I[ (z(Fo())
Lose(n+t)== I n+e f=ntte m(Fo())
(25)
Eg Tl ((Fa())
where the last line follows because the factor I15=1 (z((S) is invariant to permutations and thus cancels in the ratio.
However, the computational complexity for computing Eq. (25) is still computations), to enable tractabile estimation, we focus only permuting the test point value with values assumed be from the source distribution (prior to the adaptation beginning); that is, we the sums to permutations o' ; In - [n t such that o' d'(j)= = j for all j E {n + tud ... n + t- 1}:
choose lnd (i.e., requiring (x-(ntt-
(n++-))
Ear ((F0(()))))
P(Zn+t = 2z [b( w.!0
En ((Fr((+1))
@(x:)
(26)
E (eefntts-llu(nee ((x3))
Then, the corresponding weighted-coofoormal p-vulee is
Putt
Σ
ai(()
1 > untel + un-el vntr]]
(27)
ve(ntH (1]((++)))
Due to the modification made in Eq. (26), Theorem 3.1 only directly applies for the covariate shift null hypothesis (i.e., for testing H)( H((2) in Eq. (23)) for the timepoint t fus- This is due to that, when weighted weighted-conformal p-value is approach in Section B.1 may not apply; in other words, for two test
not computed fully online," " the proof-by-induche points n - t and n + f', their p-values p and po+ may not be statistically independent due to their common dependency on the calibration data [n + lnd - 1] in Eq. (27). This gap can be resolved to enable testing of Eq. (23) if the calibration data [n + lad - 1] are resampled for each test point, which restores independence trivially (due to each calibration set being entirely new). Alternatively, we can modify Eq. (23) to also assume that we have enough initial source data (i.e., n is large enough) so that the calibration scores' empirical CDF is equal to the true source score distribution, that is, F p
Ev.
That is, not computed relative all the full observed data sequence with indices {n. + t), but only relative strict subset of [n + f]; for example, Eq. (27) computes the p-value relative to the data with indices [n + tad 1] Ü {n + t). where fad < f.
20

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
which is of course true in the limit - oc. Accordingly, the p-value in Eq. (27) is effectively testing the modified null hypothesis of H6)) and platta-ll_ Fv:
(X,,
sdd Fxix Ex,
F uid
i [n tad 1]
H A (F Fv)
(Kntes Yn+t) pluttas11 Pv.
Fxix G
2 lnd
(28)
In practice, Eq. (28) roughly be thought of as also testing whether the weighted CP calibration set "large enough" for CP coverage to hold precisely conditional on the calibration data. That is, letting Zeal denote the event that { (71,-.. Zn+ta-1) = (zi,.... tn 1, taking Eq. (28) as given would imply that, for all a E (0,1).
P(Pate S Zeal) =a
P(Yn+* E
(Xn+e) Zeal = a,
(29)
where Patt is the random variable taking values in Eq. (23) and the probability is only over the draw of the test point Zn++- In other words, Eq. (28) = Eq. (29), so via the contrapositive, violations of calibration--st-cconiiioiii coverage (Eq. (29)) would imply violations of 76000 A (
Fv) (Eq. (28)).
Step 4: Construct WCTM by betting process on the weighted p-values
Lastly, the practical WCTM algorithms implemented in this paper are constructed on top of a sequence of weighted- conformal p-values Ph+1 Pw+2: Pa-ts
..., as defined in Eq. (27). The resulting WCTM thus achieves the anytime-valid false alarm guarantee in Proposition 3.2 for the null hypothesis Ho((') = H662) A (FV (Fintts-1) = Fv). defined in Eq. (28)
(and analogously for the scheduled, Shiryaev-Roberts monitoring procedure based on WCTMs).
D. Additional Experimental Results
D.1. Root-Cause Analysis with WCTMs
D.2. What Makes a Covariate Shift Mild, Moderate, or Extreme?
While the distinction between mild, moderate, and extreme covariate shifts can be gradual, problem-specific, and sometimes user-driven, these disctinctions are not arbitrary, and WCTMs arguably even provide an approach to such delineation with statistical guarantees. Intuitively, benign shifts can be considered those where the "safety" of the CP coverage maintained nontrivially (i.e., without the prediction set covering the whole label space). This intuition corresponds to the WCTMs' null hypothesis (and, harmful shifts violate it), follows:
The betting martingale's 'ss null hypothesis is that the WCP p-values are IID Unif(0, 1] (Appendix B.2); this null implies that coverage is satisfied (exactly) for all a E (0, 1) (ie, the martingale's 's null = intuitive definition of "benign" regarding coverage validity).
Benign:
Harmful:
(Contrapositive of the above.) If coverage is not satisfied (exactly) for some (0, 1), then the p. are not IID On 1] (ie, violation of coverage validity = violation of martingale's 's null, thus possibility for detection). Larger violations are easier to detect, and thus more likely to quickly raise an alarm. Note that this can be due to under-coverage (safety violation) or over-coverage (uninformative prediction sets); we further penalize trivial overcoverage i.e., when ((X++))=)-by using anticonservative WCP p-values whenever this occurs. (See pseudocode in Appendix F.)
Medium: shift may initially be "harmful" as deseribed above, due to density-ratio estimator having insufficient data, but later become "benign" once enough data has been collected.
Figure 5 provides an ablation study illustrating synthetic data example of WATCH performance for different magnitudes of covariate shift (in the input X distribution). Each row corresponds a specific magnitude of covariate shift and illustrates WATCH's response regarding coverage (prediction safety), interval widths (prediction informativeness). and WCTMs (monitoring criteria for alarms). The post-changepoint test points are sampled from the full source distribution with probabilities proportional to eap(1æ 18) * A); larger values of À thus correspond to more severe covariate shift toward extreme (and particularly toward large) values of the input X. Experiments are averaged over 20 seeds.
21

---

WATCH: Adaptive Monitoring for Al Deployments via Weighted- Martingales
X-CTM (Adaptation Ciiterion
Terget coverage
X.CTM Adaptation Threshold
Standard CP
Standard
WCTM Criterton)
Changepoint Data
Weighted Changepoint
Weighted
Severe Alarm Threshold
Post Changepoint Data
Changepoint
Moderate Alert Threshold
Changepaint
Synthetic- -Data Example
Mean Coverage
Mediaan Interval Widths
WCTM Values
(Pre vs. Post Changepoint)
(Prediction Safety)
(Prediction inforaativeness)
(Monitariing Criteria]
Median Morual with
Y00 s4p00 roa)
Morthngar
No shift (A- 0.00
9.
20
45
99
0.2
1090
2005
3000
400
1000
1069
2000
3003
4000
5089
10
5905
20:0
3000
4050
soes
1ig. age)
(Deplyymnn) Detepart Indeot
(Dxploymant) Detappar:: Inder
Text (Depasyment) Danapennt
1-3
Modian Feteraa meath
505
23"
"Mild" covariate shift (A 0.1)
0.-
PR coub en
eccordiigyy.
20
1000
3000
400
1009
3300
3003
4700
2030
soet
Test (Deployment)
(Deploymenti Detapoim
Text (Deplymmet) Danapeint
Yoog 94595 (001
"Moderate" covariate shift (A 0.2)
WATCH AINT
Cautian wartarted
20
40
30
1000
3000
4000 5000
1063
3000
2003
4300
2000
2006
4000
5005
(ing ago)
(Deployment) Datapo oe index
(Deployment) Datapoar: Indes
Text (Deplommnt) Dacapoint indea
1.0
Dhad Intarval waltt
"Extreme" covarlate shift (A 1100)
0.2
3P
0.5
Duercouerspe Intervats)
Unintamnativve Interval
33"
LIVI detected.
40
1590
20:0
1900 500
1009
2000
2002
4300
56.9
20:0
(Deplamment
(Usploymentt
Figure 5: Ablation study illustrating synthetic data example of WATCH performance for different magnitudes of covariate shift (in the input X distribution).
D.3. Wall-Clock Runtimes
Table 2 presents wall-clock runtimes averaged over 20 random seeds for the same experimental settings reported in the average detection delay (ADD) experiments in Table I, except with the true changepoint occurring 200 datapoints after deployment (whereas the results for Table are for 100 samples after deployment). Across all datasets and for both the anytime-valid and the scheduled monitoring criteria, the CTM and WCTM-based methods have faster empirical runtime than baselines. The starkest difference is between the Shiyaev-Robert, (W)CTM methods and the online changepoint detection method from Podkopaev and Ramdas (2021b), due to the former's runtime having complexity O(t) and the latter's complexity being (t2), where the number of post-deployment timesteps.
Table 2: Wall-clock runtimes in seconds, averaged over 20 random seeds for each monitoring method and dataset (error bars are standard errors), Anytime-valid monitoring methods are WCTMs (proposed), CTMs (Vovk. 2021), and sequential testing from Podkopaev and Ramdas (2021b) (PR-ST); multistage monitoring methods are Seiryaev-Roberr (SR) procedure applied WCTMs (proposed), SR applied to CTMs (Vovk, 2021), changepoint detection on the miscoverage risk from Podkopaev and Ramdas (202 in both online (PR-CD-online) and minibatched (PR-CD-50) variants. Results corresponding to our method are in blue. The best runtimes are bolded.
Monitoring Criterion (-)
Anytime- Criterion
Scheduled, Multistage Criterion
Monitoring Method (-)
(W)CTM
PR-ST
SR via (W)CTM
PR-CD-online
PR-CD-50
Dataset (L)
CP Type (L)
Time (+ SE)
Time (#: SE)
Time (+: SE)
Time (+: SE)
Time (#: SE)
Weighted
0.38 (+0.01)
7.65 (+1.29)
0.73 (+0.05)
81.75 (+10.17)
0.15 (+0.01)
MEPS
Standard
0.38 (10.01)
5.55 (+1.01)
0.73 (10.05)
81.97 (+1.01)
0.15 (10.01)
Weighted
0.44 (+0002)
10.26 (+1.72)
0.87 (+0.08)
138.11 (+16.36)
0.14 (=0.01)
Bike Sharing
Standard
0.43 (10.02)
8.45 (11227)
0.88 (10009)
135.94 (#16.92)
0.14 (20.01)
Weighted
0.43 (+-0.0)
9.93 (+133
0.88 (+0009)
215.72
(+20.63)
0.20 (=0.01)
Superconduct
Standard
0.42 (#0002)
9.94 (#1.03)
0,86 (=t0.0)
213.72 (+23.82)
0.18 (=0.01)
22

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
D.4. Ablation Experiments for Density-Ratio Weight Estimation
Another factor determining whether covariate shift can be considered benign or harmful deployment is whether deployed density-ratic estimator is well-specified and thus able to approximate the shift. Figure 6 provides selected synthetic-data example where legistic regression is a misspecified probabilistic classifier for distinguishing between pre- and post-changepoint data, but where a neural network (MLP) is able to accurately discriminate between the same pre- and post-changepoint data. That is, in this example the pre- and post-changepoint data are not linearly separable in the input X domain, so logistic regression is not able to reliably discriminate, and thereby it is unable to reliably estimate density-ratio weights via probabilistic classification. The result is that the changepoint causes large increase in coverage, despite some adaptation (decreasing interval widths); the estimator's 's misspecification thus causes WCTMs to raise an alarm, indicating that the covariate shift cannot be adapted to by the estimator. In contrast, the MLP estimator is able appropriately adapt by maintaing target coverage, improving interval sharpness, and avoiding unnecessary alarms.
X-M (Adaptation Criterton)
Target coverage Standard CP
Adaptation Threshold (for CTM)
Changepaint Data Pos--Chnngepiint Data
Standard
WCTM (Alam Criterion)
Weighted
Weighted
Alarm Threshold (for WCTM)
Changepoint
Changeeoint
Chargepoint
Density-Ratio
Synthetic-Data Example (Pre vs. Post Changepoint)
Mean Coverage
Median Interval Widths
WCTM Values
Weight Estimator
(Prediction Safety)
(Prediction Informativeness)
(Monitoring Criteria)
1.00
with
Moan Mastiogie Wide
39'
Logistic
Regression
0.30
0.95
0.93
5M0 2290 1999
sike tike Text (Depleyment) Detapcent indec
x (eg. aga)
Text (Deployment) Datapor re Index.
Test (Daplsymant) Datepsim Index
Yieg, poots 1ida
1.30
Moan Mastrogate Wide
30'
Neural Network (MLP)
23'
Moan
0.35
0.5
0.93
1he 1200
250 100
750
1290 1999
10"1 390
his like like
X (mg. ago)
Text (Deployment) Dataparne st Indeo.
Text (Daglisymnnt) Detepoin: Insect
Tand (Deplayment Detapcent indec
Figure 6: Ablation study on density- ratio estimator for synthetic data.
D.5. Ablation Experiments for Betting Function
The primary role of the betting function in (W)CTMs and resting-by-betting more broadly is for quickly rejecting the null hypothesis (i.e., raising an alarm) when it is violated. Figure 7 provides ablation experiment on the betting function used for X-CTMs and WCTMs, on three settings of the synthetic-data example. The "Composite" Jumper betting function is the betting function used in all other experiments, and it is an average of Simple Jumper betting functions over "jumping parameters" J E [0.0001, 0.001, 0.01, 0.1, 1]: here, we set the Simple Jumper baseline to have J = 0.01. See Vovk et al. (2021) for pseudocode and exposition of the Simple Jumper algorithm. J = 1 means conservatively spreading bets across all options to avoid cumulative losses, while smaller J encourages "doubling down" on bets that were previously successful. The CTMs with Composite betting are thus lower bounded at M, = 0.2, whereas those with Simple betting continually decrease, resulting slightly delayed detection speed relative to Composite.
D.6. Further Experiments on Corrupted Image Data
Here we provide additional experiments on the corrupted image data. Our evaluation metrics include the average detection delay (ADD), the average number of unnecessary alarms, and the average number of missed alarms. First, WCTM exhibits shorter detection delays compared to CTM. Given that CTM often doesn't trigger alarms in the face of severe corruption, which significantly increases its overall ADD. Additionally, the corruption level intensifies, WCTM detects shifts more quickly, whereas CTM's s detection speed shows minimal change- consistent with our earlier visualizations. To evaluate unnecessary alarms, we again create a mild, benign shift in the target distribution by mixing clean samples into MNIST and level-1 CIFAR- 10 corruption data, while treating higher-level corruptions as harmful. We find that WCTM's unnecessary alarm rate is roughly one-third that of CTM. Moreover, WCTM also exhibits fewer missed alarms, especially under more severe corruptions. These findings highlight the flexibility of our framework, which not only avoids unnecessary alarms but
23

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
X-CTM
(Composite Betting)
Adapt Thresh. X-CTMp)
WCTM (Simple Betting)
WCTM (Composite Botting)
Alarm Thresh. (for WCTMs)
Changepoint
Synthetic-Data Example (Pre Post Changepoint)
WCTM Values (Monitoring Criteria)
reg, sapais raae
Peechangepart Oata
Mild/benign covariate shift (in X)
Martingass
60
30m
540
lice
2003
240
1993
(eg. agel
Test (Deployment) Detapoint Index
Martingale Vals
leg. sepoil rak)
We er-CCvoorpcer Dara Changepoie
Extreme/harmful covariate shift
(eg, age]
Test (Deployment) Datapoint Index
Mean Mertingete Vals
321
Leg. sepais e1s0)
Wis changioee Cats
Fundamental concept shift (in Y[X)
Charge 1IX
1008
(eg. agel
(Deployment) Datapoint
Figure 7: Ablation study on betting function.
Table 3: WCTM achieves lower Average Detection Delay (ADD), especially under severe image corruptions, and also exhibits fewer false alarms and missed alarms. Results are averaged over 10 random seeds and all corruption types.
MNIST-C
CIFARIO-CLA
CIFARIO-CL
CIPARIO-C.5
ADD
WCTM
285.3
156.4
188.0
176.5
163.6
175.2
129.7
170.3
Unnceessary Alarm
WCTM
CTM
25.4
7.3
34.3
2.9
6.6
3.8
0.9
CTM
2.4
5,9
4.6
1.2
also better detects harmful shifts.
Whereas Table 3 provided metrics averaged over all corruption types, Figure 8 provides specific example results on each corruption type separately, focusing on harmful shifts with high severity of corruption. These plots illustrate how WCTMs quickly react to harmful shifts.
E. Experiment Details
E.1. Datasets
The evaluation datasets include both real-world tabular and image datasets. The tabular datasets are for regression tasks (where conformal methods used the absolute value residual nonconformity score S(x, y) ly - ()xx)), )) and the datasest span various sizes and dimensionalities: the Medical Expenditure Panel Survey (MEPS) dataset (33005 samples, 107 features) (Cohen et al., 2009), the UCT Superconductivity dataset (21263 samples, 81 features) (Hamidieh, 2018), and the UCI bike sharing dataset (17379 samples, 12 features) (Fanace- T, 2013). The image datasets were for classification tasks (where conformal methods used the one-minus-softmat score 5(x, y) = 1 p(y: æx)) were the MNIST-comuu AINIS--orrrptton (Mu and Gilmer, 2019) (60000 clean samples, 10000 corrupted samples) and 1FFAR-10-cooruption (Hendrycks and Dietterich, 2019) (50000 clean samples, 10000 corrupted samples), which are standard benchmarks for assessing distribution shifts.
24

---

WATCH: Adaptive Monitoring for AI Deployments via Weighted--onffrmall Martingales
Comrupted Martingale Paths
Comrupted Martingale Paths (Arylime-Valii Moritoriing Cfiterion)
Corrupted Martingale Paths
Corrupted Martinga le Paths
(Anytime-Valid Monitoring Criterfon)
AAnytime-Vliid Monitoring Criterion)
(Anytime-Valid Monitoring Criterion)
Mean Martingaa Values (80,0))
WITTT (arppwed)
WCTV (prppssed (Ubue 2023)
WCJM (popouall
WCTR troposedl
(/buc 2020)
Orarc 2001;
yezemn
penane
Chargessen
soportt
304
3p"
434
Moar Marsingate
Meat Marti Martingak
12
Test: (Deployment) Datapoint Index
Test (Deplyymen) Datapoint index
Test (Deplayment) Datapain Index
Test (Deployment Datapoint Indext
(a) brightness
(b) contrast
defocus blur
(d) clastic transform
Corrupted Martingale Faths
Corrupted Martingale Paths (Anytime-Valid Moritering Criterion)
Corrupted Martingale Paths VAnytieeValid Monitoring Criterion)
Corrupted Martinga Paths
(Anytime-Valid Monitoring Criterion).
(Anytme--alla Monitoring Criterion))
Meei Martingele Velues (66, (M6,66
tieuic 2021)
(bbcc atal, 1020)
CTN (bbsk 20221
(rorc 2023)
Chinpepartt
peppontt Aam streetoid
Changgseeit
the peport
Alam threshord
Alamm threshold
Alim thrshhald
2,3
307
2.3m
Maan Marthn Mwrttiggu
1//
20
Mosi
300 Test: (Deployment) Datapoint Index
200
200 Test (Deployment) Datapoint
Datapoint
Datapoint
(c) fog
(f) frost
(g) gaussian blur
(h) gaussian noise
Corrupted Martingale
Corrupted Martingale
Corrupted Martingale
Corrupted Martingale
(Anytime-Valid Monitoring Criterion)
[Anytime Val Criterion)
[Anytime-Valid Moriitoring Criterion)
Annyüine-Valid Criterion)
Mean Marting Kellve (R220)
WCNN pposcaed)
WCTh (proponad)
WCTh grrppesed:
Cme Dovi et 2023)
CTM (levic et al, 20231
CTH (vivk 20231
CTM Drawi at 20233
3210
Chanpspartt
changepeit
2,30
Changaseins
30
Chii paport
Zkim theddwid
Abvim trrehhit
L
33-4
Moan Martingale
2:2
Marting
Meain Martii Maritnng
cam
Test (Deploymeng) Datapoint Index
(Deployment) Datapoint
300
Test (eploymentt Datapoirt pint Indox
Test (Deployment Datapoint Indext
glass blur
(i) impulse noise
(k) jpeg compression
motion blur
Figure& 8: Results on CIFAR-10 with various corruption types, all at the highest severity level. WCTM reacts more quickly than the standard CTM under these conditions. Moreover, with several types corruption, the standard CTM does not raise any alarms all.
25

---

WATCH: Adaptive Monitoring for Al Deployments via Weighted-Conformal Martingales
Class Coverage Rate
Class Coverage jerzgartion Thue Classes Prediction
Class Coverage Rate
Class Ceverage Rate
jBrepartion Thue Classes Prodiction
(Bregartion
Clacees in Prediction
(Bregartion
Prediction Sets)
WCTM (orepened) Wius et 2021 Char geasin
WCTU proppssd:
CTM (thes all 20201
45-
Crangepshe:
Coveern Poeeragee.o Rate [-+)
Covelgge oveeage Rate!
Coverage
62
WCTTM 22 proppsedd
NCTV propssedd
2.2
(Vbuc 20211
CTN (Uhec at 20211
Crargeasini
Changepcin
são 13kce Test (Deplnyment) Datapoint index
5o0o 22533 zeles
32593 Test (Depllyment) Datspoint Incle
Test (Depiymment) Datapeint index
Test (Deployment) Datapoint Indle C
Prediction Set Size
Prediction Set Size
Prediction Set Size
Prediction Set Size (Numbee of Classes in Sell
(Number Classes Sat)
(Number of Classes in Set)
(Number of Classes in Seil
WCTM (proposeet
CTM trosk et al. 202. Changegcint
2.0
MMM WW
2.0
Average Set Size (-)
Average Set Size
1:25
Aversge Set
Average
14
22
WCTN (propssed)
WCIM (proposec)
WCTH (propesed)
CTM (Vosk al. 252
CTM (9owk
222
CTM (ook 2523)
Changepent
10
10
Test (Depleyment) Datapoint Indlxx
1000
3200
Test (Deplayment ent) Datapoint Index
2010
4302
1000
9030 20000 18900 1400
5000 Test (Deployment) Datapoint Index
2000
4393
20000 12030 1400
5000 Test (Deplayment) Datapoint Index
2600
4307
20000 12020 14000
(a) No Corruption
(b) Minor Benign Corruption
(c) Benign Corruption
(d) Extreme Corruption
The results supplement Figure 3 in the main paper. They demonstrate the coverage rate and prediction set size under four different corruption scenarios. In the multi-class classification setting, we adopt metrics different from those used in the regression experiments and follow Romano et al. (2020) to measure the prediction set size and coverage rate (defined as the proportion of true classes in the specified range, NOT conformal coverage) as principled risk metrics for distinguishing benign from harmful shifts. We increased the size of the validation set and the number of samples visualized to yield more robust performance and provide a clearer view of the trajectory; the results are averaged over a window size of 200, while all other configurations remain unchanged from the original setting. As discussed in the paper, we mixed test samples (target corrupted) with validation samples (source clean) improve the estimation of weights for CTMs. So under corrupted scenarios, the "starting points" before the change points for WCTM and CTM differ, the mixture allows the validation set to contain corrupted data; however, this difference is not clearly reflected in the martingale paths. Overall, the models initially exhibit relatively high classification performance under the clean setting, while CTM rapidly declines to a lower performance level under all corruption conditions. Although WCTM adapts to changes in benign scenarios, it eventually demonstrates severe metric changes under extreme shifts as well, which corresponds to the results in Figure 3.
Figure 9:
E.2. Simulating Shifts in Tabular Data
To evaluate the online adaptation performance of our proposed WCTMs on the tabular, regression- task datasets, we simulated mild or benign covariate shifts by exponentially tilting (i.e., up-sampling) the test samples from the full dataset based on selected covariates. On the MEPS healthcare dataset, h selected to simulate demographic shift towards younger, higher-educated patients (expected to be a benign shift due to youth tending to correlate with fewer, less variable health issues). On the bike-sharing dataset, / simulated a shift in weather patterns to colder, windier days; meanwhile, a more complex shift was simulated on the superconductivity dataset by sampling proportional to the projection onto the first principal component representation of the training data. Harmful concept shifts in tabular data were simulated by biasing the label values as function of selected input covariates for each dataset.
E.3. Further Details of Image Classification Experiments
We provide further discussions on the image classification experiments from Section 4.2, Figure demonstrates the coverage rates and set sizes of different corruption levels as discussed in the main paper. Details of model architectures and training configurations can be found in Table 4 Table 7.
26

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Table 4: MNISTDiseriminator Model Details
Component
Details
Model Name
MNISTDiseriminator
Purpose
Binary classifier to distinguish between source (uncorrupted) and target (cor-
rupted) MNIST data
Architecture Type
Convolutional Neural Network (CNN) with 2 conv blocks + FC layers
Input Shape
(Batch_size, 1, 28, 28) Grayscale MNIST images
Output Shape
(Batch_size, 2) Binary classification logits
Total Parameters
1.33 parameters
Layers
First Conv Block: Conv2d(1-332, 3x3) - BatchNorm + ReLU 7
MaxPool + Dropout
Second Conv Block: Conv2d(32-464, 3x3) + BatchNorm + ReLU +
MaxPool(2x2) + Dropout
Fully Connected: Linear r
BatchNorm ReLU
Dropout
Output: Linear(128-22)
Regularization
Dropout (rate=0.3), Batchhormalizat
Calibration Method
Temperature scaling (initial temp=1.5)
Activation Functions
ReLU
Loss Function
Cross-entropy (implicit in the code)
Training Epochs
30
Batch Size
64
Learning Rate
0.001
Optimizer
Adam
Special Features
Temperature scaling parameter for improved probability calibration
Usage Context
Used for estimating likelihood ratios in weighted conformal prediction
Algorithm 1 Calculate weighted conformal prediction set for covariate shift (Tibshirani al., 2019).
Input: Calibration data Zi. == {(x,, Y=) Test point input X Score function (X 2) - (R+)":
Density-ratio weight function à X [0, 1] Significance level (target miscoverage) a.
Output:
Conformal prediction set C,, V.
Calculate noncon.ormity scores and n +1 density-ratio weights: UI..., Un = Zn)
01 Wnt1 (X...... Xn+1)
Calculate - a quantile over weighted score distribution (with conservative ad- justment un+1 = 00, as Yn+1 unknown):
Q1-a Qi-a (En W; on + Wn41 -00) [Note: hn4t 2 a = Q1-a = =o.
Calculate weighted conformal prediction set: (xn(X))1) = y': 8(X.+1,8) S {n-a) (Note: 91-a = (Xn-)= = =t}}
Return: (nn(()n41)
F. Algorithms
Only limited algorithm pscudocode provided at this time, focused on algorithms that explain how we penalized noninfor- mative conformal prediction intervals (i.e., with anticonservative p-values). We aim to update the arXiv version of this paper with more comprehensive pseudocode.
27

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Table 5: CIFARIO Discriminator Model Details
Component
Details
Model Name
UFFAROODiscimmiatoo
Purpose
Binary classifier to distinguish between source (uncorrupted) and target (cor-
rupted) CIFAR- 10 data
Architecture Type
Convolutional Neural Network (CNN) with 3 conv blocks + FC layers
Input Shape
(Batch_size, 3, 32, 32) RGB CIFAR-I0 images
Output Shape
(Batch_size, 2) Binary classification logits
Total Parameters
R:48M parameters
Layers
First Conv Block: Conv2d(3-164, 3x3) - BatchNorm + ReLU 7
MaxPool (2x2) + Dropout
Second Conv Block:
Conv2d(64-1128, 3x3) - BatchNorm + ReLU +
MaxPool(2x2) + Dropout
Third Conv Block: Conv2d(128-2256 3x3) BatchNorm ReL.J
MaxPool(2x2) + Dropout
First FC Layer: Linear((56x4x44551] > BatchNorm + ReLU
Dropout
Second FC Layer: Linear(512-+122) BatchNorm ReLU - Dropout
Output: Linear(128--2
Regularization
Dropout (rate=0.3), Bat aat.chhomliiaaaiiii. ion at each layer
Calibration Method
Temperature scaling (initial temp=1.5)
Activation Functions
ReLJ throughout the network
Loss Function
Cross-entropy
Training Epochs
30
Batch Size
64
Learning Rate
0.001
Optimizer
Adam
Special Features
Temperature scaling parameter for improved probability calibration
Usage Context
Used for estimating likelihood ratios weighted conformal prediction
Algorithm 2 Calculate weighted conformal p-vuuee that penalizes noninformativeness.
Input: Calibration scores Vim == (Vi..., Vn W Test point score Vn+1; Weight vector ã == (w1, ... Wn+1) )); Signifi-
cance level (target miscoverage) a. Output: Weighted conformal p-value ph+1-
if Wn+1 < a then
Calculate exact weighted conformal p-value (Eg. (9))
Sample un+1 ~ 1]
sdd
p=
= > 11 W; [1 > vn+1) + unti = =0++]]
n+1
else
(Note: This "else" condition implies the corresponding CP set for the test point was noninformative, i.e., (X,((1)11) }; thus, penalize neninformative- ness by using derandomized, anticonservative p-value, equivalent to setting Mnt1 0. These will cause WCTM to raise alarm more quickly, they inten- tionally break IID uniformity based on the magnitude of Wz41.)
@ E W; 1e > en+13
end
Return: ph-1
28

---

WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales
Table 6: RegularieedMNISTModee Details
Component
Details
Model Name
RegularieedMNNiSTModde
Purpose
Classification model for MNIST digits (0-9) with robustness to corrupted images
Architecture Type
Convolutional Neural Network (CNN) with 3 conv blocks + FC layers
Input Shape
(Batch_size, 1, 28, 28) Grayscale MNIST images
Output Shape
"Batch_size 10) Logits for 10-class digit classification
Total Parameters
R600K parameters
Layers
First Conv Block: Conv2d(1-322, 3x3) > BatchNorm + ReLU 7
(2x2) - Dropout
Second Conv Block:
Conv2d(32-164, 3x3) - BatchNorm + ReLU
MaxPoo1(2x2) + Dropout
Third Conv Block: Conv2d(64-+188, 3x3)
BatchNorm ReLU
MaxPool (2x2) - Dropout
Fully Connected: Linear(128x3x332255)) + BatchNorm + ReLU +
Dropout
Output: Linear(256-11))
Regularization
Dropout (rate=0.3), Bat Batchhoormalizaiio at each layer
Activation Functions
ReLU throughout the network
Loss Function
Cross-entropy
Training Epochs
30
Batch Size
64
Learning Rate
0.001
Optimizer
Adam
Special Features
Extensive regularization for robustness corrupted images
Usage Context
Primary classification model for MNIST digits conformal prediction frame-
work
29

---

WATCH: Adaptive Monitoring for AT Deployments via Weighted-Conformal Martingales
Table 7: ResNet20 Model Details (Standard Format)
Component
Details
Model Name
ResNet20
Purpose
Classification model for CIFAR-10 images
Architecture Type
Residual Network (ResNet vl) with basic blocks
Input Shape
(Batch_size, 3, 32, 32) RGB CIFAR-10 images
Output Shape
(Batch_size, 10) Logits for 10-class CIFAR- classification
Total Parameters
R270K parameters
Depth
layers initial conv + layers blocks final linear)
Block Structure
BasicBlock: Cony-BBN-ReLL--Coo--BB shortcut connection, fol- lowed by ReLU
Network Architecture
Initial Layer: Conv2c(3+16, 3x3) - Batchoon ReLU
Stage I: 3 BasicBlocks (16 channels, stride=1)
Stage 2: 3 RasicBlocks (32 channels, stride=2)
Stage 3: 3 BasicBlocks (64 channels, stride=2)
Output: G1 Global AvgPool > Li (inear(64-110)
Regularization
BatchNormalization in each block
Activation Functions
ReLU
Weight Initialization
Kaiming normal for convolutional layers, constant for batch normalization
Loss Function
Cross-entropy
Training Epochs
30
Batch Size
64
Learning Rate
0.001
Optimizer
Adam
Special Features
Skip connections (residual learning) for better gradient flow
Usage Context
Primary classification model for CIFAR- 10 in conformal prediction framework
30


Reviews:
[{"review_id": "guPPXK2BZm", "reviewer": "Reviewer_b4ce", "summary": "This works proposes a weighted generalization of conformal test martingales (WCTMs), for online change point detection, which can continuously adapt to benign shifts without raising unnecessary alarms and quickly detect harmful shifts.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback! We refer to the following anonymous link for supplemental figures and algorithms: https://sites.google.com/view/authorresponse/home\n\n(1) **Clarifying writing, especially novelty of contributions relative to Vovk et al:** Regarding your comment on how “writing can be improved,” we are actively revising our paper for clarity. For example, re your comment on clarifying “innovations as compared with the previous literature,” we have revised the last part of our Introduction section to the following (this references a slightly revised Fig 1, available at the provided link):\n\nRecent advances in anytime-valid inference (Ramdas et al., 2023) and especially conformal test martingales (CTMs) (Volkhonskiy et al., 2017; Vovk, 2021) offer promising tools for AI monitoring with sequential, nonparametric guarantees. However, existing CTM monitoring methods (e.g., Vovk et al., (2021)) all rely on some form of exchangeability (e.g., IID) assumption in their null hypotheses---informally, meaning that the data distribution is the same across time or data batches---and as a result, standard CTMs can raise unnecessary alarms even when a shift is mild or benign (e.g., Figure 1a). Meanwhile, existing comparable monitoring methods for directly tracking the risk of a deployed AI (e.g., Podkopaev and Ramdas (2021b)) tend to be less efficient than CTMs regarding their computational complexity, data usage, and/or speed in detecting harmful shifts (Sec. 4.3). \n\nOur paper's contributions can be summarized as follows:\n\n- Our main theoretical contribution is to propose weighted-conformal test martingales (WCTMs), constructed from weighted-conformal $p$-values, which generalize their standard conformal precursors. WCTMs lay a theoretical foundation for online testing of a broad class of null hypotheses beyond exchangeability, including shifts that one aims to model and adapt to.\n\n- For practical applications, we propose WATCH: Weighted Adaptive Testing for Changepoint Hypotheses, a framework for AI monitoring using WCTMs. WATCH continuously adapts to mild or benign covariate shifts (e.g., Figure 1a) to maintain end-user safety and utility (and avoid unnecessary alarms), while quickly detecting harmful shifts (e.g., Figure 1b \\& c) and enabling root-cause analysis.\n\n\n(2) **Additional Explanatory Ablation Experiments on Covariate Shift Magnitude:** To address your comment that our distinction between mild and extreme covariate shifts are somewhat “vague,” we have added additional ablation experiments to illustrate how WATCH performs with different magnitudes of covariate shift--please see Supplement S1.1 at the provided link. Ie, this ablation experiment empirically clarifies that covariate shifts can indeed be viewed as a spectrum from benign to harmful rather than as a binary category. For the monitoring purpose that our paper focuses on, however, we consider “benign” shifts to be those that can be safely ignored without sacrificing system safety (i.e., coverage) or system informativeness/utility (i.e., prediction set sharpness/interval width); our WCTM methods can thus be understood as performing a flexible (nonparametric) statistical test for whether a given shift is benign or harmful to the system, in this sense, with sequential false-discovery guarantees given by our reported theoretical results.\n\n\n(3) **Added Pseudocode, Highlighting How Algorithm Treats Benign vs. Harmful Shifts:** We are adding comprehensive pseudocode for all proposed algorithms as a new appendix section (Appendix E). For now, we have provided the annotated pseudocode for the specific sub-module of our methods that responds slightly differently to mild versus extreme shifts in $X$--please see Supplement S.1.2 for this pseudocode at the provided link. Specifically, the pseudocode highlights one way that our methods penalize (and thereby more quickly detect) covariate shifts that are so extreme (i.e., out-of-support or very little support) that the corresponding weighted conformal interval becomes noninformative (i.e., predicting the whole label space, $\\hat{C}(X_{n+1})=\\mathcal{Y}$ ). \n\n\n(4) **Spillover responses for Reviewer 7BRr (due to char lim) -- Methods And Evaluation Criteria:**\n \n- *Widths shrunk in Fig 2:* Due to the density-ratio estimator appropriately placing more weight on cal points with smaller scores--no further correction.\n\n- *Ablations on betting function:* In S2.3 at the provided link, we compare the Composite Jumper betting function to the Simple Jumper betting function used in baselines (eg, Vovk et al).\n\n- *Classification into benign and severe shifts:* See responses #2 & #3 to reviewer b4ce. Regarding a “target risk score,” note that (W)CTMs test for IID uniformity on the $p$-values; IID uniform $p$-values implies valid coverage at any significance level, which is more powerful than for a threshold. So, we argue our def is at least as principled. See sec 4.2 at the link provided for coverage results.", "claims_and_evidence": "partially, see \"Other Comments Or Suggestions\" below, mainly the distinction between mild covariate shifts and extreme covariate shifts is not clear.", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "partially, the theorems seems to be correct to the best of my knowledge.", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "it is related to the broad community of online learning and distributional change detection, and is mostly related with Vovk 2021 and Vovk 2022 which studies the non-weighted version of conformal test martingale.", "essential_references_not_discussed": "N/A, and I am not fully familiar with all literature in this area.", "other_strengths_and_weaknesses": "Strengths: a new weighted version of CTM that can adapt to benign changes and detect harmful changes quickly.\n\nWeakness: the writing can be improved, and more mathematical concepts and a clearer definition of benign and harmful changes will make the problem setup easier to understand.", "other_comments_or_suggestions": "The mild covariate shifts and extreme covariate shifts are a bit vague in this paper, without clear distinction in terms of, for example, the covariate shift magnitude. Only with a clearer definition of benign shifts and harmful shifts, users can adjust the boundary between benign and harmful and tailor the detection algorithm accordingly. Due to this reason, the overall detection algorithm seems vague to me, and I would appreciate it if the author(s) could provide a detailed algorithm (including the choice of f functions) for performing the detection online, and justify how the algorithm treats benign and harmful shifts separately.\n\nThe proposed method seems to depend a lot on Vovk 2021 and Vovk 2022. And it should be discussed whether the weights added is the only difference or maybe there are more innovations as compared with the previous literature.", "questions_for_authors": "see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TytUVxZ640", "reviewer": "Reviewer_oGBa", "summary": "This paper proposes WATCH, a novel method that is able to check machine learning models after deployment to identify if their input data changes unexpectedly. WATCH uses a new approach, Weighted Conformal Test Martingales (WCTMs), to detect these changes. WATCH can ignore small data changes that do not affect the model much and only raises warnings when big changes occur. The authors conduct experiments on real datasets and show that WATCH performs better than previous methods at correctly identifying important data shifts without many false warnings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback! Please find our responses to your questions/comments roughly in order below. We refer to the following anonymous link for supplemental figures and algorithms: https://sites.google.com/view/authorresponse/home\n\n**Claims and Evidence:** \n- *Novelty of weighted-conformal $p$-values:* The general weighted-conformal $p$-values presented in Section 3 are novel in this paper. References [1] and [2] have a similar setup based on a weighted empirical distribution of nonconformity scores, but they only consider taking quantiles on the weighted distribution to compute prediction sets, they do not define, introduce, or mention weighted-conformal $p$-values nor any equivalent quantity, and they do not discuss how weighted conformal ideas can be used for hypothesis testing or monitoring (the focus of our paper). \n\n- *References for concept shift:* Concept shift--a shift in $Y|X$, the label distribution conditional on the input--is a common term and topic of study in the ML robustness literature. The earliest refs we could find discussing the idea of concept shift are the following--we have added these to the paper: (a) Webb, G. I., & Ting, K. M. (2005). On the application of ROC analysis to predict classification performance under varying class distributions. Machine learning, 58, 25-32; (b) Fawcett, T., & Flach, P. A. (2005). A response to Webb and Ting’s on the application of ROC analysis to predict classification performance under varying class distributions. Machine Learning, 58, 33-38. While prior CTM methods are also capable of detecting concept shift, our main novel contribution regarding concept shift is to enable valid continual monitoring for concept shift *even when covariate shift is also possible,* i.e., by enabling diagnosis between the two, and demonstrating fast detection empirically. At the link we have provided above, we have added additional ablation experiments on the simple synthetic data to illustrate how this behavior.\n\n**Methods And Evaluation Criteria:**\n- *Eval on larger models and datasets:* The theoretical guarantees of our methods make no assumptions on the ML model architecture or on the modality/complexity/size of the dataset, so yes in theory we could demonstrate similar performance with ViT on ImageNet. Our paper focuses on laying a theoretical and algorithmic foundation for WCTM-based monitoring methods, while opening the door to future evaluation on frontier and foundation models.\n\n- *Other types of shifts:* Yes, there are other types of shifts that can be studied. One example is “label shift,” which is a shift in the marginal $Y$ distribution (but not in $Y|X$); another example is slow distribution drift over time (rather than at a single changepoint). We have added discussion of WCTMs for these settings to our Conclusion.\n\n**Other Strengths And Weaknesses:** \n- *Practical use in large-scale ML system:* In large-scale ML deployments, the proposed WATCH framework could be used to monitor the performance of the system to detect harmful behavior patterns (e.g., increases in LLM hallucinations, jailbreaking attacks, toxic prompts/generated text), while also adapting online to more benign shifts (e.g., seasonality, user trends, etc). For instance, WCTMs could be run to monitor the distribution of LLM outputs while serving as a “safety filter” to provide probabilistic guarantees on the safety of the output shown to users. The massive computational cost of training large models paired with their wide-reaching effects makes the task of monitoring crucial, to quickly catch unsafe behavior while also minimizing costly retraining. \n\n- *Improving model dynamically at deployment time:* WATCH implements one approach to dynamically improving the model at deployment time, that is by performing online density-ratio estimation to maintain prediction safety (coverage) and informativeness (widths). Root-cause analysis can also inform other approaches to improving performance at deployment time, eg, via distributionally robust training (to be robust to detected shifts).\n\n**Spillover from Reviewer 7BRr (char lim)--Methods & Evaluation Criteria:** \n- *Specific baseline e-process:* We compared to the betting-based $e$-process in Podkopaev & Ramdas (2021) (from Waudby-Smith & Ramdas (2024)) since those methods performed best and to standardize. We compared WCTMs to sequential testing variant with standardized anytime-false alarm rate of 0.01; for SR-WCTMs we compared vs changepoint detection variant with common average-run-length (under the null) of 20,000.\n\n- *Runtime metric:* Runtime can be surprisingly nontrivial for monitoring long time-horizons. Ie, the changepoint baseline in Podkopaev & Ramdas (2021) has $O(t^2)$ complexity (due to starting a new sequential test at each time $t$); our comparable SR-WCTM is only $O(t)$. \n\n- *Coverage vs alarms:* In our Appendix, we plan to add coverage & width plots for all expts in our paper; some are available already at the link.", "claims_and_evidence": "- The claim on the contribution of using weighted conformal p-Values could be discussed more clearly. After reading section 3.1, i feel i am not clear on whether weighted conformal p-values are novel in this paper. I would like the authors to indicate clearly on how this is related to prior works such as [1, 2].  \n\n- Is the notion of the concept shift defined before? I couldn't find any discussion on this. I think this is important, given that this paper shows benefit most on this type of shift. So i wonder if it is new or it is considered an important task in this area before. I would love the authors to provide more evidence in this aspect to support the claim of the strong results.\n\n\n[1] Conformal validity guarantees exist for any data distribution (and how to find them). 2024. \n[2] Conformal prediction under covariate shift. 2019.", "methods_and_evaluation_criteria": "- In the experiment section, the evaluated models are mostly simple neural networks, which seems rather limited (e.g., MNIST and CIFAR-10). I wonder if the evaluation can be conducted on large models and datasets, for example, ViT on ImageNet.\n\n- In the paper, the authors mostly consider covariate shift or concept shift, I wonder if there are other types of distribution shifts that could exist. I would like to see more discussion on this aspect.", "theoretical_claims": "I check the proof of theorem 3.1, 3.2 and 3.3, which is correct. But i would suggest to highlight the idea of the proof in the main text, rather than deferring them entirely to the Appendix.", "experimental_designs_or_analyses": "I find no issue in the soundness or validity of the experimental design or analysis. The evaluation mostly follow prior setups in this area.", "relation_to_broader_scientific_literature": "I think the idea of detecting changes in the test data distribution could have many practical applications. The first is to monitor the deployment setting in real time. Also, this could help model developers adapt the model better to address potential catastrophical failures.", "essential_references_not_discussed": "I don't have concerns on this aspect.", "other_strengths_and_weaknesses": "- I am wondering if the authors could discuss more on how this result is practical, like in a real-world setting, how can it be used in large-scale machine learning systems. This could help reader better understand the impact of the paper.\n\n- If the shift is detected, then what can we do to address the problem? Are there ways to leverage the proposed detection method to improve the model dynamically at deployment time? If so, how would you do it?", "other_comments_or_suggestions": "None", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8KTeDtZetO", "reviewer": "Reviewer_7BRr", "summary": "The paper proposes an extension of conformal test martingales (CTMs) by the use of weighted conformal p-values rather than standard conformal p-values. The authors argue that such weighted p-values permit testing for more general hypotheses beyond simple (online) exchangeability, and propose a hypothesis that helps disambiguate between covariate and concept/label shifts. A density ratio estimator is employed and the weighted p-values are shown to experimentally adapt to small covariate shifts, whereas stronger covariate and concept shifts are detected. The method is compared to standard CTMs and one type of sequential e-process-based testing on tabular and image data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time, interest, & detailed feedback! We refer to the this anonymous link for supplemental figs: https://sites.google.com/view/authorresponse/home\n\n**Claims and Evidence:** \n- *SOTA statement:* We have removed the quoted statement, & revised it to refer specifically to how our methods compare to the evaluated baselines (regarding adaptation & detection speed).\n- See “Suggested baselines/Relation to Lit” section.\n\n**Methods And Evaluation Criteria:** See “**Spillover from 7BRr**” in response to **oGBa** for:\n- *Specific baseline e-process* \n- *Runtime metric* \n- *Coverage vs alarms*\n\n**Theoretical Claims:** \n- *Thm 3.2 & 3.3:* We are happy to rename “Thm 3.2” to “Proposition 3.2,” and same for 3.3 (we think proposition is more accurate than lemma). \n\n- *Intuition for Thm 3.3:* The guarantee is controlling the average-run length (ARL) *under the null hypothesis*, meaning how long the “scheduled” monitoring procedure can be expected to run without a false alarm before it needs to be reset. What you describe about detection delay would be a guarantee *under an alternative hypothesis;* we do not report such guarantees.\n\n**Experimental Designs Or Analyses:** \n\n- *X-CTM design & ablation of WCTM without X-CTM:* The $X$-CTM uses the same design (ie, same betting function) as the WCTM, but with a nearest-neighbor-distance score. Without a XCTM, a WCTM has 2 main options: (1) no adaptation, which reduces to standard CTM; or, (2) assume that the changepoint occurs at deployment time. See S2.1 at the link for an ablation study on (2).\n\n- *Eq (17):* If every test point is added to the calibration data (as in standard CTMs), then the weights become intractable to compute (Prinster et al. 2024); but, can avoid this by, at some point $t_{ad}$ (given by $X$-CTM), treating the cal data as fixed (no longer adding recent test points to it). (Fixing the cal set results in the WCTM testing a hypothesis that is *conditioned* on the cal set, vs marginal over it; we will add an appendix section to elaborate.)\n\n- *Density-ratio estimator:* The density-ratio weights were estimated via probabilistic classification, as described in Tibshirani et al (2019), Eq (9). Tabular experiments used sklearn’s LogisticRegression; image experiments used a prefit MLP with 2 hidden ReLU layers, fit on 5K pre-change and 5K post-change points. An example ablation study for the effect estimator misspecification is provided in S2.2 at the link provided.\n\n- *Widths shrunk, betting ablations & benign/severe shifts:* **See response #4 to b4ce.**\n\n- *Table 2 questions:* We meant to write “Unnecessary Alarm” to refer to alarms raised when there is a mild/benign shift. The rejection threshold for Table 2 was $10^6$ (as in Fig 3). WCTM unnecessary alarms are evidence that density-ratio estimation is harder for image data. We will add clear discussion of empirical vs nominal rates.\n\n- *CTM higher ADD:* Related to our discussion around Eq (17), once the WCTMs treat the cal data as fixed, they can sometimes reject faster than standard CTMs (the latter adds each test point to the cal set, which confuses the source/target distinction and sacrifices power). \n\n- *Benefit of WCTMs for tracking loss metric:* A loss metric can take the role of the nonconformity score, and WCTMs then would adapt to distribution shifts to maintain statistically valid estimates of how that loss compares to cal data. \n\n- *Suggested baselines/Relation to Lit:* (1) weighted CP methods are not comparable to WCTMs; rather, any WCP method offers a different opportunity to construct a WCTM *on top of it.* Ie, our expts are roughly construct WCTMs on top of WCP methods similar to [1]; in future work it would be valuable to consider WCTMs constructed for continual drift [2] and label shift [3]. We are happy to add comparisons to a subset of (2) and (3) in a camera-ready version, and we will discuss all of these more thoroughly in Related Work. In particular, [4-6] appear to primarily differ from our CTM baseline by betting function and ensembling (which can also be done with WCTMs). [7] is similar to us in the goal of online adaptation, although they focus on adapting the point prediction (while we adapt the weights); we have added more discussion of this. [8] and [9] could be compared to, although the CTM references are likely more closely related.\n\n**Other Weaknesses:** We have updated sec 3.3 subheading to “Main Practical Testing Objective.”\n\n**Other:** We are carefully revising to address typos, to improve clarity and readability. See response #1 to reviewer **b4ce** re contributions outline. $[0,1]^*$ is notation for a vector of any length with entries in [0,1], we will clarify. We meant Eq 7 is the definition for any $t$, we can update if unclear.\n\n**Eq 13 Q:** Should mean a shift to some $\\hat{F}_X^T$ *such that* the density-ratio is $\\hat{f}_X^T/f_X$. So, $f_X$ does *not* need to be known, only an estimate of the *ratio* is needed. We will update notation here.", "claims_and_evidence": "The approach is motivated by three goals listed in the introduction and in Figure 1, namely adaptation to small covariate shifts, detection of stronger shifts, and disambiguation on shift origin. From what I can see, all of these goals are addressed to some extent by the method, and are not overclaiming.\n\nI was nonetheless bothered by the claim that  \"Our empirical results show that WATCH substantially outperforms SOTA in these areas\" which I find to be overly strong, given the actual reported results (e.g. Table 1 and 2), as well as lack of baseline comparisons and some of the issues I had with the experimental designs in general (see below).", "methods_and_evaluation_criteria": "The method (WATCH/WCTM) leverages a single test martingale design adapted from prior work (fixed betting function design, nonconformity scoring function, density ratio estimator, Shiryaev-Roberts supplement, predictor) and is compared against standard CTMs on three tabular datasets with simulated benign shifts (Fig. 2), and against CTMs and a sequential e-process-based testing procedure (not clearly specified which one from that paper) on MNIST and CIFAR-10 with corruptions. Methods are compared in terms of adaptability (for the benign shifts), detection delay, alarm rates, and runtime. Overall the considered criteria seem reasonable, although I don't necessarily think that runtime is a strong metric for this type of online monitoring given they're all relatively fast, and I believe more focus needs to be placed on the disambiguation in terms of (false) alarm rates and maintaining or breakdown of coverage guarantees coinciding with raised alarms.", "theoretical_claims": "The theoretical results are relatively straightforward and rely on key arguments from prior work, with an adaptation to weighted p-values. Thm. 3.1 seems most novel by extending the exchangeability testing argument of conformal p-values to weighted variants. Thm. 3.2 and 3.3. are exclusively reliant on existing arguments (e.g. Ville's Inequality applied to standard CTMs), and simply a direct invocation thereof. In fact, I would suggest to soften the notion of novelty associated with a \"Theorem\" here -- why not call them Lemmas instead? \n\nOn another note, it would be nice to add intuition to the use of Thm. 3.3. Am I to read this as a softening of the high-probability false alarm guarantee given in 3.2 in terms of an expectation term? I am also unsure because it seems to read as a lower bound on the detection delay, but this does not say anything about the \"good\" performance of the method (i.e. akin to worst-case or upper bound?)", "experimental_designs_or_analyses": "My key issues with this paper are in the experimental section, which reads unclearly and omits all specifics on experimental design, metrics, baselines etc. Instead very similar-looking plots are repeatedly shown (but at times anecdotal, e.g. Fig. 3) to double-down on the difference between WCTM and standard CTM, whereas the comparison to sequential testing ends up allocated to two hard-to-read results tables with no explanations on the considered metrics and their calculation.\n\nSome of the questions I had reading this section where:\n- The X-CTM is introduced very late and in a rushed manner, even though it is fundamental to the method. Is the same design used as for WCTM? How would the WCTM perform in benign covariate shifts without the additional information given by the X-CTM, just on its own?\n- I didn't follow Eq. 17 on improving the martingale's adaptivity -- is this meant to symbolize a truncation of the past observations?\n- How exactly are the density ratio weights computed, and why was this particular estimator chosen? Ablations/comparisons to other options? How can we trust that this estimator is doing a good job, given its crucial role? \n- How exactly are the interval widths shrunk in Fig. 2? Is this exclusively due to the obtained weighting, or is there some kind of coverage-based correction?\n- Ablations/comparisons on the role of the betting function, which is known to be very important in dictating the growth of the test martingale?\n- The classification into benign and severe shifts is somewhat ad-hoc -- a shift is essentially called severe after the fact if the WCTM does not manage to adapt appropriately. E.g. in Fig. 3c the associated shift is a Level-3 corruption on 60% of samples -- who is to say that this is not severe? Rather than rooting those statements in their own method's ability, it seems more principled to root them in some kind of target risk score (e.g. conformal coverage or miscoverage risk reaching some threshold). I see something along those lines in Fig. 2 (left column) where coverage seems satisfied even under benign shift (also for CTMs). Can we see coverage results for sec 4.2 and Table 2?\n- Standard CTMs are equipped with the same Type-I error control or false alarm guarantees as the proposed method since they both rely on test martingales. So, I was surprised to see that Table 2 records such high false alarm rates. How can I explain this? Is this because in fact they are designed for the simple exchangeability hypothesis, whereas here you consider the one from Eq. 13, and thus CTMs do not have that guarantee? Then the comparison seems somewhat unfair or not clearly disambiguated. Associatedly, what kind of rejection thresholds are used (these are no-where discussed). Can we see a clear discussion on nominal vs. empirical guarantees and associated empirical false alarm rates for sec 4.2 and 4.3?\n- How come that CTMs record higher detection delays than WCTMs, given that they (1) do not try to adapt to shift and (2) are repeatedly shown to be more sensitive and reject faster in the plots? The statements made in L410-413 seem contradictory to that intuition.\n- CTMs can also be used without a rejection threshold, merely as a reflection of evidence for occured changepoints that does not disambiguate between severity of shifts. If this was coupled with the tracking of some loss or risk measure to assess severity such that decisions on raising alarms can be taken, what benefit is obtained by WCTM?\n- Why are there no baseline comparisons to (1) standard weighted CP methods which are clearly related, e.g. [1,2,3], (2) prior conformal test martingale methods, e.g. [4,5,6], and (3) related sequential testing and changepoint detectors, e.g. [7,8,9], including methods that similarly try to adapt to shifts online? These all seem very relevant, and a meaningful subset thereof would help clarify the benefits of WCTM by highlighting the limitations to each method (e.g. CTM vs. WCP) and how the combination gives new benefits. \n\n[1] Tibshirani, Ryan J., et al. \"Conformal prediction under covariate shift.\" Advances in neural information processing systems 32 (2019).\n\n[2] Barber, Rina Foygel, et al. \"Conformal prediction beyond exchangeability.\" The Annals of Statistics 51.2 (2023): 816-845.\n\n[3] Podkopaev, Aleksandr, and Aaditya Ramdas. \"Distribution-free uncertainty quantification for classification under label shift.\" Uncertainty in artificial intelligence. PMLR, 2021.\n\n[4] Volkhonskiy, Denis, et al. \"Inductive conformal martingales for change-point detection.\" Conformal and Probabilistic Prediction and Applications. PMLR, 2017.\n\n[5] Eliades, Charalambos, and Harris Papadopoulos. \"A conformal martingales ensemble approach for addressing concept drift.\" Conformal and Probabilistic Prediction with Applications. PMLR, 2023.\n\n[6] Eliades, Charalambos, and Harris Papadopoulos. \"A betting function for addressing concept drift with conformal martingales.\" Conformal and Probabilistic Prediction with Applications. PMLR, 2022.\n\n[7] Bar, Yarin, Shalev Shaer, and Yaniv Romano. \"Protected test-time adaptation via online entropy matching: A betting approach.\" Advances in Neural Information Processing Systems 37 (2024): 85467-85499.\n\n[8] Shekhar, Shubhanshu, and Aaditya Ramdas. \"Sequential changepoint detection via backward confidence sequences.\" International Conference on Machine Learning. PMLR, 2023.\n\n[9] Shin, Jaehyeok, Aaditya Ramdas, and Alessandro Rinaldo. \"E-detectors: A Nonparametric Framework for Sequential Change Detection.\" The New England Journal of Statistics in Data Science 2.2 (2023): 229-260.", "relation_to_broader_scientific_literature": "Even though the general exposition and background in the paper is very nice, I believe the method is not clearly put into context to other related approaches and existing methods. I've provided some references above. One could also think about discussing other ways of combining conformal p-values (beyond within a test martingale framework), e.g. see [1]\n\n[1] Vovk, Vladimir, Bin Wang, and Ruodu Wang. \"Admissible ways of merging p-values under arbitrary dependence.\" The Annals of Statistics 50.1 (2022): 351-375.", "essential_references_not_discussed": "I've given what I consider relevant references missing or not clearly discussed above.", "other_strengths_and_weaknesses": "- I liked the motivation in the introduction, the clarity of Fig. 1 to help the reader understand the general problem, and the nice background/exposition in section 2 at the appropriate level of depth. I also enjoyed the connection to the permutation interpretation of weighted p-values in sec 3.1 and 3.2.\n- If Eq. 13 ends up being the single hypothesis considered and experimentally examined even though the approach is motivated from a more general hypothesis class perspective $H_0(f)$, then it seems misleading to call sec 3.3. an \"Example\" unless more are given. Perhaps it should be emphasized that this is the primary testing objective.", "other_comments_or_suggestions": "- The paper could use a substantial proof-read since there are quite a few typos (e.g. some noted are L88, L186, L171, L301, L302). \n- Table 1 is confusing to read since there is a lot of bolding. Perhaps one can omit the blue bolding of the own method.\n- The plots in general are too small, and figure legends etc. are only readable if zoomed in. I would suggest a proper overhaul for readability there.\n- A clear outline of contributions (e.g. bullet points) in the introduction would be nice to have\n- In L140 what is $[0,1]^*$? \n- In Eq. 7 the $\\forall t$ seems incorrect here, rather its for a given fixed $t$", "questions_for_authors": "- Please see questions in the experiments section that I would like to be addressed/clarified. \n- Can you clarify the interpretation given in L258-L264 for Eq. 13? To me this seems like a test for covariate shift and not the approximation quality of $\\hat{f}^T(X) / f(X)$? Also, does the estimation of only $\\hat{f}^T(X)$ still subsume a known $f(X)$, or is it supposed to symbolize an estimated *ratio* rather than estimated shifted covariate distribution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}]

Author rebuttal:

Meta-review: The paper proposes a novel approach called WATCH, which is based on a weighted generalization of conformal test martingales, which automatically adapts to mild data shifts while detecting substantial shifts and raising alarms. Compared to conformal test martingales with uniform weight, the novel, generalized version can test a broader range of null hypotheses beyond exchangeability. Specifically, authors show that in the case where the estimated density ratio function is used as the weight, the method can detect either an *extreme* covariate shift or concept shift, while under moderate covariate shift, the method is able to adequately adapt its conformal prediction interval. The method ensures control over the false alarm rate. Theoretical claims are supported by experiments.

Overall, reviewers agree on the novelty of the proposed approach. However, the paper could be strengthened by clarifying:
-the exact distinction between ‘benign’, ‘medium’, and ‘harmful’ covariate shifts (based on the WCTM values)
-novelty of the proposed ‘weighted-conformal-p-values’
-details on the experiments (including metrics, experiment parameters, interpretation of the results)
Decision: Accept (poster)