When to retrain a machine learning model
Authors: Florence Regol
Leo Schwinn
Kyle Sprague
Mark Coates
Thomas Markovich
Venue: ICML 2025 poster (2025)
Keywords: retraining;sequence modeling; forecasting performance
Primary area: general_machine_learning->everything_else

Abstract: A significant challenge in maintaining real-world machine learning models is responding to the continuous and unpredictable evolution of data. Most practitioners are faced with the difficult question: when should I retrain or update my machine learning model? This seemingly straightforward problem is particularly challenging for three reasons: 1) decisions must be made based on very limited information - we usually have access to only a few examples, 2) the nature, extent, and impact of the distribution shift are unknown, and 3) it involves specifying a cost ratio between retraining and poor performance, which can be hard to characterize. Existing works address certain aspects of this problem, but none offer a comprehensive solution. Distribution shift detection falls short as it cannot account for the cost trade-off; the scarcity of the data, paired with its unusual structure, makes it a poor fit for existing offline reinforcement learning methods, and the online learning formulation overlooks key practical considerations. To address this, we present a principled formulation of the retraining problem and propose an uncertainty-based method that makes decisions by continually forecasting the evolution of model performance evaluated with a bounded metric. Our experiments, addressing classification tasks, show that the method consistently outperforms existing baselines on 7 datasets. We thoroughly assess its robustness to varying cost trade-off values and mis-specified cost trade-offs.

When to retrain a machine learning model
Florence Regol 12
Leo Schwinn
Kyle Sprague? Mark Coates Thomas Markovich
Abstract
the form of missed opportunity. However, retraining a new model, for, on more up-to-date dataset, D', is also costly. Beyond the obvious costs of computational resources and energy (Strubell et al., 2020), there are human resource costs associated with assigning experts to deploy and maintain the model, as well as collecting and cleaning data. Deploy- ing a new model also generally comes with a higher risk. Therefore, the optimal retraining schedule depends on this comprehensive cost of retraining, on the cost of making mis- takes, and on future model performance. Figure I provides visualization of the task.
A significant challenge in maintaining real-world machine learning models is responding to the con- tinuous and unpredictable evolution of data. Most practitioners are faced with the difficult question: when should I retrain or update my machine learn- ing model? This seemingly straightforward prob- lem is particularly challenging for three reasons: 1) decisions must be made based on very limited information - we usually have access to only a few examples, 2) the nature, extent, and impact of the distribution shift are unknown, and 3) it involves specifying a cost ratio between retrain- ing and poor performance, which can be hard to characterize. Existing works address certain aspects of this problem, but none offer a compre- hensive solution. Distribution shift detection falls short as it cannot account for the cost trade-off; the searcity of the data, paired with its unusual structure, makes a poor fit for existing offline reinforcement learning methods, and the online learning formulation overlooks key practical con- siderations. To address this, we present a princi- pled formulation of the retraining problem and method that makes
Although this retraining problem is ubiquitous in indus- try (Gama et al., 2014), there are few works in the machine learning literature that tackle it directly. It has been framed as an application of the distribution shift detection prob- lem (Bifet & Gavaldå, 2007), where the conventional strat- egy involves triggering retraining whenever substantial shift is detected (Bifet & Gavaldà, 2007: Cerqueira et al., 2021; Pesaranghader & Viktor, 2016). However, this ap- proach overlooks retraining costs. This can be particularly problematic when training expensive, as demonstrated in our experiments. Others have reduced the need for retraining by incorporating robustness to distribution shifts (Schwinn et al., 2022) adapting to them (Filos et al., 2020), but these methods have limits on the extent of the shift they can handle. Other related areas include online, adaptive, life-long, and transfer learning (Hoi et al., 2021), which aim to update models to new or evolving data distributions. However, these methods are primarily concerned with max- imal model performance, while the goal of our work is to explicitly minimize the overall cumulative cost. In particu- lar, continual learning approaches and the like cannot delay updates due to future cost considerations. Moreover, in practice, the cost of retraining can go beyond the number of gradient updates or sample complexity, as discussed above. Finally, because this a sequential decision problem, can be framed within the offline reinforcement learning frame- work (Levine al., 2020). In theory, offline RL methods should be applicable, but few, if any, are designed for very low-data settings. They require substantial amounts of data for training and hyperparameter tuning, and are therefore largely unsuitable to use in this context.
propose an uncertainty- decisions by continually forecasting the evolution of model performance evaluated with a bounded metric. Our experiments addressing classification tasks show that the method consistently outper- forms existing baselines on 7 datasets.
1. Introduction
In many industrial machine learning settings, data are con- tinuously arriving and evolving (Gama et al., 2014). This means that a model, fe. that was trained on a fixed dataset, D, will become outdated. This usually translates to a cost in
'MeGill University, Canada Block, Toronto, Canada TTchhncal University of Munich, Germany. Correspondence to: Florence Regol <fforencerr @ww.@bbbooo.o.......
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
A direct treatment of the cost consideration the retraining problem is presented by Žliobaité al. (2015) and by Ma-

---

When to retrain a machine learning model
0.94
retrain
retrain
0.92
0.90
Accuracy
0.88
model trriined on dataset:
De
7)
17g
0.84
D
D4
Dj
D10
0.82
D2
-
D3
Dg
retraining schedule
Do D1
D2
D3
D4
Dg Evaluated on dataset
D3
D7
Da Dg D10
Figure 1. The Retraining Problem: The performance of a model trained on a dataset D; gradually decreases when evaluated on more recent datasets in the presence of distribution shift. The task is to determine when retraining is beneficial compared to keeping an older model. We must take into consideration the trade-off between potential accuracy gains and the costs associated with retraining. In the training schedule 0 shown here, retraining occurs twice, at 4 and t
hadevan & Mathioudakis (2024). The formulation by Ma- hadevan & Mathioudakis (2024) accounts for the trade-off between the cost of retraining and the cost of performance. Their method, CARA, relies on approximating the perfor- mance of model on new data, and the retraining decision is based on this value. However, this approach makes several limiting assumptions: I) the relative cost objective assumes that the "difficulty" of the task remains constant; and 2) the performance approximation assumes the data distribution almost stationary. Instead, we consider a more general objective that combines both the retraining cost and the av- erage performance over a specified horizon. We detail the relationship between our objective and CARA's objective in Appendix A.12. Our formulation is more general and does not depend on strong assumptions regarding the data distribution and its impact on performance. Additionally, our method can leverage new observations of the model's performance. Our proposed method involves forecasting the performance of both future and current models and making decisions based the uncertainty of our predictions. There is no constraint on how the "retrained" model is obtained. It can be fine-tuned from previous model, trained from scratch, or derived using any other procedure. We show the effectiveness of our approach on five real datasets and two synthetic datasets. We make the following contributions:
on performance forecasting: UPF. Our proposed algo- rithm outperforms existing baselines. It requires minimal data by fully leveraging the structure specific to the retrain- ing problem, employing compact regression models, and balancing the uncertainty caused by data scarcity through uncertainty-informee decision process.
2. Related Work
We discuss related work and fields relevant to the retraining problem. A more detailed literature review, including con- nections other related fields is provided in Appendix A.3.
Retraining problem Few works explicitly target the re- training problem. Zliobaité et al. (2015) propose a return investment (ROI) framework to monitor and assess the retraining decision process, but do not introduce a method for actually deciding when to retrain. Mahadevan & Math- ioudakis (2024) develop a retraining decision algorithm, CARA, which integrates the cost of retraining and intro- duces a "staleness cost" for persisting with old model. CARA approximates the staleness cost using offline data consisting of several trained models and their historical performance. Three versions of CARA are proposed: (i) retraining if the estimated staleness exceeds a threshold; (ii) retraining based on estimated cumulative staleness; or (iii) identifying an optimal retraining frequency. While provid- ing promising results, CARA requires access some of the data that will be used for retraining, and is very computa- tionally intensive, so there is no adaptation to data obtained during the online decision period. Hoffman et al. (2025) address a related problem: deciding whether to retain the current model (i.e., no retraining). fully retrain it, or refine via finetuning. The authors formulate an objective that balances retraining cost, the impact of concept drift (ambi-
We introduce a principled formulation of a practical ver- sion of the retraining problem. We explain connections to existing formulations and offline reinforcement learning.
We establish upper limits the optimal number of trains based performance bounds which can be used to determine whether you should consider retraining not.
We propose a novel retraining decision procedure based
2

---

When to retrain a machine learning model
guity), the uncertainty associated with each option (risk), and the expected performance.
sequence of distributions D, Pt- In practice, this re- flects the gradual distribution shifts that occur when col- lecting data over time, we specifically cannot assume that pt Pt+1 (this would correspond to special case of the problem, which we refer to as the no distribution shift case). The datasets are acquired at discrete times = [-w.... o,... T]. The sequence is split into an of- fline period that spans = [-w,.. .0. followed by an online period [t 1.... T... At each time step & of the online pe- riod, we are given the option to (re)train model ft, using the data acquired up until time I, for retraining cost of et. Datasets and trained models can be formed and obtained through any means depending on the task; for example, fi could be fine-tuned from fo and Dj could contain Dp-
Distribution shift detection The retraining problem is closely connected to distribution shift detection and mit- igation (Wang et al., 2024a; Hendrycks & Gimpel, 2017; Rabanser et al., 2019). Some approaches decide to adapt a model after detection of a changed distribution (Sugiyama & Kawanabe, 2012; Zhang et al.. 2023). Since the sig- nal for these methods is designed to adapt model rather than trigger a full retraining, they are not appropriate to be used as full retraining signals. Other approaches, how- ever, directly treat the detection of a distribution shift as a cue for retraining. ADWIN (Bifet & Gavaldå, 2007) uses
statistical testing of the label or feature distribution. An- other approach to directly monitor the model's perfor- mance. FHDDM (Pesaranghader & Viktor, 2016) employs Hoeffding's 's inequality, while Raab et al. (2020) propose method that relies on Kolmogorov-Smimoo Windowing test. These approaches work well with low retraining costs, but perform poorly when retraining costs are high, as they tend to recommend retraining far too often. Additionally, they lack adaptability varying costs, and it is difficult to determine the correct significance level to use for given toreriinggt---effommcccc cost ratio.
The complete sequence of decisions can be encoded as a binary vector 0 E {0, 1)"", where 8, indicates that we retrain the model at time I. We introduce re (1) as a mapping function that returns, at time t, the last training time, i.e., re(t) maxxee(....(.......... l', or (t) 0 if ||]|| = 0.).
At each time step t, we are required to generate a certain number of predictions N₂ on a test set, which incurs a loss (ū. scaled by a cost e,. This would correspond actu- ally using the model to make predictions, for example, to detect fraud - failing to detect fraudulent transaction costs e₁ and approximately N, transactions are verified at time L. To make these predictions at time I, we use the most re- cently trained model, which we denote by fro(t). To ensure that there is always at least one model available during the online period, we always train the last offline model fo-
Offline reinforcement learning Lastly, we discuss con- nections to offline reinforcement learning (ORL), where an agent must Iearn a policy from a fixed dataset of rewards, actions, and states. This subset of RL is challenging, as the agent cannot explore and must rely on the dataset to infer underlying dynamics and handle distribution shifts. Levine et al. (2020) provide an extensive review. Q-learning and value function methods, which focus on predicting future action costs, have become the preferred approaches (Levine et al., 2020; Kalashnikov et al.; Hejna et al., 2023; Kostrikov et al., 2022). Some methods incorporate epistemic uncer- tainty into the Q-function address distribution shifts of unseen actions (Kumar et al., 2020; Luis et al., 2023).
The target cost is a function of O which encodes the retrain- ing decisions, and combines two costs: the cost associated with model performance, Σ._ Ct EN (EIo)) and the cost to retrain, 0,c;:
T
N e
(o E
Σ
Σ
(Foco(ow)))) Occo (1)
If we view the states as encoding both time and the model in use, and actions as either retraining or maintaining the current model, we can frame our problem as ORL. However, most existing RL approaches focus on scaling to large state or action spaces, employ large models, and assume access to abundant data, making them unsuitable for our context. A more detailed discussion on the connections and limitations of ORI. methods is included in Appendix A.
To make the expression more concise, we condense the ex- pected loss into a scalar pei. where the two indices denotes the model index, and the timestep, respectively:
ED, ((F(())))) 0.
ifi otherwise.
pei,
(2)
We can simplify the problem by assuming fixed cost of retraining, C, c, cost of loss, e = e, and number of predic- tions, N₂- = N. The solutions we develop are easily extended to the case where these are varying, but known, quantities. Introducing a cost-to-pefformance ratio parameter a = aN we can compactly write the online objective as:
3. Problem Setting
We outline our formulation of the retraining problem. We have access to a sequence of datasets, D-wy.... Do.... ... Dr with features and labels Fi,t Xx, yi.t Y,, D, (T..,, Br,))) which are assumed to be drawn from
e
T Ca( = eN (al10)) [ PPerOCos)) 1-1
(3)
3

---

When to retrain a machine learning model
3.1. Offline and Online data
horizon to consider, T. Then if ( 1, no retrain- ing should be performed. We demonstrate how this result should be used in practice in Appendix A.4.I.
The cost C. (0) is only evaluated over the online period. We assume that we have access to all the datasets and trained models during the offline period. In practice, the number of models and datasets is typically limited to only few (around 10 to 20 at most), which is why we characterize this problem as being in a low-data regime. We denote this data as Ioffline (D-w1... ... f-w....... fo). In the online mode, each decision at time t can only rely on information available prior to that time, which therefore contains both the offline data Iofline and the online data that was collected up to the timestep t: Ionline. The online data is similar to the offline data, but it only contains the models that were actually trained; Tonline (DI, D {fa)s s.t. 9.-1)).
Bounding L General bounds for L are too loose to be helpful; however, in some cases, reasonable estimates can be derived. For the specific case of "no distribu- tion shift" IID data, where the data simply accumulates (D, 1 De ~ (t)) we can leverage known theoret- ical results, such as Probably Approximately Correct (PAC) learning theory (Valiant, 1984) or Rademacher Complex- ity (Bartlett& & Mendelson, 2002). For large-scale training settings, precise empirical scaling laws are available (Ka- plan et al.; Hoffmann et al., 2024). Kaplan et al. derive that the loss L of the neural network scales with respect to dataset size N as L (N/5.4- 10 -0.095 Such scaling laws enable accurate estimation of expected performance improvements from expanded datasets L. Thus, they enable informed decisions about when retraining can yield substan- tial benefits. For more discussion see Appendix A.5. Even in real-world applications, where data often exhibit tem- poral or spatial dependencies, making the non- -inddistribttion shift III) assumption unrealistic, bounds have been derived using stability analysis or tailored Rademacher complexity bounds (Mohri & Rostamizadeh, 2008; 2007; 2010).
denote by Let Ict
Each entry of can therefore be modeled by a binary func- tion (tt, E {0,1):
0 [,0 ....... )(.
(4)
Given Ct2 Gt2 and Nr, the task is to determine the g that gen- erates the retraining schedule 0" minimizing cost (0):
0" arg min Ca (0))
(5)
0c(0,1)
3.2. Some analysis
Before introducing methods that learn to generate such schedule 0, we characterize basic properties of the problem. Specifically, we establish bounds on the number of retrain- ing actions of the optimal solution. These can be used to determine whether we even need to consider retraining. We also provide guidance leveraging existing performance bounds (such scaling laws) to compute the relevant quan- tities in these bounds. These theoretical insights can be used to derive practical -flls-of-thumb on a case-by-case basis.
4. Methodology
A retraining decision algorithm must specify the decision functions Ittt, e {0, 1) (where & contains the algo- rithm's 's parameters) used to build the decision vector O. To make perfect decisions, we would need future performance values, i.e., pe,,, (i > t or j > t). This is infeasible; how- ever, we assume that there is a temporal correlation between the performances of different models trained at different times, which we aim to exploit to build a predictive model. We therefore propose to 1) model these future values as random variables and learn their distributions; and 2) base our decisions on the predicted distributions to construct our method, the Uncertainty- Ticertainty-Pefformaanc Forecaster (UPF).
Our upper bound depends on the difference between the ex- pected performance of a model trained on dataset D, and the performance of a model trained on the subsequent dataset Di+1, evaluated on the same dataset from any timestep D,:
L 2 peec 1, V [T]
(6)
4.1. Future Performance Forecaster
Given this quantity, we derive the following result of an up- per bound for the number of retrains of the optimal solution, which we denote by r" 110"|11:
The first component of our algorithm involves learning a performance predictor to forecast unknown entries in pe, which are defined as pei, = En, [X,)) Y2)] for i s j (see Eqn 2). In a classification setting where we consider the 0-1 loss ((g', y) = 1[ * y). these are accuracy. We introduce random variables Aij and model the entries peij as realizations of these. Although this prediction task may initially seem similar to the performance estimation (PE) problem (Garg et al., 2020), it is fundamentally different. PE focuses on estimating the performance of an existing model under distribution shift, whereas our task involves
Proposition 3.1. Given that L > horizon T E N, and a relative cost of retrain a, the number of retrains of the solution to Equation 5 r** = ||0"|| satisfies:
Vt E [T]. a
a
S
VL
The proof is provided in Appendix A.4. Suppose a prac- titioner has reasonable approximations of I. and a, and a
4

---

When to retrain a machine learning model
forecasting the future performance of models that do not yet exist. Crucially, PE lacks temporal dimension, it does not account for the evolution of models over time.
as linear regression. Once the parameters are learned, we can recover the corresponding 0gg (ra,)) (, (rag) parame- ters to obtain our predictive distribution (see Appendix A.7 for additional details);
Since the As,j random variables are bounded, we model them (after appropriate scaling) as Beta distributed with pa- rameters (Ir,,), (T(..) that depend on some input feature Ft.j. These features contain information about the current state of the feature distribution as well as the timestamp (see the section Inputs in Appendix A.7 for full details). This forecasting formulation allows us to capture both covariate and concept drift. This choice of the Beta distribution is particularly appropriate when the performance metrics A are accuracies, as in our experiment, since accuracy can be interpreted as the sum of Bernoulli random variables. Of course, other distributions could also be considered. We also define their associated mean (L(r,,)) and variance o (Fa.g). Given the parameters a a(r,)) ), (r..), we model the ran- dom variables to be independent of each other:
(o().)) = Beta(a,(T..),
(14)
As stated, this parameterization is appropriate for bounded losses. Other distributions can be used to model different loss domains (see Appendix A.8). As Tea grows at each time step, our training data increases, so we retrain and obtain a new (A(,..)) each time. As constructed, past decisions influence the dataset Iouline available for the next iteration, but this effect is ignored by the algorithm. Empirically, we find that the algorithm performs well despite this. One di- rection worth investigating is the incorporation of random decisions to allow the predictor to learn over a broader re- gion of actions and responses. As our methodology involves forecasting future performance as a key subtask, we evaluate and quantify the impact of success in this task on the overall performance of our algorithm, as detailed in Appendix A.7.
(8)
II Beta((((s.), 153
(9)
4.2. Decisions under uncertainty
where Beta() denotes the pdf of a Beta distribution. We choose the input features Fty to include the indices of the training and evaluation datasets (i and j. respectively), along with additional features that capture the gap between the training and evaluation timesteps (the difference j -, and summary statistics of the distribution shift Fshafe (see Ap- pendix A.7 for details). The input features are thus given by Kij = [i.j.j- i, 2shist]-
Now we describe how we use (o(A..) to decide whether to retrain. We introduce random variable C' that represents the total cost (Eqn. 3) (given a sequence of decisions A):
I Ō(0)= eN C (110)) + Σ Anooo)) 1-1
(15)
We can therefore define our decision rule based on this random cost using our learned distribution of performances P& (A.,). Given the past decisions Oct, our next decision 0, is obtained by comparing the 5-level quantiles of the total cost incurred if we retrain, denoted by Co.,Iretrain and the cost incurred if we do not, denoted by Ca., Using (o) as the quantile function of random variable, our rule is given by:
From the offline data, we have access to observations anj ~ Arj. and can build a regression dataset learn the We specify the learning task by constructing (Tn.. 01,)) pairs:
parameters a (ri,)),
Ma = ( (n. VI: E VD, Iec).
(10)
Direct learning of the a parameters can be unstable. Therefore, we use a Gaussian approximation:
0. = 1 Iretrain (6)
Co., keep))))))))))))) (16)
Beta(a(ru,)) ((r,.) & *N ((u(r.)),
(11)
The quantile parameter ō allows us to control how conser- vative we are. Lower values of å lead to decisions that prioritize costs with lower variance, while setting 8 0.5 simply selects the decision that minimizes the median total cost. As defined, the retraining decision Q is deterministie.
This allows to write the likelihood of our dataset as:
((Mct:)) II P(ac.)lra..))
(12)
i.eMMe
=I Massiootrsss iJeMa
(13)
We begin by giving explicit expressions for the conditional random variables Co_,, Ietrainn and Ceep If we de- cide to retrain at time step f. the incurred costs include the retraining cost a, the performance cost of the most recent model Ar,t, and future costs for the decisions we will make. Specifically, we incur Coan Iretrain if the next decision is to retrain, and keep if it is not. If we choose not
We parameterize the variance as constant a (ra.)) a Maximizing the likelihood w.r.t. to the mean parameters H4 (T.,)) then becomes a standard mean square error min- imization. Given the expectation of operating in a very dow- regime, we rely on simple inference models, such
5

---

When to retrain a machine learning model
to retrain and keep the current model, we only incur the performance cost of the old model, (t-()-)) These random variables can therefore be recursively defined:
5. Experiments
Evaluation Metrics The performance of a retraining de- cision method evaluated based on both the average perfor- mance and the total retraining cost. The tradeoff between these factors is controlled by a. When using the zero-one loss in classification, a can be seen as the ratio of retrain- ing cost to the cost of misiiassiiisations. In practice, is application-dependent and should be set by the practitioner. The retraining cost would be low (small a:) for situations such as fine-tuning small models. By contrast, when re- training large language models, or in high-stakes settings requiring extensive validation, the retraining cost is high (large a). The retraining decision method should be robust across all scenarios. The appropriate value of a can be very difficult to estimate and will likely be an approximation in practice. Consequently, we present experiments that test the robustness of the method to inaccuracies in a Section A.2.
Iretrain a A1, + 0 Coa.
(17)
+ (1 (1-0pp 1-0 (cx1)Coc..))) Co kkep
(18)
T
a Art +
Σ
Argcos. t00
(19)
p-t-1
I
Čou,keeep= A
Σ t=++1
Angen) ab. (20)
As shown, the cost random variables are constructed recur- sively by summing the distribution of the cost of perfor- mances Aig that would be selected by the decision rule O, as 1 and the a parameter are both deterministic.
The decision rule introduced in Eqn. 16 can therefore be written as:
In our experiments, we address classification tasks with zero-one loss, and set e.V = 1. We report empirical estimate of the target cost (0) (Eqn. 3), obtained from the test set, over varying a:
0, 1[ IlF a+ r (0)
(21)
F A A Arges, (0)].
(22)
(o) x C. (0) al|0||| +
test ro())e) [=1
pe
(24)
We use the learned Beta distributions, introduced in the previous section, plugging them into Eqn. 22 in order to make a retraining decision.
where pe.. = 1 acctest with ((y 1[y # v'), To summarize the results at multiple a operating points, we report the under-theecuuneee
(AUC) of (0). We com- pute 10 a operating points and we allow a to range from 0 (no retrain cost) to amaz (where the cost is too high to jus- tify any retraining). The upper bound, O'maas is determined by the a value which the oracle reaches 0 retrains.' The oracle is obtained by determining the optimal schedule that minimizes the target cost, assuming exact knowledge of all future peij entries, i.e., goracie = arg min, Ca( (0).
If the parameterization (A.,) does not lead to closed form expression, we use Monte Carlo methods to obtain quantile estimates:
Fed Ieetrriin (6) Irettran (ō)
F-11
(23)
where Coo Ietrrain (0) is obtained through bootstrapping.
Connection to offline reinforcement learning The for- mulation closely resembles a Q-lcarning formulation. The C' values defined in Eqns. 18- 20 strongly align with Q functions. Indeed, one possible approach is to bypass the learning of the pe and directly optimize the decision-making process using Q-learning approaches. The problem we are considering can be viewed as a corner case of offline RL, where the state space is finite and enumerable, the training data are extremely limited and the transition function is deterministic and fully known. In fact, our methodology can be reinterpreted as an offline variant of a Q-learning approach with a specific parameterization of the Q function, further justifying the motivation behind our method. We explore and formalize this connection in Appendix A. However, as we have explained in the related work section, existing ORL methods are not suitable for this setting. We provide the results for one ORL baseline in Appendix A. to exemplify that point.
Datasets We present results on synthetic and real datasets. For the real datasets, we use datasets with a timestamp for each sample and partition the data in time to create se- quence of datasets Do, Di,.... For each trial, we sample different sequence of length T within the complete dataset sequence available. We report results on: (i) the electricity dataset (?), binary classification task predicting the rise or fall of electricity prices in New South Wales, Aus- tralia; (ii) the airplane dataset (Gomes et al., 2017), which records whether flight is delayed; (iii) yelpCHI (Dou et al., 2020), which classifies if a user's review is legitimate; and (iv) epicgames (Ozmen al., 2024), where the task is to
'The use of the oracle to define the range of a values for the AUC computation does not bias the performance assessment via pollution with future knowledge. None of the algorithms makes use of the oracle information. Using the oracle merely ensures that the performance comparison is conducted over the range of relevant a.
6

---

When to retrain a machine learning model
predict whether author's critique of game selected top critique. As base model use XGBoost (Chen & Guestrin, 2016).
3.5
3.0
We also present a larger vision dataset that requires a larger network to process. iWildCam (Beery et al., 2020) con- sists of images of animals in the wilderness, captured at various locations, and the task involves multi-class animal classification. Our approach uses a pretrained vision model, augmented with linear layer that processes the image rep- resentation along with the location domain to produce the final classification output. We allow for a different pre- trained architecture model at each timestep t, and perform a random search over a set 188 choices from the Hug- gingface library (Wightman, 2019). These encompass a wide variety of networks, including ViT (Dosovitskiy al., 2021), ResNeT (He al., 2016) and CNNs (O's Shea & Nash, 2015). Appendix A.6 provides additional details on the architecture, training procedure, and hyperparameter search. For the synthetic dataset, we follow Mahadevan Mathioudakis (2024) to generate two 2D datasets with covariate shift (Gauss) and concept drift (circles) (Pesarang- hader et al., 2016). Appendix A.6 contains details on the generation. We report trials for iWild and 10 trials for the other datasets.
2.5
2.0
1.5
0.0
0.2
0.4
0.6
0.8
1.0
strategy
UPF
num. retrains
CARA cumul.
ADWIN-5%
4
ADWIN-50%
oracle
0.0
0.2
0.4
0.6 a
0.8
1.0
Figure 2. Results the electricity dataset. Top) Cost C. a. Bottom) Number of retrains vs a. In the top figure, we can see that UPF consistently reaches low Ca(0) across diffcrent In the bottom figure, the number of retrains of UPF follows the optimal baseline more closely.
Baselines and algorithm settings We set the confidence threshold of our UPF algorithm to 8 95%, as it is stan- dard value used for confidence intervals. For U (Ti,), we use a linear regression model, ElasticNerCV (Zou & Hastie, 2005), from the scikit-learn library. All other optimization parameters are set default choices from the scikit learn libraries. We report results on shift detection baselines and the three variants of the CARA baseline, well the oracle.
formance; and the periodic strategy, CARA per., searches for the best retraining frequency. Appendix A. provides additional details the CARA baseline.
Appendix A.11 includes results for an offline RL baseline.
For the distribution shift detection baselines, we set the window size to the size an individual dataset (D), and re- train when the algorithm detects a distribution shift. (Then we reset the algorithm with the dataset of the last retrained model.) As these methods cannot take into account the vary the significance level threshold
6. Results
We start by presenting in Table 1 the area-under-the under-the-curve (AUC) of the total cost value (f). The AUC is computed as the area over range of a values determined by the oracle performance. Lower values of AUC are better because we aim to reduce the cost over the operating range. Overall, our proposed method achieves the best trade-off between the number retrains and average accuracy across baselines and datasets. To gain better insight into the behavior the di different algorithms and how they are impacted by varying retraining cost parameters, we provide detailed overview for one dataset with two values of a: one where the cost of retraining is low and one where it is high, as shown in Table 2. Figure 2 depicts how the the total cost (+) and the number of retrains vary as a is changed. Appendix A.10 contains the complete set of results and figures.
cost of retraining, 6 to obtain different frequencies of retraining. We include A (Bifet & Gavaldå, 2007), which is based on statistical testing of the label distribution, FHDDM-6 (Pe- saranghader & Viktor, 2016), which based on Hoeffding's inequality, and KSWIN- (Raab et al., 2020), which is based on the Kolmogorov-Smingg test.
CARA (Mahadevan & Mathioudakis, 2024) searches for the best strategy with fixed parameters using the offline data. The standard version, CARA, searches for the best thresh- old of approximate performance and retrains when it drops below it. The cumulative version, CARA cumul,, searches for the best threshold of the cumulated approximate per-

---

When to retrain a machine learning model
Table 1. AUC of the combined performanccc/ierraiiing cost metric C. (0), computed over range of values, for all datasets. The bolded entries represent the best, and the underlined entries indicate the second best. The denotes statistically significant difference with respect the next best baseline, evaluated using Wilcoxon test at the 5% significance level.
electricity
Gauss
circles
airplanes
yelpCHI
epicgames
iWild
ADWIN-550 55%
2.8099
0.4533
0.0753
2.6353
0.1298
0,3217
3.7371
ADWIN-505%
2.8131
0.4848
0.0753
2.7147
0.1298
0,3238
4,2564
KSWIN-590
3.8979
0.3975
0.0753
3.2300
0.1322
0.3420
4.4268
KSWIN-50%
4.0521
0,9530
0.0794
3.2042
0.1655
0.3537
4.4268
FHDDM-5%
3.1525
0.3893
0.0753
2.6577
0.1324
0.3298
4.4267
FHIDM-
3.4037
0.5918
0.0772
2.7077
0.1450
0,3389
4,4268
CARA cumul.
2.7147
0.3862
0.0731
2.2900
0.1299
0.3228
3,8922
CARA per.
2.8986
0.4678
0.0800
2.4061
0.1318
0.3260
3,7527
CARA
2.7198
0,3841
0.0726
2.2753*
0.1294
0.3202
3,9506
UPF (ours)
2,5782
0.3829*
0.0668
2.2865
0.1293#
0.3189
3.0498
oracle
2.4217
0.3724
0.0627
2.2298
0.1275
0.3170
2,4973
Table 2. Wc compare the best performing algorithms for the electricity datasct with the optimal dccisions (the oracle) in both high and low retraining cost settings. For each baseline, we report the number of retrains and average accuracy, as well as our primary metric Č. (0) that combines both factors using a. The results show that the proposed method achieves the best C,(O)) value and closely approximates the oracle's 's behavior both scenarios, highlighted hold.
High retrain cost 0.9
Low retrain cost 0.1
#retrain
Average Ace
(O)
#retrain
Average Acc
Ca(
ADWIN-5%
1.0 = 0.58
0.7 0.04
3.27 + 0.4
1.0 = 0.58
0.7 # 0.04
2.47 + 0.25
ADWIN-50%
1.17 0.38
0.72
3.32
1.17 + 0.38
0
2.39 0.26
CARA
0.0 0.0
0.02
2.78 0.19
0.33 + 0.75
0.66 0.04
2.73
CARA cumul.
0.0 0.0
0.02
0.33 0.19
0.67 0.02
2.68 0.18
CARA per.
1.0
0.02
3.14
1.0
0.09
2.54 0.14
UPF (ours)
= 0.3
0.68 0.04
2.69 0.26
2.5 = 0.67
0.75 0.03
2.24 + 0.17
oracle
0.0 = 0.0
0.66 0.03
2.68 0.26
5.6 1.44
0.83 = 0.02
1.93 : 0.06
First, examining the behavior of the optimal solution (ora- cle), we unsurprisingly observe that the high retraining cost scenario, both the number of retrains and the average accuracy are lower, while in the low retraining cost sce- nario, the number of retrains and the average accuracy are higher. Next, we observe that the proposed UPF method follows the oracle more closely than the other baselines and is more sensitive to the a parameter compared to the cost-aware method (CARA). This is particularly apparent in Figure 2. The CARA baselines relies heavily on its as- sumptions about performance and is therefore not as robust in scenarios where those assumptions do not hold. The de- tection shift methods cannot take the varying parameters as input, so the results remain the same for both values of as Since these methods do not account for retraining costs, they perform better when the cost is very low, as they simply retrain whenever a shift detected. This can be good strategy if retraining costs little. Indeed, we observe that all ADWIN and FHDDM variants are closer to the op- timal values in the low range of a in the left of Figure 2.
However, as the cost of retraining increases, these methods become impractical. Varying the threshold can yield better results-a lower significance requirement (50%) allows for more retraining and therefore works better when retraining costs are low, while the inverse holds in a high-cost regime, where a more conservative retraining strategy is preferable. However, it is not possible to know in advance which sig- nificance threshold should be used for a given a, making these methods largely impractical for such a setting. Finally, we present comparison of the timing complexity of the algorithms in Appendix A.9, where we report the online and offline processes for each method. We observe that UPF is among the lowest-cost methods in both categories.
Ablation study: Importance of uncertainty We perform an ablation study, with results reported in Appendix A. In our approach, we model the distribution of future costs and set targets at the 95% quantile to ensure robustness against noisy predictions. In Appendix A. we compare this uncertainty-aware strategy with deterministic counterpart,
8

---

When to retrain a machine learning model
which does account for uncertainty. The result of the ablation study confirms that accounting for uncertainty does, in fact, enhance robustness and improve performance.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Sensitivity study: Robustness to wrong o: Our approach requires the practitioner to specify the ratio of cost-to- performance, which could be difficult to determine and could be specified incorrectly. In Appendix A.2, we assess the robustness of our method to a wrongly specified and find that performance is not significantly impacted by the misspecification of a.
References
Adams, R. P. and MacKay, D. C. Bayesian online change- point detection, 2007.
Bar-Shalom, G., Geifman, Y., and El-Yaniv, R. Window- based distribution shift detection for deep neural net- works. In Proc. Conf. on Neural Inf. Proces. Syst. (NeurIPS), 2023.
7. Conclusion and limitations
We have proposed a practical formulation of the important problem of model retraining, which has been neglected in the literature, and highlighted its complexity. Our method outlines promising avenue,
Bartlett, P. L. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. J. of Machine Learning Research, 3:463-482, 2002.
experiments have shown that even with distribution shift, it is not unreasonable to expect some patterns in future performance that could be predicted with the help of uncertainty modeling. This data- driven approach is lightweight, practical, and outperforms existing approaches. is robust to varying cost settings misspecified cost-to-
Beery, S., Cole, E.. and Gjoka, A. The iwildcam 2020 competition dataset. arXiv:2004. 2020.
and has demonstrated resilience performance ratios. We have also highlighted the quan- tities of interest to estimate in order to better understand the characteristics of specific problem. While our study demonstrates promising results in predicting optimal retrain- ing schedules, several aspects warrant further exploration. Since our model treats performance prediction as a forecast- ing task, it is primarily suited to gradual changes. Handling abrupt shifts in performance remains an open challenge. On the experimental side, our main experiments investigate setting where the offline dataset (w 7) is non-negligible in size. However, we achieved good performance even with a reduced dataset, which shows that initial training costs can be reduced (see Appendix A. 13).. We evaluated the method individually for each dataset, but future work could further reduce costs by transferring schedulers across datasets and tasks. Additionally, adapting techniques from Hyperparam- eter Optimization could enhance performance forecasting.
Bifet, A. and Gavaldà, R. Learning from time-changing data with adaptive windowing. In Proc. SIAM Int. Conf. on Data Mining (SDM), 2007.
Cerqueira, V., Gomes, H. M., Bifet, A., and Torgo, L. STUDD: a student-teache method for unsupervised con- cept drift detection. J. of Machine Learning Research, 1 0144351-4378 2021.
Chebotar, Y., Vuong, Q. Hausman, K., Xia, F,, Lu, Y., Irpan, A., Kumar, A., Yu, T., Herzog, A., Pertsch, K., Gopalakrishnan, K., Ibarz, J., Nachum, O., Sontakke, S. A., Salazar, G., Tran, H. T., Peralta, J., Tan, C., Man- junath, D., Singh, J., Zitkovich, B., Jackson, T., Rao, K., Finn, C., and Levine, S. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Proc. Conf. on Robot Learning, 2023.
Chen M., Goel K., Sohoni*, N., Poms, F., Fatahalian, K., and Re, C. Mandoline: Model evaluation under dis- tribution shift. In Proc. Int. Conf. of Machine Learning (ICML), 2021.
Acknowledgements
This research was partially funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), [reference number 260250]. Cette recherche a été partielle- ment financée par le Conseil de recherches en sciences naturelles et en génie du Canada (CRSNG), [numéro de référence 260250J. Ce projet de recherche n" 324302 est rendu possible grâce au financement du Fonds de recherche du Québec.
Chen, T. and Guestrin, C. XGBoost: A scalable tree boost- ing system. In Proc. Int. Conf. on Knowledge Discovery and Data Mining (ACM SIGKDD), 2016
Dai, Z., Yu, H., Low, B. K. H., and Jaillet, P. Bayesian optimization meets Bayesian optimal stopping. In Proc. Int. Conf. Machine Learning (ICML), pp. 1496-1506, 2019.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,
9

---

When to retrain a machine learning model
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. Int. Conf. on Learning Representations (ICLR), 2021.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, W., and Sifre, L. Training compute-optimal large language models. In Proc. Conf. on Neural Inf. Proces. Syst. (NeurIPS), 2024.
Dou, Y., Liu, Z., Sun, L., Deng, Y., Peng, H.,, and Yu, S. Enhancing graph neural network- based fraud detectors against camouflaged fraudsters. In Proc. Int. Conf. Inf. and Knowledge Management (ACM). 2020.
Hoi, C., Sahoo, D., Lu, J., and Zhao, Online learning: A comprehensive survey. Neurocomputing. 459:249 229. 2021.
Fang. T., Lu, N., Niu, G., and Sugiyama, M. Rethinking importance weighting for deep learning under distribu- tion shift. In Proc. Conf. on Neural Inf. Proces. Syst. (NeurIPS), 2020.
Janner, M.. Li, Q., and Levine, S. Offline reinforcement learning one big sequence modeling problem. In Proc. Conf. Neural Inf. Proces. Syst. (NeurIPS), 2021.
Filos, A., Tigkas, P., Mcallister, R., Rhinehart, N., Levine, S., and Gal, Y. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In Proc. Int. Conf. Machine Learning (ICML), 2020.
Kalashnikov, D., Irpan, A., Pastor, P., Iharz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Van- houcke, V., and Levine, S. Scalable deep reinforcement learning for vision- robotic manipulation. In Proc. Conf. on Robot Learning.
Gama, a., Zliobaitundefined, I., Bifet, A., Pechenizkiy, M., and Bouchachia, A. A survey concept drift adaptation. J. ACM Comput. Surv., 46(4), 2014.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv:2001.0838
Garg, S., Wu, Y., Balakrishnan, S., and Lipton, Z. A unified view of label shift estimation. In Proc. Conf. Neurul Inf. Proces. Syst. (NeurIPS), 2020.
Gomes, H. M., Bifet, A., Read, J., Barddal, J. P., Enem- breck, F., Pfahringer, B., Holmes, G., and Abdessalem, T. Adaptive random forests for evolving data stream clas- sification. of Machine Learning Research, 106: 1495, 2017.
Kossen, J., Farquhar, S., Gal, Y., and Rainforth, T. Active testing: sample-efficient model evaluation. Proc. Int. Conf. Machine Learning (ICML), 2021.
Kostrikov, I., Nair, A., and Levine, Offline reinforcement learning with implicit Q-learning. In Proc. Int. Conf. on Learning Representations (ICLR), 2022.
Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., and Schmidt, L. Predicting with confidence on unseen distri- butions. In Proc. Int. Conf. on Computer Vision (ICCV), 2021.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative Q-learning for offline reinforcement learning. In Proc. Conf. on Neural Inf. Proces, Syst. (NeurIPS), 2020.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learn- ing for image recognition. In Proc. Conf. Computer Vision and Pattern Recognition (CVPR), 2016.
Lagoudakis, M. G. and Parr, R. E. Least-squares policy iteration. J. Machine Learning Research, 1:1107-1149,
2003.
Hejna, J., Gao, J., and Sadigh, D. Distance weighted super- vised learning for offline interaction data. In Proc. Int. Conf. Machine Learning (ICML), 2023.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv:2005.01643, 2020.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and auttof-distribution examples in neural networks. Proc. of Int. Conf. Learning Representations (ICML), 2017
Li, A., Boyd, A. J., Smyth, P., and Mandt, S. Detecting and adapting to irregular distribution shifts in bayesian online learning. In Proc. Conf. on Neural Inf. Proces. Syst. (NeurIPS), 2021.
Hoffmman, K., Salerno, S., Leek, J., and McCormick, T. Some models are useful, but for how long?: A decision theoretic approach to choosing when to refit large-scale prediction models. arXiv:2405.1 13926, 2025.
Liu, W., Wang, X., Owens, J., and Li, Y. Energy- based duitr--diiti detection. In Proc. Conf. on Neural Inf. Proces. Syst. (NeurIPS), 2020.
10

---

When to retrain a machine learning model
Luis, C. E., Bottero, A. G., Vinogradska, J., Berkenkamp, F,, and Peters, J. Model-based uncertainty in value functions. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTAT), 2023.
Rakotoarison, H., Adriaensen, S., Mallik, N., Garibov, S., Bergman, E.,, and Hutter, F. In-context freeze-thaw bayesian optimization for hyperparaametr optimization. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
Mahadevan, A. and Mathioudakis, M. Cost-aware retraining for machine learning. Knowledge-Bate Based Systems, 293: 111610, 2024.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita- tion learning and structured prediction to no-regret online learning. Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTAT), 2011.
Mohri, M. and Rostamizadeh, A. Stability bounds for non- iid processes. In Proc. Conf. on Neural Inf. Proces. Syst. (NeurIPS), 2007.
Scheirer, W. J., de Rezende Rocha, A., Sapkota, A., and Boult, T. E. Toward open set recognition. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 35 1D:1757-177,,001
Mohri, M. and Rostamizadeh, A. Rademacher complexity bounds for non-iid processes. In Proc. Conf. Neural Inf. Proces. Syst. (NeurIPS), 2008.
Schmidt, I.,, Santurkar, S., Tsipras, D., Talwar, K., and Madry, A. Adversarially robust generalization requires more data. Proc. Conf. on Neural Inf. Proces. Syst. (NeurIPS), 2018.
Mohri, M. and Rostamizadeh, A. Stability bounds for sta- tionary ---mixing and 3-mixing processes. of Machine Learning Research, 11(2), 2010.
Schwinn, I.., Bungert, I.., Nguyen, A., Raab, R., Pulsmeyer, F. Precup, D., Eskofier, B., and Zanca, D. Improving robustness against real-world and worst-case distribution shifts through decision region quantification. In Proc. Inf. Conf. on Machine Learning (ICML), 2022.
Montiel, J., Read, J., Bifet, A., and Abdessalem, T. Scikit- multiflow: A multi-output streaming framework. J. of Machine Learning Research, 19(72):1-5, 2018.
O' Donoghue, B., Osband, I., Munos, R., and Mnih, The uncertainty Bellman equation and exploration. In Proc. Int. Conf. on Machine Learning (ICML), 2017.
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and de Freitas, N. Taking the human out of the loop: A review of bayesian optimization. Proc. of the IEFE, 104 (1):148-175, 2016.
O' Shea, K. and Nash, R. An introduction to convolutional neural networks. arXiv:1511.0845. 2015.
Ozmen, M., Regol, F., and Markovich, T. Benchmarking edge regression temporal networks. J. Data-centric Machine Learning Research (DMLR), 2024.
Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for modern deep learning research. In Proc. Conf. on Artificial Intelligence (AAAI), 2020.
Pesaranghader, A. and Viktor, H. L. Fast hoeffding drift detection method for evolving data streams. In Proc. Machine Learning and Know. Discovery in Databases (ECML PKDD), 2016.
Sugiyama, M. and Kawanabe, M. Machine Learning in
Non-Stationary Environments: Introduction to Covariate Shift Adaptation. The MIT Press, 2012.
Swersky, K., Snoek, J., and Adams, R. P. Freeze-thaw bayesian optimization. arXiv:1406.3898 2014.
Pesaranghader, A., Viktor, H. L., and Paquet, E. A framework for classification in data streams using multi- strategy learning. Proc. Discovery Science, 2016.
Valiant, L. G. A theory of the learnable. Communications of the ACM, 27(1 17:11):110 1):1 1984.
Pesaranghader, A., Viktor, H., and Paquet, E. Reservoir of diverse adaptive learners and stacking fast hoeffding drift detection methods for evolving data streams. J. Machiche Learning Research,
Wang. H., Vaze, S., and Han, K. Dissecting out-of-
distribution detection and open-set recognition: critical analysis of methods and benchmarks. Int. of Comp. 2024a.
2018.
Vision,
Raab, C., Heusinger, M., and Schleif, F.-M. Reactive soft prototype computing for concept drift streams. Neuro- computing, 416:340-351, 2020.
Wang. W., Fan, Z., and Ng. S. H. Trajectory-bas based multi- objective hyperparameter optimization for model retrain- ing. arXiv:240S. 2024b.
Rabanser, S., Günnemann, S., and Lipton, Z. Failing loudly: An empirical study of methods for detecting dataset shift. In Proc. Conf. on Neural Inf. Proces. Syst. (NeurIPS), 2019.
Wightman, R. Pytorch image models. httts:://itti:i com/ rwi http://gigmm//tttree...ee taet--imageemmeeeeee 2019.

---

When to retrain a machine learning model
Yao, H., Choi, C., Cao, B., Lee, Y., Koh, P. W. W., and Finn, C. Wild-time: A benchmark of in-the-wild distribution shift over time. In Proc. Conf. on Neural Inf. Proces. Syst. (NeurlPS), 2022.
Zhang, Y-J., Zhang, Z.-Y., Zhao, P., and Sugiyama, M. Adapting continuous covariate shift via online density ratio estimation. In Proc. Conf. Neural Inf. Proces. Syst. (NeurIPS), 2023.
Zou, H. and Hastie, Regularization and Variable Selection Via the Elastic Net. J. of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-320, 03 2005.
Žliobaite, I., Budka, M., and Stahl, T. Towards cost- sensitive adaptation: When is it worth updating your predictive model? Neurocomputing, 150:240-249, 2015.
12

---

When to retrain a machine learning model
A. Appendix
A.1. Ablation study The importance of uncertainty
In our approach, we model the distribution of future costs and set targets at the 95% quantile to ensure robustness against noisy predictions. To assess whether this strategy enhances robustness and improves performance, we compare the proposed UPF algorithm, with the 95% quantile, against deterministic version, referred to PF, which selects the predicted decision that minimizes costs. This corresponds setting the quantile 50% in our algorithm (PF = UPF-50%). We observe in Table 3 that relying on conservative quantiles in our predictions results in better overall outcomes, compared to the deterministic version, PF, with statistical significance observed across all datasets except for electricity.
Table Ablation study on accounting for uncertainty in our prediction. Targeting the 95% quantile better overall than the deterministic approach (equivalent to 50% quantile). The denotes statistically significant difference with respect to the next best bascline, evaluated using a Wilcoxon test at the 5% significance level.
electricity
gauss
circles
airplanes
yelp
epicgames
PF
2.5884 = 0.13*
0.3673 = 0.03
0.0697 0.01
2.3688 = 0.35
0.1180 0.00
0.3211 = 0.01
UPF
2.6056 0.14
0.3643 # 0.03*
0.0670 0.01
2.2688 + 0.26
0.1175 0.00*
0.3202 + 0.00*
A.2. Ablation study Robusiness to wrong a
In our setting, we assume that the relative cost of performance and retraining a is known. However, in practice, this tradeoff value can be hard to estimate accurately. value, and to identify the settings where misspecification is the most impactful. In Figure 3, we present how wrongly dataset. Both algorithms
therefore of high practical interest to assess the impact misspecified
specified a values impact the performance of our algorithm and the CARA baseline reasonably robust, as it requires large deviation from the true value (upper right and bottom left) to start seeing degradation of performance of more than 1%. UPF is generally more robust to changes of a. Both algorithms are more susceptible to overestimation of a.
% increase Č(θ)
% increase Č(θ)
0.0
0.0
0.02
provided a
0.02
5
provided
0.04
10
0.04
0.06
0.06
0.08
0.08
0max
ammx
0.0
0.02
0.04
0.06
0.08
dmax
0.0
0.02
0.04
0.06
0.08
Cimax
real
real
Figure 3. Impact of wrong a measured by the percentage increase (0) on the epicgames dataset. left) CARA right) UPF Overall, both methods reasonably robust wrong specification, with UPF being the robust.
A.3, Extended Discussion of Related Work
Retraining problem Few works explicitly target the retraining problem. Žliobaité al. (2015) propose return investment (ROI) framework to monitor and assess the retraining decision process. Mahadevan & Mathioudakis (2024) develop a retraining decision algorithm, CARA, which integrates the cost of retraining into its formulation. It introduces the concept a "staleness cost" which represents the cost of not retraining. The approach involves approximating the staleness cost and optimizing various strategies to reduce the overall cost, based on some offline data. The offline data consist of few trained models, each with an associated dataset that was collected prior to the retraining decision process. Mahadevan
13

---

When to retrain a machine learning model
Mathioudakis (2024) propose three methods: the first retrains when the estimated staleness cost exceeds a threshold; the second tracks the accumulated staleness cost and applies a threshold on that value; and the third searches for the optimal retraining frequency. The staleness cost approximation for using model on dataset relies on the loss of individual known samples. This loss is scaled by the average similarity between the features of these known samples and the features of the dataset of interest. Consequently, it assumes access to the features of some of the samples at a given time hefore deciding to retrain. Moreover, the search for the threshold or the period is computationally intensive and therefore can only be done once using some offline data; it cannot modify the parameters as new information arrives.
Distribution shift detection The retraining problem is closely connected distribution shift detection and mitigation (Wang et al., 2024a; Hendryeks& & Gimpel, 2017; Scheirer et al., 2013; Cerqueira et al., 2021; Bar-Shalom et al., 2023; Rabanser et al., 2019). Some methods adapt the model to adjust to evolving distributions (Sugiyama & Kawanabe, 2012; Zhang et al., 2023; Fang et al., 2020; Pesaranghader et al., 2018). Since the signal for these methods is designed to adapt a model rather than trigger a full retraining, they are not appropriate to be used as full retraining signals. Some approaches, however, directly treat the detection of a distribution shift as a cue for retraining (Bifet & Gavalda, 2007; Pesaranghader & Viktor, 2016; Raab et al., 2020), and can be used as baselines. ADWIN (Bifet & Gavaldå, 2007) uses statistical testing of the label or feature distribution. Another approach is to directly monitor the model's performance. FHDDM (Pesaranghader & Viktor, 2016) employs Hoeffding's 's inequality, while (Raab al., 2020) relies on a Sawtoooorro--mmmmoooo Windowing test. These approaches may work well when retraining costs are low, but they become unsuitable when retraining is expensive it is not always optimal to retrain after every minor shift. This is tied to a more general weakness of lacking adaptability to varying costs. While the significance level parameteter can be adjusted, the appropriate significance level for a given retraining-to-PP crciiiigtt--perfiin -or.--oommeaecceeeeeeeee cost ratio is unknown and difficult to estimate.
Changepoint detection Another closely related field is changepoint detection, which is similar to the distribution shift problem. Changepoint detection is the task of identifying points in a sequence where the statistical properties of the data change abruptly. This problem was introduced and presented by Adams & MacKay (2007), where they aim to infer the most probable distribution of the most recent changepoint in an online setting. Recent work, such as (Li et al., 2021), has expanded on this problem in ways closer to our retraining setting, as they incorporate adaptation into the changepoint detection process,The sensitivity of the detection is controlled by certain sensitivity parameters.
However, to transform the changepoint detection problem formulated by Li et al. (2021) into the retraining problem we consider, we would need to introduce a cost for adaptation, a cost for accuracy loss, and then formulate an optimization problem find the appropriate sensitivity parameter for achieving the optimal number of adaptations. However, since this parameter lacks a specific physical or practical meaning, it is unclear beforchand how the choice of its value will impact the adaptation rate. Furthermore, in our setting, the optimal rate of adaptation (or retraining frequency) is unknown. Determining this optimal retraining frequency is one of the major challenges of the retraining problem.
Bayesian Optimization Our method is based on forecasting future model performance using historical data. This approach closely aligns with Bayesian Optimization (see (Shahriari et al., 2016) for a review on this topic), commonly used in the Hyperparameter Optimization (HPO) field. The Freeze-Thave Thaw method, introduced by Swersky et al. (2014), leverages Gaussian Processes to predict the trajectory of validation loss, enabling early stopping and optimization of the hyperparameter search space. It remains relevant technique (Rakotoarison et al., 2024). Similarly, Dai et al. (2019) derive Bayes-optimal stopping rule using a related approach. This method can be extended to predict the performance of other models and address hyperparameter optimization challenges (Wang et al., 2024b). In our context, we predict the performance of different models under potential distribution shifts, but the underlying idea is similar.
Label-free performance estimation Similarly, our approach also related to the general fields of performance estimation without labels (Garg et al., 2020; Guillory et al., 2021: Chen al., 2021) and active testing (Kossen et al., 2021). Part of the problem is similar in that the goal is to estimate performance; however, the similarity ends there, as these methods generally assume access to the model f for which performance is estimated, as well as access to the features of the dataset (Garg et al., 2020). Our approach involves forecasting performance not only for known models but also for unknown models. While our approach does not explicitly differentiate strategies, it is true that we have access to additional information. Therefore, extensions that leverage existing techniques in this area could strengthen our method.
This forecasting problem can seem similar to the problem of uncertainty quantification (Hendrycks & Gimpel, 2017; Liu al., 2020), but we are targeting average performance of unknown models, not the probability of error of given model at a given input P(f()) = y/æ))
14

---

When to retrain a machine learning model
Offline reinforcement learning Lastly, we discuss the connection to the offline reinforcement learning (ORL) setting, where the agent must learn policy from fixed dataset of rewards, actions, and states. This subset of RL is particularly challenging, the agent cannot explore the entire MDP and can only rely the dataset to infer the underlying dynamics and handle distribution shifts (Ross et al., 2011; I; Levine et al., 2020; Hejna et al., 2023). Policy gradient methods can be adapted to the offline setting using variants importance sampling, but they are generally prone to high variance and require large amounts of data to be effective (Levine et al., 2020). For this reason, Q-learning and value function methods, where the task to predict the future costs of actions, have emerged as the preferred approaches for ORL (Levine et al., 2020; Kalashnikov et al.; Hejna et al.. 2023; Kostrikov et al., 2022). Lagoudakis & Parr (2003) presents a classical method that uses linear approximation of the Q-function, while (Kalashnikov et al.) employs convolution-basee Q-function architectures for vision tasks. Others have leveraged advancements in sequential learning, applying transformer-based architectures to predict rewards(Janner et al., 2021) or Q-functions(Chebota et al., 2023). Some methods integrates epistemic uncertainty on Q-function to account for the distribution shift of unseen actions (Kumar et al., 2020; O D'Dnoggue et al., 2017; Luis et al., 2023).
If we view the states as time and the model in use, and actions as either retraining or maintaining the current model, we can frame this problem as an offline reinforcement learning (RL) problem. The problem would also feature deterministic transition matrix and highly structured reward which unusual in RL. However, most existing approaches focus on scaling to very large state spaces, employing large models, and assuming access to abundant data, making them unsuitable for our context. A key requirement for our approach is that it must be highly efficient to train. If the resources required for making a retraining decision are comparable to those for retraining the model itself, the approach becomes impractical.
A.4. Proof of Proposition 3.1
We provide the proof for our result from Proposition 3.1, which states the following.
Given that L Ipec, pes+1.e Vt [T]. a horizon of T e N, and a relative cost of retrain a, the number of retrains of the solution to Equation y* |00||2 satisfies:
r" ST V a
(25)
We start by defining a function that takes the model index i and the timesteps t as arguments, and outputs the performance pe(i,() () p,i,t, and rewrite the objective:
(a) 2||0|]1 Σ pe (ro(t),t),
(26)
0* arg min (O),
(27)
0c(0,1)
where we still have that rt()) returns the most recent index of retraining at f.
Subproblem with fixed number of retrains We can break down this optimization problem into subproblems, where we solve for the optimal retraining schedule for given fixed number of retrains r. We define such a subproblem as follows:
(0) = or Σ pe (ro (t),
(28)
0 arg min
C.))).
(29)
0c(0,1) s.t. [
Since we know that we will have retrains, we can rewrite this subproblem by encoding the retraining decisions timesteps of retrain 4 <.. .... < by. We use a simple index mapping function / [T]" 10"
0, 1 ifft (t. tt)
((It...., 0 s.t.
(30)
0, o.w.
We can remove the constant or from the objective as it does not depend on the parameters anymore. The solution of Eqn 29
given by:
15

---

When to retrain machine learning model
0,
arg min 0e(0.1) at.l))) 01
ar Σ pe (re(t).t)
(31)
t=
arg min 0c(0,1) s.t. 11
Σ pe (re(t), since the ar is fixed
(32)
arg (,00e,
Σ pe((0,)
Σ s-ttal
pe(tt)s)
Σ pe(tris)
(33)
sutptl
0 I
arg argmin M,((1,.... 11--1,,(1111
(34)
4
5-1
pe((tss))
T Σ
where M, (.,1,,,..,,,)))))
Σ pe(O,s)
+)+
Σ
Σ 18tt+11
pe(l,,
(35)
s=1
i=1
s0btt1
We therefore can focus on the new objective (ft(t...... ,,,) as minimizing this objective is equivalent to finding 0
st.....ty.. = arg min M, (f. [
(36)
M (,((t,..... (t)))
(37)
0 I ((t1,...,,44))
(38)
Lemma A.1. Given discrete function pe: [T] x [T] R with bounded L 2 (pe(i,t) -pe(i 1,t),, timestep horizon T e N, and a number of retrains e {1,T 1}, we can show that:
L(T-r)² 2 M M
(39)
That is. the relative improvement of performance cost that you can gain by increasing the number of retrainings from r to 4 1 is upper bounded by L(T
This allows us to preemptively determine the maximum number of retains r we have to consider for solving our initial problem, as we know the cost of adding one more retrain (a). Therefore, once L(T - r is smaller than a, the optimal solution cannot have higher than retrains. That is,
L(T-r*) <a <T
(40)
This concludes our proof for Proposition A.4. We provide the proof for Lemma A. in the following section.
Proof Lemma A.1 To prove this lemma, we decompose the M,+1 quantity into the M, value we would obtain with the first r timesteps of the solution (f1,...444] fr+1)* and some value:
16

---

When to retrain a machine learning model
T
My
pe(0.
Σ
pc((;
pe
Σ
8))
Σ
p)(()+1:8)
1
s=1+1
sup
Σ pe(0, s)
Σ
Σ ,t)
+ Σ pe(t;,
Σ pe(t,,,) Σ pe(ti+1:s)
+
Σ pe(0,
Σ
Σ pe(t
+ Σ
pe(t
++++1
swtt+1
Σ pe((,,,) +
Σ
pe(()-1:88 (adding zero)
+11+1
smf s-1-111
T
(f((f,,....
1
Σ
p)
Σ pe(f)4118)
sat +11+
T
- f
Σ (pe(t),.) -p)(tr:):s)
s= +1
By definition, we know that;
(o(( ff+1 M
(41)
That is, the M value that we obtain by removing the last timestamp using the solution for the r + problem.
Using that inequality our previous result, we obtain the final result:
M M
Σ
(pe(t), - pe(ti...s))
(42)
T
M;
Σ
1((511 -(;)
(43)
s-
Mi - (T (t+r - -+))
(44)
Mr 1r)-
(45)
A.4 1. PROPOSITION 3.1 PRACTICE
In this section, we illustrate how to use the result from Proposition 3.1 in practice. To restate, proposition states the following;
Given that L > |pe,, 1.,+1.1 At [T]. horizon of T E N, and relative cost of retrain a, the number of retrains of the solution to Equation 5 r* = |[0|]] satisfies:
S ST V
(46)
We present the values that guarantee various numbers of optimal retrains r" 0, in our experiment. Since we can't tt provide a true upper bound for the L value, we approximate it using the empirical maximum value that we observe in a specific dataset for (pe,,t - pe.+ In Figure 4, we can see that the a at which we know for certain that we don't need to retrain is not too far off the operational region of the problem. The oracle decides to not retrain around 0.5, and the bound from result guarantees that don't have retrain if the selected larger than 0.96.
A.5. Bounding L
In this section, we provide more details on the known results from the literature that can be connected the bound L.
17

---

When to retrain machine learning model
UPF
0.80
Oracle
6
0.75
r S
num. retrains
r s 1
0.70
r
s
C(0)
4
0.65
0.60
2
0.55
0
0.0
0.5 a
1.0
0.00
0.25
0.50
0.75
1.00
Figure 4. Results on the Gauss dataset, with the a values from Proposition A.1 providing different upper hounds on the optimal numher of retrain r". Left) Cost Co (0) Right) Number of retrains a.
Approximating L from known upper bounds For some simple models, explicit bounds on the expected performance as a function of the number samples N have been derived. We can use those upper bounds to approximate I, under no distribution shift, where the dataset size steadily increasing by a known number of samples (D).
Theorem A.2 (Standard generalization in the Gaussian model (from (Schmidt et al., 2018) )). Let ((-1)))) (()1))))))))))))))) & Rd {+1} be drawn i.i.d. from (0", (0",o)-Gaussian model with 100112 = Vd. Let
MI,,
Id be the unit vector the direction of ((1)) E ((+1) yeu, i.e., v 2/|ž||2. Then with probability at least
1 - 2 eexp (- M the linear classifier fa has classification error most;
(i+ 1
pes,t S exp
(47)
2(2 1))PI 4 20)
For the proof please refer to (Schmidt et al., 2018). An L bound value can therefore loosely approximated to match the gap of the upper bound;
(2 (i+ 1 exp ((2 [) 40)
(2 (2v
ppei. pea.ial < L exp
(48)
(12
Beyond IID data. real-world applications, data often exhibits temporal or spatial dependencies, making the non- distribution shift i.i.d. assumption unrealistic. For non-i.i.d. processes, stability analysis (Mohri & Rostamizadeh, 2007; 2010) bounds based Rademacher complexity (Mohri & Rostamizadeh, 2008) can be used to analyze generalization performance and thus to derive retraining schedules in more complex scenarios.
In the context, of the proposed retraining framework, bounds like this theoretically allow us to make precise statements about the benefit of retraining L to derive optimal retraining schedules. In practice, deriving a retraining schedule from these bounds would provide loose and non-sufficient estimate. Thus, we introduce data-driven algorithm to estimate optimal retraining schedules in our work.
18

---

When to retrain a machine learning model
Empirical knowledge the scaling law of L on N for LLMs Kaplan et al. derive scaling laws for large language models (LLMs) concerning the dependency of the final cross-entropy loss depending model size, dataset size and compute budget used for training. They find a power-law for all of the aforementioned parameters. For example, they find that the loss L of the neural network scales with respect to the dataset size N as L = (N/5.4 10th) 1 provides valuable insights for determining optimal retraining schedules. By quantifying how loss decreases with increasing dataset size, it enables researchers to estimate the expected performance improvements from expanded datasets and make informed decisions about when retraining would yield substantial benefits.
-0.05 This empirical relationship
A.6. Dataset
Dataset statistics can be viewed in Table 4.
Table Dataset description. w denotes the number of timestep of the offline phase, denotes the number timestep of online phase. The Model describes the architecture used for each fa.
Dataset
Model
Omaz
MMaa
T
Dataset size (D)
Num. features
Task
Total N
Gauss
XGBoost
0.5
7
21
8
5000
2
Binary
(Synthetic)
circles
XGBoost
0.25
21
8
5000
Binary (Synthetic)
electricity
XGBoost
1
8
2000
Binary 4,5312
yelpCHI
XGBoost
0.1
7
21
8
4000
25
Binary 67.395
epicgames
XGBoost
0.1
8
1000
400
Binary 17,584
airplanes
XGBoost
0.7
7
21
8
3000
7
Binary 100
iWild
Vision Model (see 1.
8
40,605
224x224+1
539,383
In this section, we provide a more detailed overview of each retraining datasets. Except for the iWild experiment, each individual dataset D, is constructed with distinct samples, with no overlap between Dr and Dz-1. For the electricity, airplanes, yelpCHI, and epicgames datasets, the partitions are determined based on the timestamp of each sample (i.e., the datasets are divided in temporal sequence).
electricity (?) is a binary classification where the task is predict the rise or fall of electricity prices in New South Wales, Australia. The distribution evolve due to change in consumption patterns.
airplanes (Gomes et al., 2017) is also a binary task where the task is to predict if a flight will be delayed. We follow Mahadevan & Mathioudakis (2024) and use the Sklearn Multiflow library version (Montiel et al., 2018) of the airplane dataset.
yelpCHI (Dou et al., 2020) is spam dataset. The dataset contains users, hotels and restaurants. An interaction occurs when a user submits a review for one of these hotels or restaurants. Reviews are categorized as either filtered (indicating spam) or recommended (indicating legitimate content).
epicgames (Ozmen al., 2024) includes critiques from authors on games released on the epicgames platform. Interaction features created by vectorizing the critiques using TF-IDF and incorporating the author's overall rating. The interaction label indicates whether the critique chosen as top critique.
Gauss 2 dimensional synthetic dataset. The input features as generated as X, ~ (t) (2(), a1) where (t-1) (1) 100 (1) = 0.5 (++1) 0.1. The label is generated using fixed rule 1[4 ri 0.5) +2 r2).
circles a 2 dimensional synthetic dataset. The input features as uniformly generated as X, ~ U[0, 1] The label generated using a moving nule y/: = [((r - (0.2 0.021))² + (r2 (0.2 + 0.021))² s 0.5
iWild (Beery et al., 2020) is multiclass dataset featuring images of animals captured in the wild at various locations. Originally used as a domain transfer benchmark, we adapted it into a standard classification dataset by including the location ID as a feature for the model. To obtain a long enough sequence of datasets Do. Di,..., we create the individual datasets D, using overlapping windows on the timeframe, i.c., half of the most recent images in D, are contained in 1 We avoid data leakage by ensuring that the train/val/test splits are maintained.
19

---

When to retrain a machine learning model
A. BASE MODEL OF THE IWILD DATASET
To motivate our cost considerations, we present an experiment where the base model architecture not fixed and searched for across list of potential model architectures. This could happen in practice for important applications; nothing forces practitioner to use the same base model f at each timestep.
Our architecture involves using a pretrained vision model, with a new output layer added to match the correct number of classes for our task, which is then fine-tuned for up to 20 epochs. The fine-tuning process uses the Adam optimizer with a fixed learning rate of 10-1 and a weight decay parameter of 10-3. Training was conducted using 4 H100 GPUs for days.
At each timestep fe. we perform a random search over the pretrained vision models made available from t timm, which includes 188 vision models of varying configuration and base architecture. We include the list in Appendix A.15. We also include in our search the option to early stop not, using the validation set. The model used for fo is the one that obtains the best validation accuracy.
A.7. Performance forecaster
In this section, we provide additional details on the proposed algorithm to forecast the performance. To restate, instead of learning the ((Tr,)),(rr; parameters, we learn the mean and variance parameters;
u(ri.))
(49)
orr,)).
(50)
And convert the learned parameters to the parameters of a beta distribution using the following relation (with appropriate clipping if needed):
[u 11)
a p
(51)
0²
11(1 (11-
8 (1 p))
(52)
σ²
Inputs ri.y As stated, the input of our performance forecaster model contains the model index i, the timesteps j. the time since retrain j - and summary statistics of the distribution shift Sshust- Zsheft is constructed by taking the average feature shift between the features of the most recently available subsequent datasets D, and DD-1 (where denotes the time step of
the most recent available dataset). We compute the mean features of each dimension for a given dataset; X = D. and compute the & distance between the mean feature vector of the two subsequent datasets;
Ddl
Zohift = || x,-1l11
(53)
The input features are thus given by concatenating rı,j = [i,j..] =ahafe]-
Since our methodology involves forecasting the performance of future models and on future datasets to be used by our decision algorithm, we assess the regression performance of our forecasting models and analyze how impacts the overall performance of our UPF algorithm.
To do so, construct two versions of our forecaster module Ps (Pag)) that are designed to be less performant than proposed method.
UPF overfit: A baseline designed to overfit the training data. We use Gaussian Process-based no (ri.)) with no white noise kernel, using single dot product kernel from -eikittlearn.
UPF overfit-noise: This variant further decreases performance by using the same overfitting model and adding random noise to the target values.
We report two metrics, the average mean absolute error of our prediction 4 and the average bias of our prediction Ns (rs,)) -a on the test set. We start by reporting the retraining performance of each baseline w.r.t. our base retraining
20

---

When to retrain machine learning model
metric, the AUC of values evaluated at different a in Table 5. As expected, the best performing method is the method with proposed UPF baseline which is expected reach the best MAE enror it's performance prediction, all datasets.
Table 5. AUC of the combined performanccchitraaiinng cost metric C. (0), computed over range of values, for all datasets. The bolded entries represent the best, and the underlined entries indicate the second best. The x denotes statistical significance with respect to the next best baseline, evaluated using Wilcoxon test at the 5% significance level.
Gauss
circles
epicgames
electricity
yelp
airplanes
UPF overfit+noise
0.3845
0.0722
0.3253
2.6389
0.1194
2.3767
UPF overfit 0.3849
0.0663
0.3224
2.6001
0.1194
2.3352
UPF
0.3836
0.0662
0.3203
2.5910"
0.1175*
2.3094
We then visualize the effect of the performance forecasting precision (measured with MAE and bias) on the decision algorithm's performance (measured by C. (0)) in the following figures.
Overall, we observe that the impact of poor performance depends on the difficulty of the underlying dataset.
For the airplane dataset, which is standard difficulty, we can observe gradual impact of the degradation in forecasting performance the overall retraining metric in Figure 5. The best MAE leads the best cost metric (o), and the performance gradually decreases as the MAE and bias worsen.
The Epicgame dataset 6, which is more challenging due to its less regular performance trends, shows different behavior. Here, the overall forecasting performance worse (the best achievable MAE is higher), and we observe a less regular pattern where poorer MAE does not always result a proportional increase cost, as shown terms of scale. Similarly, when
3.4
3.4
3.2
3.2
test mae
test bias
3.0
3.0
0.1207
-0.1111
0.2785
0.1297
2.8
2.8
0.3243
0.2831
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
α
a
Figure Airplanes, Cost (#) vs a with the forecasting performance metrics (mac and bias).
3.28
3.28
test mae
test bias
3.26
0.2446
3.26
0.2393
0.3103
0.31
3.24
0.3326
3.24
0.3326
3.22
3.22
3.20
3.20
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.06
0.08
0.10
α
Figure Epicgames. Cost C., vs a with the forecasting performance metrics (mac and bias).
21

---

When to retrain machine learning model
turning to the synthetic datasets, the circle dataset, which is constructed with concept drift (changing p(Y]))) is more challenging than the Gauss dataset, which only exhibits feature drift (where p(X) changes, but p (Y|I) remains constant). This impacts the effect of poor forecasting performance. In Figure for the circle dataset, we observe that small decrease in MAE paired with stronger bias can have a more sudden and drastic effect on the decision policy. Conversely, in the Gauss dataset (Figure 8), the effect of poorer forecasting performance less pronounced.
0.30
0.30
0.25
0.25
Ci0
0.20
0.20
test mae
test bias
0.15
0.021
0.15
-0.0177
0.0216
-0.119
0.10
0.0234
0.10
0.0234
0.00
0.05
0.10
0.15
0.20
0.25
0.00
0.05
0.10
0.15
0.20
0.25
Figure Circles. Cost Co (0) vs a forecasting performance metrics (mae and bias).
0.8
0.8
C(e)
0.7
Č
0.7
test mae
test bias
0.045
-0.4416
0.6
0.0649
0.6
-0.02
0.0805
0.0805
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
a
Figure 8. Gauss. Cost Co (0) vs with the forecasting performance metrics (mae and bias).
22

---

When to retrain machine learning model
A.8. Extension non-bounded metries
In this section, we show how we can extend our methodology to model non-bounded metrics often used in regression tasks, such the root mean square error (RMSE) mean absolute error (MAE).
To do so, we replace the use of Beta distribution to log Normal distribution to model our performance metric r.v. Aij-
A log normal distribution is parameterized with location m and scale parameter v. We can learn the mean and variance parameters using the same Gaussian approximation;
LogNorm(m((:.)).ttt (N (R((s)
(54)
and recover the location and scale parameters using the relation;
In(1 + σ²
(55)
v²
m In(v)
(56)
1. BETA APPROXIMATION VS NORMAL
In our method, we approximate the Beta distribution with a Normal distribution to ease the learning process;
Beta(a((ss), N (((((s)).
(57)
We verify here that this approximation doesn't have big effect the end performance. We compare the UPF method, which uses Ai, Beta((((r,))) ()),,)) with UPF (Gaussian), which doesn't It use the Beta distribution and instead uses Gaussian with learned parameters to model the performance metric: A,, (r(,),)),,(r..)))) In Figures 9, 10, 11 and 12, we can see that this does not have too big an effect on the overall behavior and performance.
0.80
2.75
0.75
2.50
2.25
Č(0)
0.70
(())
2.00
0.65
1.75
strategy
strategy
0.60
1.50
UPF
UPF
0.55
UPF (Gaussian)
1.25
UPF (Gaussian)
0.0
0.2
0.4
0.00
0.25
0.50
0.75
1.00
α
a
Figure 9, Gauss
Figure 10. Electricity
23

---

When to retrain machine learning model
0.30
3.4
0.25
3.2
Č(0)
0.20
((())
3.0
0.15
strategy
strategy
UPF
UPF
0.10
2.8
UPF (Gaussian)
UPF (Gaussian)
0.0
0.1
0.2
0.0
0.2
0.4
0.6
α
Figure Circles
Figure 12. Airplanes
A.9. Training complexity
In this section, compare the training complexity of each baseline. We report the average time required for the offline training process, online inference and discuss runtime complexity.
The CARA baseline comprises two computationally intensive components. First, it constructs the C' matrix, representing its performance estimation. This algorithm involves inferring, with a modified model, each point of the new dataset and reweighting each, which scales with O(Dneww). This needs to be done both offline and online phases. Then, in the offline phase, it performs an annealing search over parameters to find the best value that minimizes this cost approximation, taking into account the retraining cost associated with each decision. In Table we can see that this result the highest runtime for both online and offline phases.
Table 6. Average runtime the baselines on the circles dataset.
CARA cum.
CARA
CARA per.
UPF
ADWIN
FHDDM
KSWIN
Offline ms Online (one step)ms
8.4871
8.6608
7.8461
0.0947
0.0274
0.0122
0.3392
1.5604
1.5046
1.5940
0.0247
0.0351
0.0103
0.3438
In comparison, our approach consists of fitting a linear model on a small dataset. The shift distribution features must be obtained, but they involve comparing histograms, scaling O (w)) rather than exponentially with [Dr].
The distribution shift baselines do not have offline phase, they monitor shifts in the underlying distribution continuously. Their runtime complexity is therefore very low, at O(D:)), as reflected in Table 6
A.10. Additional results
In this section, include additional figures to visualize results in Figures 13, 14, 15, 16, 17, 18, and Figures 19. Overall, the results are generally consistent and exhibit a similar trend. The EpicGames dataset, however, is more challenging and presents greater difficulties for all basclines. In particular, UPF performs worse than other baselines at low values of the retraining cost ratio a. For those operating points, UPF does reach the correct retraining frequency; however, it is unable to pinpoint the optimal moments to retrain, resulting in worse performance than baselines that retrain more frequently, as shown in the right panel of Figure 19.
24

---

When to retrain machine learning model
3.50
strategy
3.25
UPF
CARA
3.00
CARA cumuul.
CARA per.
2.75
num. retrains
5
KSWIN-5%
KSWIN-50%
2.50
FHDDM-5%
FHDDM- 50%
2.25
ADWIN-5%
ADWIN-50%
2
2.00
1.75
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
a
Figure 13.
Result on the electricity dataset. left) Cost (0) vs a. right) Number of retrains vs a.
1.300
1.250
1.225
1.200
num.
3
1.175
1.150
1.125
1.100
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.06
0.08
0.10
Figure 14, Result on the yelp dataset. left) Cost Č. (0) a. right) Number of retrains vs
25

---

When to retrain machine learning model
3.40
3.35
num. retrains
3.30
Č
3
3.25
3.20
0.00
0.01
0.02
0.03
0.04
0.05
0.00
0.01
0.02
0.03
0.04
0.05
a
Figure 15.
Result on the epicgames dataset. left) Cost (.) vs c. right) Number of retrains vs a.
1.3
1.1
1.0
retrains
(
0.9
0.8
0.7
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Figure 16.
Result on the Gauss dataset. left) Cost (.) vs a. right) Number of retrains vs
26

---

When to retrain machine learning model
0.45
0.40
0.35
num. retrains
0.30
(
0.25
0.20
2
0.15
0.10
0.00
0.05
0.10
0.15
0.20
0.25
0.00
0.05
0.10
0.15
0.20
0.25
a
Figure 17.
Result on the circles dataset. left) Cost C. vs a. right) Number of retrains vs a.
4.2
strategy
UPF
4.0
CARA
3.8
CARA cumul.
num. retrains
CARA per.
3.6
C
KSWIN-5%
3.4
KSWIN-50%
FHDDM-5%
3.2
FHDDM-50%
3.0
ADWIN-5%
ADWIN-50%
2.8
0.0
0.2
0.4
0.6
0.0
0.2
0.6
a
Figure 18.
Result on the airplanes dataset. left) Cost Ć. a. right) Number of retrains vs a.
We additionally include results with the oracle baselines in Figures 19, We can see that the UPF baseline is reasonably close to the optimal algorithm in two of the datasets (circles and electricity), but struggles for the more challenging dataset, epicgames. Looking at the number of retrains, we can see that UPF more closely follows the retraining frequency of the oracle for all datasets.
27

---

When to retrain machine learning model
3.50
3.26
0.30
3.25
3.24
3.00
0.25
3.22
2.75
Č(0)
0.20
Č
2.50
Č(
3.20
3.18
2.25
0.15
2.00
3.16
0.10
1.75
3.14
1.50
3.12
0.0
0.1
0.2
0.0
0.5
1.0
0.00
0.02
0.04
α
a
8
8
strategy
strategy
strategy
5
UPF
UPF
UPF
oracle
oracle
oracle
num. retrains
CARA
num. retrains
CARA
num. retrains
CARA
3
4
2
2
0
0.0
0.1 a
0.2
0.0
0.5
1.0
0.00
0.02
0.04
α
a
Figure 19. Result the circles (left), electricity (middle) and epicgames (right) datasets. Top) Cost (0) Bottom) Number of
retrains vS Ox
A.11. Methodology as offline RL
We can frame the retraining problem as an offline RL task (Levine et al., 2020). We define state space where each state described by the index of the trained model and the timestep; S (T) x (T). The action space is either retrain or not, so The state transitions deterministic and known:
A (0,1).
if A = 0,5.-1 = (i,t+1) T 1++
T(S+ Ai,
(58)
O.W.
Figure 20 provides a visualization of the MDP. Since the state transitions are deterministic, we can define the deterministic transition function:
St11 t(att st).
(59)
The reward function only depends on the end state (which describes the performance a model evaluated at timestep t) and on the action. Using pes to denote the performance at state S and reusing of tradeoff parameter o, we have the reward
28

---

When to retrain machine learning model
Time
t=0, f=0
t=1, f=0
t=2,
t=3, No retrain (keep)
f=0
f=0
Retrain
t1
t=2,
1=3,
Model
f=1
f=1
f=1
t=2,
1=3,
f=2
f=2
t=3,
f=3
Figure 20. Visualization of MDP
function:
r(a2, S1-1) =aa; pe
(60)
To match our setting, the discount factor has to be set to 7 1.
The goal is to learn policy ı on offline data to generalize to the online period. The offline dataset is given by: Do fline N (8n, On, Tn) m=1
The objective is defined as:
T+u Σ
=T)
[[
as)),
(61)
which is the same objective as we defined, with the added option of defining a random policy to make decisions p> (0))
Tow Σ r(s,, at)
J(x) E (p))
(62)
T+w
-Bo-p-(0))
Σ
peau1
(63)
(oa (a)
(64)
Q-learning (approximate dynamic methods) The basic idea of Q-learning to define a Q function and derive deterministic policy ı from it. The Q function is defined follows;
Tew Σ
( ar)
(65)
Pst
and the policy is set to:
(ac(s:)) -ō(a, arg max Q(s,, dz)).
(66)
29

---

When to retrain a machine learning model
Since the optimal policy x* should satisfy
(st ae) ( max ("150 (++)],
(67)
Oee1
one algorithm is train Qo until that equation is satisfied.
In our case, the transition is deterministic, so we can define 8t+1 188, ar) and have
Q" a+) = r(s;, maxa.. Q" (1(S,, az); 0c-1). The idea then to parameterize Qa, and minimize the following for all samples in the dataset using the Bellman update:
(68)
[Qqoson an) [r(sn, an) + max (s ,a)))²
(69)
First we set the target:
r(n, an) max Qo(5',
(70)
then we optimize
0 an) -yn)²
yn)
(71)
and the algorithm iterates between those two steps. We can therefore apply any Q-learning method to our problem, provided that it uses a standard Qo parameterization.
Connecting Q-learning to our UPF algorithm
In our setting, we have special knowledge the structure of Q. First, there is no randomness on the transition state, so we know that:
Un an) + max (n)
(72)
By definition, we have that:
Qs ax) -0;0 pe,,t + max Q& ((),((ss)))) ac.1)
(73)
While computing the Bellman update and setting the target, will have to predict the end performance:
that the function of one of the last states Q& (ST,2*))
Qa(ST = -PesI..
(74)
= =ff
(75)
By the DAG structure of the transition function, and since the value is known, we can parameterize recursively all the Q functions with shareable components:
(1-1)) = fo(ST-122) max(-a-
(76)
where each (sT-11..) is modeling the performance pesr, at that given state.
The MSE objective that is traditionally applied (Eqn. 71) can then be decomposed into terms, where one of the terms corresponds to our objective:
C= (Q0 an) yn
(77)
(-aan- - fo((n) max(-a- (o(1))---((
(78)
(an + pes, max Q0 (n) an+
(79)
ansl
= (Fo(sm) -een. + max(-a fo m + max Qs (t(sn, an), an))))))
(80)
C= Σ (fo(sm) pe.. )² CC.
(81)
30

---

When to retrain a machine learning model
The term (f6( -pe,,) the loss function aligns with our objective, as A. represents our model's approximation of the performance metric peij. Therefore, with this specific parameterization, we can establish connection between Q-learming and our learning method.
However, as noted the main text, applying existing ORI. methods to this problem would not be effective. The problem involves deterministic transition matrix and a highly structured reward, both of which are uncommon typical RL settings. Additionally, most RL methods prioritize scalability to large state or action spaces, use complex models, and assume access to plentiful data, making them ill-suited for our scenario. A key requirement for our approach is training efficiency, given our limited performance data and the need for online adaptation as more information becomes available. If the computational cost of deciding when to retrain is comparable to the retraining process itself, the approach becomes impractical.
A.11 A.11. OFFLINE RL BASELINES
In this section, we present results using an offline RL baseline that is appropriate for low-data settings: Least-Squares Policy Iteration (LSPI) (Lagoudakis & Parr, 2003). We follow the detailed RL formulation as previously presented. To implement LSPI, we use the model index i and timesteps as states (following the formulation from the previous section). In LSPI, various approximation methods are introduced solve the linear equation, but these are unnecessary in our case, solve it exactly due the small size of problem. We present various versions of this baseline by changing the À parameter. In Table 7, we can see that this proposed baseline is not competitive. These initial results for this basic formulation of the offline RL problem indicate that more care and design should be taken to appropriately solve this problem using offline RI,, supporting that existing RI. methods, as they are, may not be well-suited to solve the problem.
Table AUC of the combined performance/hetrriiiing cost metric C. (0), computed over a range values, for all datasets. The bolded entries represent the best, and the underlined entries indicate the sccond best. The The * denotes statistically significant difference with respect the next best baseline, evaluated using a Wilcoxon the 5% sigmificance level.
electricity
Gauss
circles
airplanes
yelpCHI
epicgames
iWild
ADWIN-5%
2.8099
0.4533
0,0753
2.6353
0.1298
0.3217
3.7371
ADWIN-50%
2.8131
0.4848
0.0753
2.7147
0.1298
0.3238
4.2564
KSWIN-5%
3,8979
0.3975
0.0753
3,2300
0.1322
0,3420
4.4268
KSWIN-
4.0521
0.9530
0.0794
3,2042
0.1655
0.3537
4.4268
FHDDM-5%
3.1525
0.3893
0.0753
2.6577
0.1324
0.3298
4.4267
FHDDM-50%
3.4037
0.5918
0.0772
2.7077
0.1450
0.3389
4.4268
CARA cumul.
2.7147
0.3862
0.0731
2.2900
0.1299
0.3228
3,8922
CARA per.
2.8986
0.4678
0,0800
2,4061
0.1318
0.3260
3.7527
CARA
2.7198
0.3841
0.0726
2.2753
0.1294
0.3202
3.9506
LSPI X ==1
4.3820
1.0530
0.2412
3.7140
0.1493
0.3523
LSPI X 0.5
4.5260
L.0837
0.2455
3,6924
0.1442
0,3566
LSPI À = 0.0
4.5317
1.0933
0.2478
3.5862
0.1378
0.3573
UPF (ours)
2.5782* *
0.3829
0.0668
2.2865
0.1293
0.3189
3.0498*
oracle
2.4217
0.3724
0.0627
2.2298
0.1275
0.3170
2.4973
A.12. Relating objective to the CARA formulation
In (Mahadevan & Mathioudakis, 2024), even though they also tackling the retraining problem, they are formulating the problem differently.
Instead of using a binary vector to model the retraining decisions, they use a sequence of model indices S = [s],... , ST with the constraint that s, e {O....,tt t). If s, t, signifies a retrain.
The cost objective they consider is similar to ours; they sum over the timesteps to get the cumulative total cost. The cost per
31

---

When to retrain machine learning model
timestep is encoded in upper triangular matrix C:
Vi if 1'' <
c[t
k if = (cost of retraining)
(82)
O.W.
where Vie is defined some "relative staleness cost". The total cost is defined as:
C (S) = C[s,
(83)
[=1
The staleness cost is defined as the cost of using model to classify data from Q2 approximated by dataset Dy:
(QQ2, D3, fi)
Σ q=Q²
Σ (((1,, Dal x.y-Ds
(84)
The aim of this metric is to predict the performance of fi on the query points Q2 by computing the loss on a reference dataset D3. The idea is to weight the loss each sample D3 by how similar they are to the query samples in Q2 (this the role of sim(q. x)).
((Ja(1)
Σ cim((,x)((J1.2)
(85)
DDal a.y-Da
v (Q2, Fi) ((fss))((((())))))
(86)
*Neper3,12
(87)
The relative staleness cost is defined as the difference between staleness costs:
V r,,' (Q,,, Se) Se, This is intended to approximate the relative gap of performance:
(88)
T 2e a Ne(per,-peet
(89)
In our experiment, we directly use V Dr, fe) as an approximation of per. and apply the CARA algorithm directly on the staleness costs instead of using the relative staleness cost.
Relating it to our formulation
Our objective is given by;
C(0) c|10|] eN Σ pergt
(90)
t=1
To understand the connection with our formulation, we start by rewriting the CARA cost as:
T
Ceara (S)
[[s t]k 1[se D
(91)
T
1[ t]k [[se < {t)
(92)
T
Σ
[s = Ae Nel < t1] (pesst per.)) from (89)
(93)
t-1
Cara (0) x| x|00|| + Ne Σ (pero.t - pet.)) switching to our notation with 0.
(94)
This reveals the assumptions that are required for both solutions to coincide. First, this approximation for the loss of future model fi should hold:
(((( x Del D.
Σ
sim((ag x)(1,,
(95)
32

---

When to retrain a machine learning model
Second, in order to have:
C(0) C
(96)
we need
Ne Σ pe1,
K C
(97)
||0||
Proof: We require that:
d|011 + Ne Σ
pe x||0|11 + Ne Σ pperot - per.))
(98)
This implies that:
dl0|| + Ne Σ pera.t = xi|0111 Ne Σ pera,t Ne Σ
per, t=1
(99)
and hence that:
==c+ Ne Σ per,
(100)
||0||
The cost of retraining the CARA formulation must thus scale with the minimum performance cost that can be obtained by always using the most recent model Ne Σi per,t, divided by the number of retrains that have been made. It is of course not possible to set this value, it depends 0, but it gives insight into how the formulations relate to each other.
A.13. Varying training data size
In this section, we provide experimental results where we assume that we have access fewer offline time steps and analyze how impacts the results. We display the relative improvement of the best baseline vs. the competing baselines reporting normalized AUC values in Tables 8,9, and10. Overall, our method remains effective in scenarios with reduced training data. It demonstrates greater robustness compared to the CARA baselines, which can be explained by the fact that it can adapt to new information received during the online process, which CARA cannot do. With very few training steps (w = 2), the CARA basclines suffer the most, reaching more than twice the error for some datasets. With more data (w 4), the relative performance more in line with larger datasets (w 7), with UPF remaining the best.
Table 8. 2. Normalized AUC of the combined performannce/eetaaiinngg cost metric Ća (0), computed over range of a values, for all datasets. normalize by dividing by best value each dataset. The bolded entries represent the best. The x denotes statistical significance with respect to the next best baseline, evaluated using Wilcoxon test the 5% significance level.
2
electricity
airplanes
yelpCHI
epicgames
Gauss
circles
CARA
1.0000
1.0101
1.0100
1.0282
2.6519
1.4792
CARA
1,0669
1.0680
0.0544
2.7437
4.0150
1.6872
CARA per.
2.1971
1.6703
0.0661
2.9131
10.6965
1,8901
UPF
1.0258
1.0000
1.0000
1.0000
1.0000
1.0000
A.14. Results on the Wild Temporal dataset
In this section, we present preliminary results on one dataset from the suite of temporal datasets from Yao et al. (2022). Specifically, we present preliminray results from the yearbook dataset.
To construct our sequence datasets Di...., we follow the construction from (Yao et al., 2022). For training, we iteratively add more samples from each year, spanning from 1930 2012. For testing, we evaluate only on samples from the most
33

---

When to retrain a machine learning model
Table w 4. Normalized AUC of the combined performance/retraaningg cost metrie C. (0), computed over range of values, for all datasets. We normalize by dividing by the best value for cach dataset. The bolded entries represent the best. The * denotes statistical significance with respect to the next best baseline, evaluated using Wilcoxon test the 5% significance level.
4
electricity
airplanes
yelpCHI
epicgames
Gauss
circles
CARA
1.0093
1.0024
1.0000
1.0063
1.0049
1.0653
CARA per.
1.1029
1.0721
1.0017
1.0168
1.0984
1.0045
CARA
1.0153
1.0060
1.0025
1.0220
1.0042
1.0501
UPF
1.0000
1.0000
1.0008
1.0000
1.0000
1.0000
Table 10. w Normalized AUC of the combined performance/rctraiiing cost metric C. (0), computed over range of values, for all datasets. We normalize by dividing by the best value for each dataset. The bolded entries represent the best. The * denotes statistical significance with respect to the next best baseline, evaluated using Wilcoxon test the 5% significance level.
7
electricity
airplanes
yelpCHI
epicgames
Gauss
circles
CARA C.
1,0530
1,0065
1.0046
1.0122
1.0086
1.0944
CARA per.
1.1244
1.0575
1.0193
1.0223
1.2219
1.1976
CARA
1.0549
1.0000
1.0008
1.0041
1.0031
1.0868
UPF (ours)
1.0000
1.0050
1.0000
1.0000
1.0000
1.0000
recent year. As for the model ft, we use the ERM model from (Yao et al., 2022), and follow the training procedure from Yao et al. (2022). We use a similar setup to the one followed in our experiment, setting the offline window size w 7, evaluating over an online phase of T steps, and presenting results over 10 trials (See table 11). Preliminary results this dataset which can be seen in Table 12 are inline with the results from the main paper.
Table 11. Dataset description. denotes the number of timestep of the offline phase, denotes the number of timestep of the online phase. The Model describes the architecture used for cach ft.
Dataset
Model
Oman
w
M
T
Dataset size ((D)
Num. features
Task
yearbook
ERM
0.5
7
21
(varies)
32X32X3
Binary
A.15. List of timm pretrained vision models
beit_base www.baaseppahhhhhhcccea 224
beitv2.base.patchh 62224 1..244
caformer_s18".
cait.s24_224
cait.xxs24.224
cait.xxs36.224
coat_lite mini
coat_lite_sm
coat lite tiny
coat_mini',
coat_tiny
coatnet r.w "
"coannet b rw 224
"coatnet_nano rw 224'
"coatnet.rmpp_1 -epp---ww22220 _224"
"coatnet rmlp nano rw 224
34

---

When to retrain a machine learning model
Table 12. AUC of the combined performancc/ietraningg metric C. (0), computed over range of values, for all datasets. The bolded entries represent the best, and the underlined entries indicate the second best. The The * denotes statistically significant difference with respect the next best baseline, evaluated using Wilcoxon the 5% significance level.
yearbook
CARA cumul
0.0351
CARA per.
0.0195
CARA
0.0322
UPF
0.0120
Oracle
0.0105
coatbext.nano.r.224"
convformer.s18
ccnvit_bbae".
'convit small',
convit_tiny',,
10nvmixer.1024.20. ,
aonvnext atto
convncxt-atto.ols.,
convnext_base"..
convmext_femto
femeto ols"
comvmext_nano".
convnext.nano.ols
convnexi_pico
convnext-pico.ols ols
convnext.small
convnext_tiny
convnext_tiny.hnf
convnextv2.atto
convnextv2 femto
convmextv2.nano
convnextv2.pico
convnextv2.tiny
crossvit 15 2
crossvit_15.das ddageerr2422
crossvit.15
crossvit,18.240
crossvit.l8.dagger 2 240
crossvit.9.240
crossvit.9.dageer2440 240'
crossvit.base.244
crossvit.small.2400
crossvit tiny 240
cs3darknettfocus...
cs3darknet.foeus.n
cs3darknet
"cs3darknet.m"
"es3darknet.x"
"cs3edgenet.x"
"sssse edgenet x',
35

---

When to retrain machine learning model
cs3sedarknet.1
es3sedarknet x
cspdarknet53
cspresnet50
espresnext50
darknet53',
ddarknetaa33
"davit_base"
'davit_small
"davit_tiny"
ddeit3 base patch16 224",
"deit3_small.patchh6.. 2
"deit_base_distilled_pacdd 1 _224"
Sist.base.patchl6..o
deit.smalll.distilled..aahhh '
deit.smaill.patchh16.2444
deit.iing.distilleddpphhhh ,
deii_iiny_ppacch1..244..
densenet121
densenet161
densenet169
densenet201
densenetblur121d';
'dla102'
'dla102x'
'dla102x2'.
"dla169"
'dla34'
dla46_e',
dla46x c',
dla60'
dla60 res2net'
"dla60_res2next"
dla60x
ddla60x
"dm ffmenfnet_f0
_dm_nfnet_f
'pp66''
dpp66b'.
'pn92'
'pp98''
"cca_nfnet_l0"
"eca_nfnee 1l
'ca_nffne__2'.
eca-rrenne33ts'"
ecc_reenext226ss
36


Reviews:
[{"review_id": "ygWrLFowr2", "reviewer": "Reviewer_sshj", "summary": "In this paper, the authors proposed a novel strategy to determine when to retrain deployed machine learning models. Specifically, the authors first developed a future performance forecaster for predicting the performance of models built in future time steps. Building on top of it, the authors model the problem of whether to retrain a model or not into a decision-making problem and connect it to a reinforcement learning problem. The empirical studies can demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Claims and Evidence\n## 1. Beta r.v. motivation:\nWe agree that there could be a better motivation for the Beta distribution and will add sentences to improve the flow. Our choice to use a Beta distribution was motivated by the facts that 1) Beta distributions are appropriate (and a common choice) for continuous r.v.s defined over a finite interval; and 2) the empirical accuracies can be interpreted as sum of Bernoulli outcomes and the Beta is the conjugate prior. Moreover, the model performs well empirically, and we conducted some experiments with a Gaussian-based alternative that can be seen in Appendix A.8.1, where we can see that the Beta model slightly outperformed. We recognize that this is a modeling choice and the framework could be applied with other distributions.\n\n## 2. Future performance forecaster is trained first and then used for derivation in Section 4.2\n\nYes, we train the forecaster on some available training data collected at some past timesteps (the offline data, denoted $\\mathcal{I}^{offline}$ in Section 3.1). We incorporate a modeling assumption that the predictive relationships learned during this offline period also exist in the operational period, allowing us to forecast performance in future timesteps.\n\n## 3. Are rules in Section 4.2 derived from RL algorithm? \n\nNo, these rules are not directly derived from the RL formulation. Section A.11 aims to clarify and highlight the connection between our approach and an offline RL formulation, though it is not a perfect match. To make this clearer, we will rename the appendix \"Connection to an Offline RL Formulation.\" Since the problem is closely related to offline RL, we found it important to emphasize these connections. However, the addressed task is not ideal for RL, as discussed in the main text, which is also reflected in the performance of a basic offline RL baseline in Table 7.\n\n# Weaknesses\n## 4. Additional discussion to explain the connection to continual learning\n\nThere is a connection between the \"when to retrain\" problem and continual learning and we will add more text to discuss. The main difference lies in how cost is incorporated in the problem formulation and whether the setting allows for future planning. In particular, continual learning approaches do not ever delay updates due to cost considerations. For example, in a when-to-retrain scenario, I might decide to hold off from retraining in period 7, because I only have budget to retrain one more time, and if I retrain in the next window in period 8, it will give me better performance for periods 9-11. In continual learning, the primary goal is typically to maintain performance, sometimes while trying to reduce the number of gradient updates. In contrast, our when-to-retrain formulation is abstracting away the training procedure and focuses more on cost considerations. Here, cost is not limited to gradient updates. It is a parameter set by the practitioner which will influence the algorithm’s behavior, as the algorithm explicitly considers the cost trade-off when deciding whether to retrain.\n\nMoreover, our algorithm does not aim to optimize how models are retrained/updated under distribution shift, which is a core focus of continual learning. \nIn fact, continual learning techniques could be combined with our algorithm to balance the decision of whether to apply an update. Even if continual learning mechanisms indicate that gradient updates should be applied to adapt to distribution shift, our approach could deem that it would still be better to wait due to other training cost considerations.\n\nAs suggested, we will add a more detailed explanation of the connections and differences between our problem and the field of continual learning, incorporating these key points.\n\n## 5. Theoretical analysis: does it apply to datasets with arbitrary distributions\n\nYes, in theory, the result in Eqn. 7 does not impose any structure on the dataset distributions, so it is the case that it would apply to datasets with arbitrary distributions. It is also the case for Appendix A.11. However, since the result relies on bounding the performance gap of two models trained on subsequent datasets $\\mathcal{D_i}$ and $\\mathcal{D_{i+1}}$, evaluated on dataset $\\mathcal{D_t}$, it will likely be related to a function of the distributions of these datasets. We will add a comment to this effect as suggested. This is exemplified in Appendix A.5, where we present examples illustrating how the performance quantities $pe_{i,t}$ can be characterized for a simple Gaussian model. In this example, the bound is indeed a function of the dataset characteristics, i.e., its size $|\\mathcal{D}|$, variance $\\sigma$ and dimensionality $d$.", "claims_and_evidence": "Overall, the claims are quite sound and clear. But there are still some minor issues:\n\n+ First, I feel that more explanations are needed to clearly explain why the random variables $A_{i,j}$ can be modeled as a Beta distribution. I feel that there is a jump in the flow which may confuse readers.\n+ Second, I am wondering how the future performance forecaster is related to Section 4.2. Is it true that the future performance forecaster is trained first and then used for the derivation in Section 4.2, right? If this is true, how is it trained in the first place without knowing the ground-truth future performance numbers? If this is not true, how is its training process correlated to the following RL algorithm?\n+ Third, in Section 4.2, the authors discussed how rules for determining $\\Theta_t$ at each time step $t$. Are these rules derived by the RL algorithm? More clarifications of its connection to the RL formulation would be essential.", "methods_and_evaluation_criteria": "I think the general method looks good to me, which is fully backed by rigorous theoretical analysis. The evaluation criteria is also appropriate.", "theoretical_claims": "Yes, I checked the theoretical claims and proofs, which are all correct.", "experimental_designs_or_analyses": "I think the experimental designs and analysis look good to me.", "relation_to_broader_scientific_literature": "This paper is closely related to the literature on distribution shifts.", "essential_references_not_discussed": "No, everything is cited properly.", "other_strengths_and_weaknesses": "Strengths: \n+ The idea of framing when to determine to retrain a model as a reinforcement learning problem is novel. This is also supported by a rigorous theoretical analysis.\n+ Except for some confusing points described above, the overall flow of the paper looks good to me.\n+ The authors also performed extensive experiments to demonstrate the effectiveness of the proposed methods.\n\nWeakness:\n- I would suggest that the authors add some additional discussion to explain the connection between the retraining problem and the continual learning problem, which I think are pretty close.\n- Although the theoretical analysis in the paper is perfect, the theoretical analysis (say the one in Appendix A.11) seems to be unrelated to the distribution differences between $D_t$ and $D_{t+1}$. Does this mean that the theoretical analysis applies to datasets with arbitrary distributions? More discussion on this point would be essential.", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Ann0X05K6", "reviewer": "Reviewer_z2yD", "summary": "This paper faces the complex task of understanding when a machine learning model needs to be retrained in the presence of drift. In doing so, it takes into account the problem associated with the trade-off between retraining cost and poor model performance. Their approach is based on forecasting the evolution of the model performance over time and using this information to later decide whether to retrain or not.  \nThe authors report a set of experiments addressing classification tasks showing the performance of the devised approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Methods \n\n## 1. Types of non-stationarities: approach is more suited for slowly changing environments rather than abruptly changing ones\n\nThat is a good point. It is true that the best scenario for our approach is a slowly changing environment, and that abrupt changes would be harder for our model to handle. We will clarify the scope of our paper by adding a comment to the effect that our model is mostly adapted to gradual performance changes in the limitations section and state that abrupt changes in performance remain an open problem.\n\nThat being said, the combination of probabilistic forecasting and a risk-averse decision algorithm in our method provides some defence against abrupt changes in the environment. Since our approach does not rely solely on pointwise estimates but also produces probabilistic estimates through mean and variance predictors, a failure in capturing sudden changes will result in high-variance estimates for past models.  Due to our risk-averse design, the model would be biased toward frequent retraining to avoid catastrophic performance drops. We believe unexpected performance rises are less likely than sudden drops, which is why our approach is designed this way.\n\n## 2. Forecasting procedure: choice of the feature vector\n\nThere are certainly various designs of input features that could be more informative. However, we aimed to keep our approach simple, as we are in some ways introducing a new problem setting. Our priority was to ensure broad applicability across settings, rather than focusing on more complex feature combinations that might yield better performance but reduce generalizability.\n\n## 3. Forecasting procedure: inability to detect concept drift\n\n   The point about concept drift is an important one. We agree that it is crucial for the model to identify concept drift, and in fact, our model is capable of capturing pure concept drift where  $p(Y_1|X_1) \\neq p(Y_2|X_2)$ with $p(X_1) = p(X_2)$.\n   In short, the presence of concept drift will be reflected for a model $f_0$ that was trained on dataset $D_0 (X_0, Y_0)$ in the shift of performance on the dataset $X_1, Y_1$ versus the performance on the dataset $X_2, Y_2$. Even if the dataset shift feature stays constant, since we also feed the time index as input, our model can pick up this trend. The performance will be a decreasing function of $t$ and that can be captured in the model.\n   \nThis is an important point worth highlighting. We will add a comment in the text about the model's capability to capture concept drift and also include the possibility of designing more informative input features as a future direction.\n\n## 4. Dependence on past predictions: prediction of future values also depends on the previous choices of retraining or not the model\n\nThis is an excellent point. It is true that retraining decisions influence the online dataset $\\mathcal{I}^{online}$, which in turn affects the prediction algorithm. This effect is, in fact, ignored by our algorithm. \nEmpirically, we do see that this is not a problem as we observe a good performance. However, it would be a good research direction to investigate strategies to allow for more non-optimal decisions to provide training data for the prediction algorithm in other decision regions. \n\nWe will add the following comment to highlight this fact by adding in line 259: \n\"*As constructed, past decisions influence the dataset $\\mathcal{I}^{online}$ available for the next iteration, but this effect is ignored by the algorithm. Empirically, we find that the algorithm performs well despite this. One direction worth investigating is the incorporation of random decisions to allow the predictor to learn over a broader region of actions and responses.*\"\n\n\n## 5. Simplifications: linear model for prediction, beta distributions. Do these limit the applicability to more general and complex scenarios?\n\nWhile our approach uses simple components, we do not consider it limits the applicability of the approach to complex scenarios. Indeed, the iWildCam experiment represents a reasonably complex setting, with distribution shift, and 188 architectures of different natures.\n\nGiven the nature of the retraining problem, where data is scarce and potentially noisy, a linear regression model with variance estimation is a suitable choice, especially when only 10-20 samples are available for learning.\nFurthermore, since this is a relatively new problem formulation, we found it appropriate to introduce a simple yet robust method that can be applied across various settings.\nThat being said, the limited modeling complexity of the chosen method may indeed restrict the model's expressiveness, and exploring more complex algorithms in the future would be valuable. \n\nRegarding the Beta assumption, it is a common distribution for modeling continuous random variables over a finite interval. Please see our response to point 1 of Reviewer 1 for more discussion.", "claims_and_evidence": "The claims made in the work seem sound to me.", "methods_and_evaluation_criteria": "I agree with the authors that this is a tough problem since many things need to be taken into account, and it is difficult to define a proper cost-to-benefit ratio or to analyse the type and amount of non-stationarity in the environment.\n\nI believe that the work may benefit from a discussion about the types of non-stationarities that can be handled using this approach. Indeed, since the authors define a forecasting approach for the model performance, I assume that the presented approach is more suited for slowly changing environments rather than abruptly changing ones, which are harder to handle.   \n\nConcerning the proposed method, I believe that one of the main weaknesses lies in the forecasting procedure. Some of the concerns are about the choice of the feature vector $r_{i,j}$. It seems to not be informative enough since the only feature containing information about data is the $z_{shift}$ feature which relates the amount of difference (in L-1 norm) of the mean of input features $\\bar{x}\\_t$ with the mean at the previous timestamp $\\bar{x}\\_{t-1}$. Using only this feature may for example prevent the identification of a \"real concept drift\" (with this expression, I mean when the input distribution does not change but the target distribution conditioned on the input features does). \n\nAnother concern is that, as far as I understood, since the first feature refers to the used model, if the algorithm chooses to retrain or not the model at time step i+1, this influences the target $a_{i,j}$. This means that the prediction of future values also depends on the previous choices of retraining or not the model. I wonder whether this may represent a problem.  \nOther simplifications lie in the use of a linear model for prediction and in the choice of the beta distribution for the prediction of forecasted values. How can these choices limit the applicability of the approach to more general and complex scenarios?\n\nConcerning the evaluation criteria, the authors make a good work in presenting their results by testing it under different values of the retraining cost $\\alpha$.", "theoretical_claims": "I did not check the proofs of the theoretical claims.", "experimental_designs_or_analyses": "The design of the experiments is done extensively and the presented results seem sound to me.", "relation_to_broader_scientific_literature": "The related works are clearly discussed. With respect to the most related work of Mahadevan & Mathioudakis (2024), this paper removes some strong assumptions concerning the data distribution and the impact on model performance and presents a more general objective that takes into account both the retraining cost and the average performance.", "essential_references_not_discussed": "The related works are thoroughly discussed.", "other_strengths_and_weaknesses": "Among the strengths of the work, I would mention the extensive and detailed numerical simulations showing the benefits of using the presented approach.\n\nConcerning the weaknesses, see the Section on \"Methods And Evaluation Criteria\".", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See Sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YAx0TsuAqG", "reviewer": "Reviewer_4zp7", "summary": "One underexplored problem is that of when to retrain a model, assuming a sequence of datasets over time that experience distribution shift. This problem can be formulated as an off-policy reinforcement learning problem, where the goal is to find a policy with minimum cost. Authors define cost as the sum of costs from retraining and also costs from worse model performance from using an older model. Authors propose at each step to use past data to fit a future performance forecaster, which is a bayesian model for the performance of models trained on data at time i and evaluated on data at time j, using features such as the time between when the evaluation data is collected and when the model training data is collected. Authors then propose an online policy that uses this future performance forecaster to model total cost, and decide whether to retrain based on comparing fixed quantiles of this modeled total cost under training vs not training. The proposed method, called UPF, generally outperforms baselines including CARA.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Methods\n## 1. Bayesian model: ablations with/without feature $z$\nExperimentally, we saw that without the shift feature $z$, the results were on average $5-10\\%$ worse,  with some datasets being more affected than others (the electricity and the synthetic datasets were mostly unaffected). We can include this as an ablation study.\n\n## 2. XGBoost justification - use for tasks involving text\n\nBetter suited models could be used for each specific dataset, but we decided to keep a single model in these experiments for simplicity. We chose a tree-based algorithm since they can handle different datatypes.\n\n## 3. Why choose between 188 models for iWildCam? What happens if you don’t?\n\nThis experiment was designed to highlight that the setup does not assume any particular model $f$ to be consistently used, and to illustrate a scenario where the training cost is not restricted to the training of a single model. The procedure to obtain $f$ is actually outside of our framework; it is part of what defines the dataset in our setting. We could have also chosen a single model, which would have resulted in lower accuracy values.\n\n## 4. Details on dataset construction\nThank you for pointing this out. We will include additional details in Appendix A.6.\n\n## 5. Section A.8.1 advertises something different from what it delivers\n\nThat is true. It is slightly misleading, and we will change the title of the section to *BETA APPROXIMATION VS NORMAL*.\n\n# 6. Theoretical claims - errors in Proof of Prop. 3.1\n\nThank you very much for taking the time to verify our result. Yes, there is double counting in 33. The summand should be offset by a minus 1 in the first two terms to avoid this. The second mistake you pointed out in (44) is also related to this missing offset. The added terms should start at $s=t^*_{r+1}+1$ and not at $s=t^*_{r+1}$. We will fix these errors.  This doesn't affect the final result.\n\n# Weaknesses\n## 7. Not a realistic setting.\n\nIf a practitioner has no flexibility on when/if to retrain the model due to some operational/organizational/procedural reasons, it is true that our algorithm is not applicable, since there is no retraining problem. However, we believe that this is a rare case and that the problem we are addressing is ubiquitous in industrial settings. Many practitioners do have this flexibility and are actually faced with the problem of deciding when to retrain their model. Indeed, we were motivated to formulate this problem due to collaborations with several industry partners who have live, deployed models, and must make this exact decision. \n\nMany companies collect data in real-time, and their interest is to forecast something based on this most recent data. That is the case for companies that operate recommender systems, perform manufacturing control (e.g., detecting faults), or employ fraud detection systems, for example. \n\nThe cost of retraining is indeed usually related to external costs (i.e., not computation), but this is precisely what we are trying to tackle in this work. While quantifying these costs can be challenging, these types of risk assessments and quantification are actually made all the time, and are inherently considered when a decision to retrain is being made. Financial transaction companies running fraud detection systems do estimate the potential financial cost that could be incurred if a newly deployed model performed poorly.\n\nThe contribution of our work is to provide\na formal framework for practitioners to incorporate cost estimates into retraining decisions. It asks them to provide a **direct specification** of what amount of accuracy gain is worth this holistic cost of retraining, at a given time step. Our formulation has the considerable advantage of enabling this explicit quantification. This formulation is the most interpretable and therefore useful for setting this problem and making decisions. \n\n## 8. A common problem in modeling under distribution shift over time is how much past data to train on. Here, this problem is not part of the formulation\n\nThis is a valid point and could be an interesting extension to our formulation. Rather than just deciding whether to retrain, we could choose to retrain on varying amounts of samples, adding another dimension to our prediction space. Instead of only the timestep index and model index, we would now have multiple models to choose from.\n\n\n# Comments\n\n## 9. Discussions about scaling laws seem unrelated to the paper.\n\nThe organization of this paragraph made it seem like the scaling law applies to the distribution shift scenario, which, as the reviewer pointed out, is not the case. It belongs to the section where we describe the performance of a given model remaining constant (i.e., no distribution shift). \nWe will reorganize this paragraph by moving the scaling law discussion before the discussion about distribution shift.\n\n## 10. Other typos\nThank you for bringing these to our attention. We will correct them.", "claims_and_evidence": "- Proposition 3.1. This seems only tangentially related to the rest of the paper. \n- Proposed method UPF performs well empirically: the experiments seem sensible.", "methods_and_evaluation_criteria": "Proposed method: \n- I think the formulation of the problem is very reasonable (also an improvement on CARA). \n- I also think the Bayesian model is largely reasonable, though I am curious about ablations with / without $z_{shift}$. \n\nEvaluations:\n- Metrics:\n    - AUC for $\\hat C_\\alpha$, to estimate cost but aggregated over a sequence of values of $\\alpha$ (tradeoff between retraining and performance costs)\n    - Number of retrains compared to an oracle\n- Datasets and models:\n    - It is a little strange to use XGBoost as the model for tasks involving text (e.g. epicgames). \n    - Why do you choose between 188 models from huggingface for last-layer training for iWildCam? What happens if you don't?\n    - There could be more detail in how the datasets were split up into different datasets $D_t$. \n    - Otherwise it seems reasonable\n\nSection A.8.1 advertises something different from what it delivers. The section title and beginning sounds like it will address the impact of using a normal approximation to fit the Beta parameters in the method. However, what it actually does is compare using a Beta model to a Normal model.", "theoretical_claims": "I checked the proof of Prop 3.1. There appear to be errors that may affect the result. \n- I think there is double counting in (33). The quantity being minimized includes $pe(t_1,t_2)$ and also $pe(t_2,t_2)$. However, the summand in (33) should not contain more than one term for $pe(\\cdot ,t_2)$. \n- When going from (42) to (44), it seems (44) is missing a $pe(t_r^*,t_{r+1}^*)$ term.", "experimental_designs_or_analyses": "They look mostly reasonable; see Methods And Evaluation Criteria.", "relation_to_broader_scientific_literature": "There is a lot of previous work on distribution shift detection and diagnosis but this work addresses cost-aware policies for when to re-train a model.", "essential_references_not_discussed": "I am not aware of essential references not discussed, but I am also not familiar with other works that address this specific problem.", "other_strengths_and_weaknesses": "Strengths\n- This paper is clearly written. \n- The proposed formulation and method are very reasonable (with caveats explained elsewhere). \n- The experiments seem reasonable. \n\nOther weakness\n- I am struggling to think of a setting where this problem is realistic. My understanding is that in many industry settings, the question of when to retrain a model is not based on cost-aware modeling, but rather based on legitimate and important operational/organizational/procedural reasons that I imagine are costly and/or difficult to try to change. \n- A common problem in modeling under distribution shift over time is how much past data to train on. Here, this problem is not part of the formulation (which is fine).", "other_comments_or_suggestions": "- Eq 12 has an extra )\n- Discussions about scaling laws seem unrelated to the paper, as scaling laws are not about retraining on new data from different/updated distributions. \n- Line 232 right side: \"setting $\\delta = 0.5$ simply selects the decision that minimizes the expected total cost\" -- I assume you mean median\n- Comments in blue in the appendix proofs are helpful but you may want to distinguish them from the proof in other ways, since they will not be blue when printed in black and white.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tl3j3ixpBd", "reviewer": "Reviewer_afm3", "summary": "This paper presents a novel approach to maintaining the performance of open-world machine learning models operating in continuously changing environments. The primary focus is on addressing a critical challenge in such settings: determining when the model should be retrained. The objective is to balance the trade-off between minimizing retraining costs and maintaining model performance. The paper argues that existing methods become impractical when retraining costs are high. To overcome this, the authors propose a methodology that explicitly accounts for retraining costs while maintaining competitive model performance. The evaluation demonstrates that the proposed approach outperforms conventional methods in scenarios where retraining costs are a significant concern.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Weaknesses\n## 1. Although the paper provides a mathematical analysis for the proposed online objective, the objective itself appears relatively straightforward, as it primarily combines a standard loss function with retraining costs.\n\n We disagree with the statement that the objective is straightforward. The objective described by Eqn. 1 is not a standard loss function with retraining costs. The loss function described in Eqn.1 is parameterized by a binary vector $\\boldsymbol{\\theta}$ that is subject to some constraints.\nThis doesn't resemble any standard setup. If the reviewer is instead referring to the training loss used to obtain the parameters of the Gaussian r.v. in Eqn. 13, it is true that a standard loss is being used, but this is only a small part of our methodology, and cannot be characterized as the objective of our method. This would overlook the second part of our methodology, described in Section 4.2, which is central to our method and is how we form our prediction $\\boldsymbol{\\theta}$. \n    \n## 2. The performance improvements demonstrated in the evaluation are modest compared to traditional methods that rely on distribution shift signals for retraining, raising questions about the practical advantage of the proposed approach.\n\nWe disagree with the statement that the improvements over distribution shift methods are modest. On average, in Table 1, the distribution shift methods are $23\\%$ worse than ours, (or ours is $17\\%$ better on average). This is a significant improvement, especially considering that they are on average  $31\\%$ worse than the oracle, and the oracle is $20\\%$ better on average. We can include these summary statistics to better highlight the improvement of our method over these baselines. \n    \nMoreover, the experimental setting that we chose is beneficial to the distribution shift methods, as we stop the evaluation at a cost-to-performance ratio where the oracle finds that no retraining should be done. Since the distribution shift methods are entirely agnostic to any cost considerations, settings with a very high cost of retraining would be catastrophic to use. This can be easily inferred by extrapolating the results of the distribution shift methods in the top figure of Figure 2 (the green and red lines).\n\n## 3. Few sentences in the related works need a bit of clarification, for example \"Since the signal is designed to adapt a model rather than trigger a full retraining, these methods are not appropriate as retraining signals.\" I think \"as\" should replaces with \"for\" but I could be wrong\n\nYes, thank you for pointing this out. It is phrased a little awkwardly. We will rewrite line 101 from:\n   *Since the signal is designed to adapt a model rather than trigger a full retraining, these methods are not appropriate as retraining signals.*\nto:\n   *Since the signal for these methods is designed to adapt a model rather than trigger a full retraining, they are not appropriate to be used as full retraining signals*.\n   \n## 4. Why the results seems incremental when compare the proposed method with the traditional method even though the paper considers high training cost?\n\nThis is related to our previous response regarding the performance of distribution shift methods. We deliberately avoided scenarios with excessively high costs, where the cost-aware methods would easily determine that no retraining is necessary, while the distribution shift methods would have a very poor performance.", "claims_and_evidence": "The claims made in the paper are clear and convincing", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I did not check for correctness of any proofs.", "experimental_designs_or_analyses": "Experimental Design seems appropriate.", "relation_to_broader_scientific_literature": "Key contribution of the paper related to the machine learning algorithms design for real world scenarios where the data goes through major distribution shifts.", "essential_references_not_discussed": "References seems adequate.", "other_strengths_and_weaknesses": "## Strengths\n- The paper is well-written and easy to follow.\n- The approach is mathematically well-grounded, providing a clear and formal formulation.\n- The paper addresses a practical and significant research problem in open-world machine learning, making a relevant contribution to the field.\n\n## Weaknesses:\n- Although the paper provides a mathematical analysis for the proposed online objective, the objective itself appears relatively straightforward, as it primarily combines a standard loss function with retraining costs.\n- The performance improvements demonstrated in the evaluation are modest compared to traditional methods that rely on distribution shift signals for retraining, raising questions about the practical advantage of the proposed approach.", "other_comments_or_suggestions": "Few sentences in the related works need a bit of clarification, for example\n\"Since the signal is designed to adapt a model rather than trigger a full retraining, these methods are not appropriate as retraining signals.\"\n\nI think \"as\" should replaces with \"for\" but I could be wrong", "questions_for_authors": "Why the results seems incremental when compare the proposed method with the traditional method even though the paper considers high training cost?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to formulate the problem of when to re-train a deployed model and applies a novel reinforcement learning approach to forecast model failure.  The reviewers had generally positive opinions of the work (1 x accept, 2 x weak reject, 1 x weak reject).  The reviewer who leaned toward rejection had the primary criticism that the problem is inherently very difficult.  Yet, this reviewer still found the empirical validation compelling: "design of the experiments is done extensively and the presented results seem sound".  Overall, I find this paper formulates a reasonable approach to an ambitious yet important problem.  For subsequent versions, in addition to addressing the reviewers' concerns, please cite and discuss Hoffman et al. (2024): https://arxiv.org/abs/2405.13926.  I believe it is quite relevant and provides some complementary methodology (although I have not read all its details).
Decision: Accept (poster)