Self-Healing Machine Learning: A Framework for Autonomous Adaptation in Real-World Environments
Authors: Paulius Rauba
Nabeel Seedat
Krzysztof Kacprzyk
Mihaela van der Schaar
Venue: NeurIPS 2024 poster (2024)
Keywords: Model performance degradation
autonomous adaptation
large language models
Primary area: other
TL;DR: Self-healing machine learning is a framework which diagnoses the reason for model degradation and proposes diagnosis-based corrective actions

Abstract: Real-world machine learning systems often encounter model performance degradation due to distributional shifts in the underlying data generating process (DGP). Existing approaches to addressing shifts, such as concept drift adaptation, are limited by their *reason-agnostic* nature. By choosing from a pre-defined set of actions, such methods implicitly assume that the causes of model degradation are irrelevant to what actions should be taken, limiting their ability to select appropriate adaptations. In this paper, we propose an alternative paradigm to overcome these limitations, called *self-healing machine learning* (SHML). Contrary to previous approaches, SHML autonomously diagnoses the reason for degradation and proposes diagnosis-based corrective actions. We formalize SHML as an optimization problem over a space of adaptation actions to minimize the expected risk under the shifted DGP.  We introduce a theoretical framework for self-healing systems and build an agentic self-healing solution *$\mathcal{H}$-LLM* which uses large language models to perform self-diagnosis by reasoning about the structure underlying the DGP, and self-adaptation by proposing and evaluating corrective actions. Empirically, we analyze different components of *$\mathcal{H}$-LLM* to understand *why* and *when* it works, demonstrating the potential of self-healing ML.

Machine Learning: A Framework for Autonomous Adaptation in Real-World Environments
Paulius Rauba University of Cambridge pr5010cam. ac. uk
Nabeel Seedat University of Cambridge ns7410cam. ac uk
Krzysztof Kacprzyk University of Cambridge ck7516cam ac. uk
Mihaela van der Schaar University of Cambridge mv4726cam ac. uk
Abstract
Real-world machine learning systems often encounter model performance degrada- tion due to distributional shifts in the underlying data generating process (DGP). Existing approaches addressing shifts, such concept drift adaptation, are limited by their reason-agnostic nature. By choosing from a pre-defined set of actions, such methods implicitly assume that the causes of model degradation are irrelevant to what actions should be taken, limiting their ability to select appropriate adaptations. In this paper, we propose alternative paradigm overcome these limitations, called self-healing machine learning (SHMI... Contrary to previous approaches, SHMI. autonomously diagnoses the reason for degradation and pro- poses diagnosis- based corrective actions. We formalize SHML as an optimization problem over space of adaptation actions to minimize the expected risk under the shifted DGP. We introduce theoretical framework for self-healing systems and build an agentic self-healing solution H-LLM which uses large language models to perform self-diagnosis by reasoning about the structure underlying the DGP, and aiff-adaptttion by proposing and evaluating corrective actions. Empirically, we analyze different components of H-LLM to understand why and when it works, demonstrating the potential self-healing ML.
1 Introduction
Consider the following scenario: You are tasked with mon- itoring the performance of black-box model deployed in production. After some time, you notice that the predic- tive performance of ff has started degrade. What would be the appropriate action a you should take to ensure that the model's performance returns its prior performance levels: a1: re-train the model on subset of the data; change the type of the model used; az: remove discovered corrupted values; 04: add new covariates?
Stable performance
Drop Performance
Possible adaptation stalegees
Accurary
Environment change
Time
Figure 1: Different adaptation strategies a1...., as might result different per- formance after environment change.
Clearly, the answer this question is "it depends". Differ- ent actions might result in different behavior of the model over time, illustrated in Fig. If could pinpoint why the performance of the model has degraded, it could help us understand what actions are most promising, since we could select an action which would directly address the root cause of the problem.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

While we take this intuition for granted, state-of oftte-of-theearr techniques for handling model degradation do not reflect this line of reasoning and rely on pre-determined actions. such as model retraining [155] re-using old models [ 6 or other specialized methods 66 7 12 Such approaches share
a common, implicit assumption the reason for the degradation in model performance is irrelevant. We refer to this as reason-agnostic methods.
The practical implications of methods being reason-agnostic are quite concerning. By not con- sidering the causes for drop in
@ssel Healling Components
Has the peckoree ence of the mode decheascdP
itss. thar
Whht are reesors for the
II Diggooois
ML. Model
Monitoring
sereeee in per for menice?
performance, the corrective ac- tions are, essentially, shots in the dark. In high-stakes applica- tions like healthcare, finance, or policing, misguided adaptations can lead to real-world harms, such as inaccurate diagnoses, fi- nancial losses, ures. In some industries Suuuh as healthcare- this has resulted in avoiding automated model retraining altogether
What adaptation stratogle can model?
Destoy
m.. Aduppaaton
must offectio??
Testing
Figure 2: Our work introduces self-healing machine learning. A healing mechanism H. interacts with deployed model f. H contains four components: monitoring, diagnosis, adaptation. and testing. The overall goal of SHML is to find optimal adaptation actions to maximize the predictive performance of a model f.
system fail-
We propose self-healing machine learning (SHML. to overcome the limitations of reason-agnostic approaches. SHML equips MI. models with the ability to diagnose the reasons for performance degradation and take targeted corrective actions. We define a self-healing system as tuple (H. f), where f is a black-box model and H is a healing mechanism that can modulate the behavior of f. An example of H modulating is by deciding what data to use to re-train f. as illustrated in our introductory example. H contains four components: monitoring, diagnosis, adaptation, and testing (Fig. The goal of H is to decide what actions to take in response to model degradation which are chosen based on an adaptation policy that provides a mapping from diagnoses to actions. We therefore formalize the goal of H. as finding optimal actions under the shifted data generating process (DGP) which are sampled from an adaptation policy conditioned on a diagnosis (Sec. (.3). In our introductory example, the optimal action is taking action @1 (Fig. 1.. Building upon these insights, we propose the first self-healing ML algorithm, H-LLLM (Sec. 5 which generates diagnoses behind model degradation and suggests diagnosis-based adaptation strategies.
Significance beyond technical contributions. By enabling systems to autonomously diagnose and adapt to model degradation, we lay the groundwork for a new class of self- eelf-healing algorithms. We envision self-healing systems as crucial for high-stakes applications where optimal model performance is essential. We also believe this work has immediate practical relevance in high-stakes areas where model degradation is common, such as medicine [3 18), fraud detection [9] or finance (20).
Contributions. ! We identify fundamental limitations in existing reason-agoosiic adap- lation approaches that do not consider the reason for model degradation (Sec. 3.2) 2
We introduce the paradigm of self-healing machine learning and establish a theoretical foundation for finding adaptation actions with dlagnnois-guuide action sampling (Sec. 3.3
3 We propose the first self-healing ML algorithm H-LLM which reasons about
the causes of degradation and modulates the behavior of ML models (Sec. 5)
4 We
demonstrate the viability of SHML by studying why and when it works (Soc. 6].
2
Related work
SHML is most closely related to concept drift adaptation or specialized drift handling methods. We provide extended discussion on related work within each component in Appendix A
Concept drift adaptation. The field of concept drift adaptation focuses on developing algorithms to maintain the performance of machine learning models in changing environments. Such algorithms are predominantly proposed within the setting of tabular data. Most common adaptation techniques are re-training models on new data [ 21 24 re-using stored models [1 6 or obtaining new
data altogether 25 26). These approaches can be implicit, like continuous retraining, or explicit,
based on drift detection in data model error (5) 23 27 Because these approaches do not explicitly

---

incorporate the reason for model degradation, we refer to them reason agnostic. SHML diverges from common approaches by introducing the core idea of diagnosing the root cause to search for optimal adaptation actions.
Specialized drift handling, Techniques have also been developed adapt in the presence of various drift scenarios, such as sliding windows [10] or adaptive classifiers [1] (8/32) which "repair" concept drift [27]. However, these methods lack an explicit diagnosis mechanism and operate under fixed decision rules, which do not incorporate the root causes of degradation into the adaptation strategy. Similarly. works that aim to understand distribution shifts [33 351 or attribute shifts to specific variables through causal mechanisms [36] 37] provide valuable insights but do not offer comprehensive framework for adaptation. We note that some work could be in principle a part of some SHML system components (discussed in Appendix
3 Self-healing Machine Learning
This section introduces self-healing machine learning and its four components. We present the problem setting (Sec. B.1. explain current limitations (Sec. (.2). and outline the four stages of SHML
(Sec. 3.3
NOTE: Table T serves as a guide for navigating the paper.
Component
Definition
Methodological contribution
Expertmental contribution
Main practical impliiaaiinss
Monitoring
Fgg
Sec
rebust models agsinst five positive drift detection (Sre
Diagnosis
Eq
Def
Def
Prop-
Sec
0.4
Esstablished framework reason about why madels degrade (Sec.
Adaptation Testing
Asmp
Targeted sduptation by identifying
Eq
Def
S
Sec.
0.0
Principted framework to evaluate actions (Sec. 6.0
Self-heallig ML
vall-haaling paradigm
acllessing prior limitarions
5.2 self-healing system (See.
Table I: A summary table of self- eff-hheling machine learning and its four stages, providing links to relevant sections and serving as navigation guide for the paper.
3.1 Model degradation over time
Preliminaries. Let X' and y denote the input and output spaces, respectively, and let Pe denote the data distribution over X' 3 at time step [T]. At each 4, observe batch of data )) ()))))))))))))))))))))))
Pht where n, = in the streaming setting and n, > the batch setting. We will drop the superscripts where clear from context.
D. - ( yt
The goal to learn a sequence of functions (Se FFT that minimize the cumulative risk:
R(S fT) Σ (,
where F is a function class and (: x y R20 is loss function.
In the fime-invarian setting where P. P Vt [T]] the goal reduces learning a single function f
F that minimizes the risk R(f) = (x) When P is unknown, f is often approximated by minimizing the empirical risk on training set ((x(), ( } P" However, when Pe evolves over time, the optimal predictor ft ε arg rigmin Ep, ((((()(()))))) changes across time
step Failing to adapt ft in this time-varying setting leads to model degradation, as the learned function becomes increasingly suboptimal w.r.t. the current data distribution.
3.2 Limitations of existing approaches in adapting changing environments
Maintaining stable model performance in the presence of a changing environment poses unique challenges. As the optimal predictor fi evolves over time, the estimated predictor should also adapt.
(8) ) and minimize the empirical Ideally, we could obtain a large batch of data D {x((') V2-
risk over this dataset. However, this is often impractical due constraints such as (i) ground-truth labels not being immediately available [38); (ii) the streaming setting, where each new batch contains
only one data point (39]; (iii) gradual shifts, where past data remains relevant [40]: or (iv) the presence of corrupted data in new batches (1)
'Three primary mechanisms through which Pt varies are covariate shift: P:(x) # (x)(()) A (((x)) Pe-1 (ylx), label shift: Pv))) Pe+i (xlyy) Pet (xly), and concept drift: Pt(y)x) Pp+x())x).
3

---

To address this, the research community has developed specialized methods determining the appro- priate corrective actions in such drifts. As discussed Sec. 2 these methods primarily execute pre-defined actions upon detecting change. such as model retraining 115 re-using old models 168, or other more specialized methods 6 12] However, such methods are reason-agnostic, disregarding valuable information that inform better adaptation actions. Consider an illustrative
example: suppose a batch of new data arrives, but due to a sensor malfunction [42], 80% of the labels become corrupted and are independent of the input for that batch only, Naively retraining the model on this noisy batch would degrade its performance. This is because this strategy implicitly assumes:
((x [<[ y)] (x.x))- [<(f(()),))] (
where Si-1 is model trained on Di+1- Relying on this assumption results in worse performance than doing nothing. Similarly, re-using old models assumes that the past data distribution is still relevant:
y) (x..)))P...44 [()=))))..II((((()..))))()))) [x))(xx)
(3)
for some k > O. Similarly, this might result in suboptimal performance due the nature of the shift. Each adaptation method discussed in Sec. has such implicit assumptions about the model or DGP.
By not taking into account the reason for the model degradation (such as corrupted data), the adaptation strategy is defaulting to suboptimal corrective actions. While any adaptation strategy inherently involves some assumptions about the relationship between the predictor and the data, we would like to prioritize making informed assumptions. As we discuss in Sec. 3.3 a key source of such information is diagnosing why the model's performance has dropped.
To address the reason-agnostic nature of such adaptation methods, we propose paradigm shift called self-healing machine learning (SHML), where deployed models autonomously diagnose the reason for degradation and take diagnosis guided corrective actions.
Takcaway. Existing adaptation methods make implicit, pre-defined assumptions about the nature of model degradation. Neglecting its reason can lead to poorly chosen actions.
3.3 The four stages of self-healing machine learning
Self-healing machine learning is a framework for autonomously detecting, diagnosing, and correcting performance degradation in deployed ML models. It aims to maintain model performance in changing environments without constant human intervention. The motto of self-healing ML is "understanding your problem is half the solution" (and the most important half). SHML system is defined by a tuple (H. f), where f X + ) is the deployed machine learning model we aim to heal (i.e., the function that makes predictions on input data), and H. is healing mechanism that interacts with the environment and acts upon the model f by proposing and implementing actions, such as selecting when to retrain a model, what data to use or how to change the input data before making predictions. Thus, H can modulate the behavior of the deployed model f.
Self-Healing Machine Learning in nutshell.
Self-healing ML contains four components: monitoring, diagnosis, adaptation, and testing. After these steps, the best action is implemented on the ML model, illustrated Fig. 2.
I. Monitoring. The first step is the delection of degradation, potentially due to a shift in the data distribution. We formalize this as monitoring component Hs that takes as imput the sequence of data batches (D,M..: up time and outputs 8, e [0. 1], indicating the likelihood of model degradation. Formally,
HMr x)* [0, 1].
(4)
where higher values of & indicate a greater likelihood of a shift.
II. Diagnosis. The diagnosis component HD detects the reason of degradation. Tt takes data batches (D.) up time along with any available contextual information e C (e.g. back- ground knowledge), and outputs distribution ( A(Z) over a space of possible reasons
Ho: : (X' x y x C)* > A(Z).
(S)
2 represents the finite space of possible reasons of the shift and is a stochastic vector.
III. Adaptation. The adaptation component is a policy ı that outputs distribution over actions. Given a diagnosis vector Ç, actions e A are selected from finite space A by:

---

T-K)) where æ A(2) - A(A).
(6)
Fach action modifies f. We denote the model used at time t, selected by action a, fa. IV. Testing. The testing component Mr evaluates each action A relevant distribution and outputs a performance measure:
Hr : AxP
(7)
Objective. The goal of sell-healing ML is to select the optimal action a' that minimizes the expected loss B[E(SF vx).)]] the data distribution Ps:
a" arg arrgin ((x.)))P,(((fE) (x))) (x),2)).
anA
where f:3 x R20 loss function, and fe denotes the model selected by action to be used at time t. The action a is selected according to the adaptation policy =(I(); which maps the diagnosis vector Ç (a distribution over possible reasons for degradation) to a distribution over actions.
Suppose the following motivating example to guide the notation above.
Illustrative example. Consider a deployed ML model fi for predict-
Time
ing diabetes. The monitoring component His detects a significant drop in performance, with 0.95 (Eq. 4 The diagnosis com- ponent HD outputs three most likely reasons 21, 23 Z, with ((z1) 0.95 for data quality issues, ((z2) 0.03 for concept drift, and ((2s) = 0.02 model overfitting (Eq. 5). Based on, C. the adaptation policy и samples two actions an, a2 ~ x+IC) (Fq. G a1: remove detected biologically implausible values (e.g. Age > 200) and retrain fa; az: include interaction terms between features to capture uon-linea,isies The testing component Hr evaluates the adapted models fe and f, on new incoming data (Eq. 7 and selects a1 due to lower estimated loss.
Hl
HM
M
H
A
The primary insight of SHML is that the action π (-K) should be based on the diagnosis Ç. which is a distribution over possible reasons Z for model degradation. In contrast, standard approaches (Sec. optimization
H2
3.2) assume that 1 Ç. SHML formalizes this problem over a space adaptation actions-we we aim to find the optimal actions to take each time the model f degrades, with these actions chosen by the policy ı of the self-healing system H (Fig. 3). Different policies иi and π2 might propose different actions in response to the same performance drop. While the diagnosis { informs the policy, we do not assume it is necessarily useful. Since Ç ()))& is probability distribution, it can encode no knowledge by being uniform over the diagnosis space: ((z) 2 Vz Z. These components and their interactions between two time points ( and +++ shown in Fig.
The self-healing mechanism H. modulates the function f via four stages. The chosen adaptation action is implemented onto the function f at the next time step.
Figure 3:
The effectiveness of the adaptation actions depends on the diagnosis, i.e. how well can we identify the root cause. Therefore, we turn to the diagnosis component next.
Takeaway. SHML framework which selects actions based on the reason for model degradation. It contains four stages: monitoring, diagnosis, adaptation, and testing.
An analysis of the properties of self-healing diagnosis
Self-healing systems have the unique property of having a diagnosis stage. But what constitutes good diagnosis? this section, we analyze the properties of self-healing diagnosis and establish its connection to the performance of adaptation actions.
To effectively use diagnosis information to guide the search for adaptation actions, we require way to quantify the usefulness of diagnosis. We propose three desirable properties for such measure: (i) concentration: it should favor diagnoses that provide more information, i.e. assign higher probabilities to fewer possible reasons; (ii) sensitivity: it should be sensitive to changes in the diagnosis distribution, such that small changes probabilities would result in small changes in the measure; (iii) maximum uncertainty: it should reach its maximum value when the diagnosis
5

---

distribution is uniform, indicating no knowledge about the reason for degradation. Therefore, we propose using the entropy of the diagnosis vector as a useful proxy for quality which satisfies all three properties. Because entropy measures uncertainty, we refer to this as the certainty of the diagnosis. Definition 1 (Certainty of the Diagnosis). Let be the finite space of possible reasons for degradation and (( be the diagnosis space. The certainty of diagnosis Ç ((2) in a self healing machine learning system is measured by its entropy H(S), defined as:
= [(())=-[<(a)oog (o)
where {(=) is the probability of reason under the distribution Ç-
The link between diagnosis quality and adaptation performance highlights the importance of obtaining informative diagnoses in SHML. Building upon these concepts, we define the optimal diagnosis: Definition 2 (Optimal Diagnosis). The optimal diagnosis 5* is defined as:
-" arg
arg min (e-s(t))R(a))) (<A(Z)
(10)
where ( is the diagnosis space, x(-() is the conditional distribution over actions induced by diagnosis vector and R(a) denotes the risk of fi associated with action A.
This formalizes the intuition that the best diagnosis is the one that leads to the best adaptation actions, on average. To characterize the properties of this optimal diagnosis, we introduce an assumption about the structure the adaptation policy:
Assumption (Independent actions). We assume that a+1() has a hierarchical structure. First, reason z E Z is sampled according to the diagnosis Ç: 2 C. Then, an action is sampled conditioned 1. TT-() can then be described as
on this reason, ~ T--22),, where z (2) such that z (z)=1. the following mixture.
x(aks) = Σ [=(al='((())
(11)
ZE2
Ç are the mixture weights and (x(1²²) 2 Z} are the mixture components.
Under this hierarchical structure, we can prove a useful property of the optimal diagnosis: Proposition 1. Under Assumption
7 the optimal diagnosis Ç* has a zero entropy, i.e., H((") ) = O.
Proof in Appendix E To ensure that the optimal diagnosis is well-defined, we also prove its existence under mild assumptions:
Proposition 2 (Existence of Optimal Diagnosis). Suppose that the action space A is compact subspace of Rn and R is continuous. Then there exists at least one optimal diagnosis Ç*.
Proof. The expected risk [R(a) is continuous function of ( by the continuity of R and the compactness of A. Since the diagnosis space A(Z) is also compact (being a probability simplex), the extreme value theorem guarantees the existence of minimizer Ç"
Takeaway. The existence of an optimal diagnosis establishes foundation for designing algorithms that can accurately approximate it in practice. By identifying the underly- ing reasons for performance degradation, high-quality diagnosis enables self-healing system to take the most offective adaptation actions.
5 Building self-healing systems: H-LLM
This section outlines the challenges of building SHML systems (Sec. 5. describes how LLMs can address these challenges (Sec. 5.2). and introduces the first self-healing system, H-LLM (Sec. 5.3)
5.1
Unique challenges of building self-healing systems
Implementing SHML systems (Sec. 3.3 poses unique challenges in diagnosis and adaptation.
Challenges diagnosis. Discovering the reasons for model degradation poses significant practical challenges because: (i) the space of possible reasons Z is often poorly defined intractable to

---

Diagrocis
Evstence
Comfidene (1-10)
There the
nusiman values many variables the new daraset siguificanlly higher than dataact, suggssting
the proence cutlier.
Incornect data transformatons have
The mean valzes for mane varsalles the new dataeet are significnnly higher than the ol datase., sugpesting that the
lees applied
Jasu may have bren insorecally canderme
Them are dars entry ences
The minimum value for Inautin the new dhinaer is negrtien, which is not posoble in real-sorid contrst
10
Table 2: Example diagnoses suggested by H-LLM. The system proposes diagnoses and suggests evidence for the diagnosis. A post-hoc relative confidence score, constructed using the "evvdence" column, helps to guide which diagnoses to pay most attention to while designing adaptation policies.
specify exhaustively in real-world scenarios; and (ii) assigning well-calibrated probabilities to reasons for model degradation difficult due both the epistemic and aleatoric uncertainty that exists in real-world environments. This makes it difficult to approximate the optimal diagnosis (Def. 3
Challenges in adaptation. The adaptation policy я (Eq. 6) requires selecting optimal adaptation actions a based on the diagnosis Ç. This is challenging because (i) it requires reasoning about how actions interact with diagnoses; and (ii) the space of adaptation actions may be extremely large in practice, making it difficult to find the optimal action (Eq.
5.2 Language models to empower self-healing
We posit that LLMs have the potential to satisfy many of the required properties of self-healing components because of the following capabilities: (i) Hypothesis proposers. LLMs known to be "phenomenal hypotheses proposers" 433 which are required to hypothesizing diagnoses of ML model performance degradation; (ii) Contextual understanding. L.MSs have been pretrained with vast corpus of information and hence have extensive prior knowledge around different contexts and settings 44 45); (iii) Language model agents. Language models can work as agents within larger system 46 47 which is required to actively interact with a deployed model, trigger and implement changes. We therefore see LLMs as capable proxies for different self-healing components.
5.3
Design of H-LLM
We instantiate the healing mechanism H with an LLM using its useful properties (Sec. 5.2) to address the practical challenges of designing SHML systems (Sec. 5.T..
H-LLM a mutshell.
H-LLM is the first SHML algorithm that modulates the behavior of f following Fig.
1. Monitoring, We statistical drift detection algorithms monitor model degradation from previous time points 29 39 48 Diagnosis triggered if a shift is detected.
II Diaguosis. Upon delection, we use a pre-defined prompt template to obtain information about the datasel before and after the diagnosis. The prompt template gives us numerical insights into how the dataset has changed and includes covariate information before and after the shift, together with other numerical details. We denote this prompt E: DD' De obtain an information vector v. Using v and a sain-of-thought (CoT) module with self-reflection, H-LLM generates Io candidate reasons for degradation {2.)e1 I(-lv) via Monte Carlo (MC) sampling with associated confidence scores. As before, this oblained by following pre-defined prompt templates conditioned on the obtained information (e.g. " "Suggesl (self.n) possible reasons why the model might have failed on the basis of the issues presented"). These candidates form an empirical diagnosis vector č, approximating the optimal diaguosis <*. Table 2 illustrates diaguoses generated by H-LLM.
extractor function
111. Adaptation. Conditioned the empirical diagnosis distribution & H-LLM generates m candidate adaptation actions fa t(-k) via Co'T-based MC sampling. This approximates sampling from <+I(") ) (Def. G). The actions sampled from 1 are textual representations, use an interpreter function execute each a f.
1V. Testing, The sampled actions are evaluated on an empirical dataset (Def. 7 and the empirically optimal action â" = argmin,, R(a) is implemented. Limited access to the shifted
DGP complicates evaluating R(a), but it can be approximated with empirical data Drsa by using backtesting window, continuously incoming data, or historical data (Appendix B.4].
Goal. This procedure aims to approximate the optimal action (Def. g These actions are orchestrated by an orchestrator component in H-LLM which can navigate between these steps.
7

---

Appendix B provides an extended discussion of H-LLM, including the algorithm, prompts, ex- amples, and outputs. The following table links H-LLM with the theoretical framework.
Componeas
Theeey
R--LM
Approximation
Monitoring
Monitor for degradiatio
Drift detection algorithm
Any detection algorithm
Diagnoeis
Optimal diagnosie <* A
Empirical diagnosis via LLM ?
MC sampling LLM
Adaptation
Sample action =((()
Sample actions via LLM a 1((()
MC sampling with LLM
Teesing
Evaluate each on Pt
Evaluate each on Dreat
Any suitable dataset
Table 3: A comparison of the theoretical components, their implementation, and their approximations.
6 Experimental viability studies
The previous sections constituted the primary contribution of paper-establishing SHML framework. The goal of this section is to provide a viability study by analyzing different components of SHML. We conduct six viability studies
Experimental setup. We desire to meet two properties: (i) have full control of the DGP to vary experimental parameters; and (ii) need benchmark against existing adaptation methods (Sec. 2) which are predominantly tabular-base based. Therefore, we simulate a diabetes prediction task [49] 51 based on the setup in Sec. 3.1 We predict diabetes Y₁ e {0,1] at each time point t for set of n observations, generated according to (changing) pre-specified DGP log P(V,-0)X.))
a, ΣKEK Xr, et. where K includes relevant parameters such as Age BMI, Br, are time- varying covariates and e, (0, o²) is a noise component. For evaluating H-LLM actions, use backtesting window-a representative sample of the shifted distribution obtained after detecting the change but before deploying the adapted model (Sec. B.4 Details provided in Appendix C
6.1 Viability study I: Adaptation in the presence of model degradation
Setup. We aim to empirically demonstrate the limitations of existing approaches in adapting to changing environments (Sec. 3.2 We benchmark H-LLM against four common drift adaptation methods: (i) new model retraining on post-drift data, (ii) partially updating models with new data; (iii) Ensemble methods by re-using old models, and (iv) No retraining of the models [.. At time t, we introduce a sudden, single intervention by changing the DGP parameters and corrupting a percentage r of k columns. Table 4 shows the performance of different methods across and k.
Number of corrupted culumns (r 0.05)
Corruption percentage (k = 3)
Method
4
6
8
0.01
0.02
0.05
0,10
0.20
retraining
0.44 0.02
0.44 0.02
0.45
0.45 +
0.43
0.44
0.44 0.02
0.45 0.02
0.46
Partially Updating
0.71 + 0.02
0.69
0.67
0,54 0.02
0.06
0.04
0.02
0.00
0.66
0.02
0.02
New model training
0.70 a 0.02
0.69
0.02
0.67
a 0.02
0.50
a 0.02
0.77 4: 0.02
0.74
0.69
a.69
0.66 a
0.61 2002
Ensemble Method
0,70 0.02
0.69
0.02 0.67
0,02
0,50 0.02
0.77
0.02
0.02
0.69
0.66 0.02
0.61 0.02
H--LM
0.93 1 0.01 0.87 2 0.01 0.79
2 0.02 0.68 1 0.02
0.95 a 0.01
0.04
0.02
0.70 0.002
0.02
Table 4: Accuracy of a deployed model f upon an intervention which changes the DGP and corrupts
7 percentage of k columns. Error represents standard deviation. 1 is better.
Discussion.
The performance of f degrades if the corrupted columns are not handled appropriately, such as removing or inputting the corrupted data. Defaulting to standard techniques of adapting to changed environment results in poor performance. H-LLM diagnoses issues by observing that some values have drifted too much from their original values and the DGP has changed. One of the proposed adaptation strategies is to remove samples which were estimated to be corrupted, and re-training the model on the remainder of the data. This results in superior performance.
Takeaway 1. Diagnosing the root cause of degradation can guide better adaptation actions.
6.2 Viability study II: Adaptation across datasets
Setup. We aim to empirically analyze whether SHML can provide benefits across different datasets.
We cover five different datasets: Airlines 522 Poker 53 Weather (54), Electricity (55) Forest Type
"Code can be at: http:///w...gtoooooooooo/o/////a htrtee/ononnnoooooososs///gigt hoa//aaaderrccaarra////sSf....
or https://gitthub
8

---

[56. We simulate real-world unexpected degradations by assuming lagged labels and corrupting features at test time and evaluating models for different datasets (Table
Discussion. Ground-truth labels are often not immediately available [88. a core feature of many streaming settings. We evaluate how H-LLMM compares existing approaches in such scenarios. Across five datasets with different characteristics, H-LLM consistently outperforms traditional adaptation methods by adapting fe at each time point t. Therefore, SHML's ability identify and decorrupt features provides a robust adaptation strategy across varied data distributions and schemas.
Takeaway 2. Identifying the root cause and restoring features dacn provide consistent benefits across datasets.
Accuracy when lo
5 weather
Accuracy when
Method
airlines
poker
coyBye
poker
weather
elec
coy Type
retraining
0.53
0.57
0.05
0.66
004
0.51
0.53
0,7 0.01
0.04
0.67 = 0.0
0.58
Partially Updatinn
0.53 2 0:01
B.48 a
0.57
: 0.05
0.66
0.51
+ 6:03
0.53 0.01
047
001
0.59
0.94
0.67 = 003
0.58
New mudel training
0.54 0.02
0.99 000
0.56
a 0:03
0.66
+ 0.05
0.51
0.02
0.53
0.02
0.47
0.00
0.60
= 0.02
0.67 = 0.03
0.58 + 0.02
Eesemble Method
0
0 0.1
0.58 0.06
0.50
0.52 0.02
0.52 0.01
046 0.00
0.59
0.05 0.6 e 0.04
0.59
R-ILMM
0.56 a 0.00
0.8 0.66 a 0.02 0.72
0.73 0.00
0.56 0.00
0,03 0.66 0.02 0.72 a 0.01 0.73 a 0.00
Table 5: Accuracy of various methods on different datasets with corrupted columns and varying corruption values. Error represents standard deviations of five runs. 1 is better. We simulate concept drift by corrupting the test set as follows: randomly selected k features and multiplied their values by a corruption factor r. H-LLM identifies that the test data has been corrupted and perform a relevant transformation decorrupt the value to its original feature space test time.
6.3 Viability study III: Monitoring
Average Recovery Time
Setup. We use the same setup experiment I and vary the drift detection threshold which influences the sensitivity of a detection system to changes in the DGP. Low values mean high sensitivity, and high values mean low sensitivity [57]. We measure average recovery time for H-LLM to return recover from degradation and post-intervention accuracy, H-LLMs
Average Recovury Tima
2.
10
average performance after intervention. Fig. 4 shows this relationship.
detection threshold
Post mest-intervention Accuracy
Discussion. Intuitively, one might expect earlier drift detection (lower threshold) to consistently yield faster recovery and higher accuracy. In reality, concept drift algorithms often struggle with false positives which can result worse model performance because of unnecessary re-training [5,58]. Self- ML exhibits greater robustness to these false pos- itives, as any action will be implemented only if it outperforms doing nothing. This contrasts with traditional systems which would automat- ically trigger the selected action. In Fig. 4 this represents the higher post-intervention accuracy with smaller thresholds.
Pust Pust-inurveetton Accuraty
0.8
0.
0.5-
10
Drift detection threshold
Figure 4: Lower drift detection thresholds can benefit SHML.
Takeaway 3. SHML has greater robustness implementing poor adaptation actions.
6.4 Viability study IV: Diagnosis
Quality diagrosis based on magnhice
Quallty aggooss laced on actent comuptisn
Setup. We evaluate how well self-healing systems identify the root causes of problems. We corrupt a proportion of observations (corruption coefficient) by multiplying their values by factor (outlier factor) and see if the H- LLM detects issues related to these factors. We output probability distribution over diagnoses of which variable is corrupted. Knowing the true corrupted variable, we measure the difference between the distributions using KL- Divergence, with lower values indicating closer matches to true corruption. A uniform diagnosis baseline represents random guessing. Fig. Sishos these differences.
B.Beeoperce
Outter Factor
Comupt Confricen!
Unistorme slagreds
Figure 5: KL- KL-Divergence between esti- mated probabilities of which variables are corrupted, and true probabilities, based on outlier factors and corruption coefficients. 1 is better.
Discussion, As the outlier factor and corruption coefficient increase, making data issues more apparent, H-LLM assigns higher probabilities to the corrupted variables. Thus, the diagnosis accuracy improves as the problem becomes more evident.
Takeaway 4. The quality of the diagnosis improves when the issues become more apparent.

---

6.5 Viability study V: Adaptation
Accurasy per actien
90%
Accuracy by compptiin
Accuracy by test set
(ishwei det the bed
Setup. We study the sensitivity of SHML adap- tation actions by examining how well actions perform based on (i) the number of corrupted values and (ii) the size of the backtesting dataset. Fig. (666 shows this relationship.
80%
70%
60%
150%
2%
5% 10% 20% 50% 75%
10
10a
10a
10a 10"
Range of values comupled
Size of backte window
Discussion. As more values are corrupted, adap- tation actions become more concentrated and less effective. With larger backtesting dataset, actions are more spread out. This suggests (i) action evaluation is more reliable with non-corrupted data and (ii) larger backtesting windows help in selecting better adaptations.
Figure 6: Model f accuracy for actions with vary- ing corruption range and backtesting window size.
Takeaway 5. large test dataset and high-quality data can improve adaptation action selection.
6.6 Viability study VI: Testing
Effect of backtesting window
Setup. We study the importance of the testing component (Eq.
7 by evaluating H-LLM suggested actions with and without the testing phase (backtesting window) and comparing their accuracies. Fig. shows this relationship.
2 75%
Corruption
50%
20%
10%
2%
Discussion. Having a dataset to evaluate actions significantly improves the self-healing process. We see that better data quality results in more reliable adaptation policies.
50%
60%
70%
80%
90%
Accuracy of policy
Figure 7: Accuracies of optimal ac- tions with and without testing com- ponents
Takeaway 6. The testing component is important to effectively evaluate the proposed actions.
6.7 Other studies
We provide further experiments in Appendix D Our framework shows strong performance with lower warm-start parameters and increasing benefit as data degradation becomes more severe (Sec. D.4p. A component-wise ablation analysis (Sec. D.6) reveals each stage of SHML is essential. Extended benchmarks (Sec. [D.5) and model agnostic evaluations (Sec. [D.7] demonstrate consistent improvements across different adaptation approaches and ML architectures.
7 Discussion
Algorithms hold significant decision-maiee making power in high-stakes applications, yet little has been done to ensure their optimal performance. This work presents major leap towards that goal. By enabling systems to autonomously diagnose and adapt to new environments, we aim to create a wave of self-healing systems beneficial to both the ML community and society. Our theoretical framework (Sec. 3.3 builds the foundation for the development of self-healing theory, such as optimal adaptation or diagnosis methods, and our viability study shows the potential benefits of SHML. Our largest contribution is formalizing this field-we we hope to spur new theoretical developments and encourage the adoption of such systems in critical domains like medicine [3 and finance (20)
Limitations. SHML's success relies on accurate root cause identification and finding effective adaptation policies which could pose challenges in some complex, real-world settings (Sec. (.11) Furthermore, the prioritization of adaptation strategies also not trivial. Currently, H-LLMM primarily looks for subgroup-leve issues. We see future work tackling all areas of self- self-healing ML: finding better diagnosis strategies, improving adaptation selection, and enabling better testing of actions in the presence of changing environments.
Broader impact. Because SHML could empower many positive technologies, could also be misused to amplify the impact of more problematic systems, such as surveillance technologies.
Acknowledgements
We would like to thank the anonymous reviewers, Julianna Piskorz, Katarzyna Kobalezyk, Haris Mackevicius, and Andrew Rashbass for their helpful feedback. PR is supported by GSK, KK is supported by Roche, NS by the Cystic Fibrosis Trust. This work was supported by Microsoft's Accelerate Foundation Models Academic Research initiative.
10

---

References
[I] Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12): 2346-236 2263, 2018.
[2] Haider Raza, Girijesh Prasad, and Yuhua Li. Adaptive learning with covariate shift-detection for non-stationary environments. In 2014 14th UK Workshop on Computational Intelligence (UKCI), pages 1-8. IEEE, 2014.
[3] Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. Advances Neural Information Processing Systems, 32, 2019.
[4] Firas Bayram, Bestoun S Ahmed, and Andreas Kassler. From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge-Bade Baased Systems, 245:108633 2022.
[5] Supriya Agrahari and Anil Kumar Singh. Concept drift detection in data stream mining: A literature review. Journal of King Saud University- Computer and Information Sciences, 34(10): 2022.
[6] Paulo Mauricio Gonçalves Jr and Roberto Souto Maior De Barros. Red: recurring concept drift framework. Pattern Recognition Letters, 34(9):
2013.
[7] Cesare Alippi, Giacomo Boracchi, and Manuel Roveri. Just-in-time classifiers for recurrent concepts, IEEE transactions on neural networks and learning systems, 24(4):620-634, 2013.
[8] Stephen H Bach and Marcus A Maloof. Paired learners for concept drift. In 2008 Eighth IEEE International Conference on Data Mining, pages 23-32. IEEE, 2008
[9] Joao Gama and Petr Kosina, Recurrent concepts in data streams classification. Knowledge and Information Systems, 40:489-507, 2014.
[10] Gerhard Widmer and Miroslav Kubat. Learning in the presence of concept drift and hidden contexts. Machine learning, 23:69-101, 1996.
[11] Geoff Hulten, Laurie Spencer, and Pedro Domingos. Mining time-changing data streams. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 97-106, 2001.
[12] Pingfan Wang, Nanlin Jin, Wai Lok Woo, John R Woodward, and Duncan Davies. Noise tolerant drift detection method for data stream mining. Information Sciences, 109:1318-1333, 2022.
[13] Daniel Vela, Andrew Sharp, Richard Zhang, Trang Nguyen, An Hoang, and Oleg S Pianykh. Temporal quality degradation in ai models. Scientific reports, 12(1): 1654, 2022.
[14] Zachary Young and Robert Stecle. Empirical evaluation of performance degradation of machine learning-basee based predictive models-a case study in healthcare information systems. International Journal of Information Management Data Insights, 2(1 ((1):10000 100070, 2022.
[15] George Alexandru Adam, Chun-Has Hao Kingsley Chang, Benjamin Haibe-Kains, Kains, and Anna Goldenberg. Hidden risks of machine learning applied to healthcare; unintended feedback loops between models and future data causing model degradation. In Machine Learning for Healthcare Conference, pages 710-731. PMLR, 2020.
[16] Berkman Sahiner, Weijie Chen, Ravi K Samala, and Nicholas Petrick. Data drift in medical machine learning: implications and potential remedies. The British Journal of Radiology, 96 (1150))20220878, 2023.
[17] Ayne A Beyene, Tewelle Welemariam, Marie Persson, and Niklas Lavesson. Improved concept drift handling surgery prediction and other applications. Knowledge and Information Systems, 14:177-996, 2015.
11

---

[18] Hamish Huggard, Yun Sing Koh, Gillian Dobbie, and Edmond Zhang. Detecting concept drift medical triage. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Reirieval, pages 1733-1736, 2020.
[19] Tung-Duong Mai, Kien Hoang, Aitolkyn Baigutanova, Gaukhartas Alina, and Sundong Kim. Customs fraud detection in the presence of concept drift. In 2021 International Conference on Data Mining Workshops (ICDMW), pages 370-379. IEFE, 2021.
[20] Indré Zliobaite, Mykola Pechenizkiy, and Joao Gama. An overview of concept drift applications. Big data analysis: new algorithms for a new society, pages 91-114, 2016.
[21] João Gama, Indré Zliobaitè, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-37, 2014.
[22] Ralf Klinkenberg and Ingrid Renz. Adaptive information filtering: Learning in the presence of concept drifts. Learning for text categorization, pages 33 33-40, 1998.
[23] Joao Gama, Pedro Medas, Gladys Castillo, and Pedro Rodrigues. Learning with drift detection. In Advances in Artificial Intelligence-SBI 2004: 17th Brazilian Symposium on Artificial Intelligence, Sao Luis, Maranhao. Brazil, September 29- 19-Oootber 2004. Proceedings 17, pages 286-295. Springer, 2004.
[24] Manuel Baena-Garcia, José del Campo-Ávila, Raul Fidalgo, Albert Bifet, Ricard Gavalda, and Rafael Morales-Bueno. Early drift detection method. In Fourth international workshop on knowledge discovery from data streams, volume 6, pages 77-86. Citeseer, 2006.
[25] Bartosz Krawezyk, Bernhard Pfahringer, and Michat Wožniak. Combining active learning with concept drift detection for data stream mining. In 2018 IEEE international conference on big data (big data), pages 2239-2244 IEEE, 2018.
[26] Nicolás Astorga, Tennison Liu, Nabeel Seedat, and Mihaela van der Schaar. Partially observable cost-aware active-learning with large language models. In The Thirly-Eight Annual Conference on Neural Information Processing Systems, 2024.
[27] Ben Halstead, Yun Sing Koh, Patricia Riddle, Russel Pears, Mykola Pechenizkiy, Albert Bifet, Gustavo Olivares, and Guy Coulson. Analyzing and repairing concept drift adaptation in data stream classification. Machine Learning, 111(10):3489-355 3523, 2022.
[28] Dewan Md Farid, Li Zhang, Alamgir Hossain, Chowdhury Mofizur Rahman, Rebecca Strachan, Graham Sexton, and Keshav Dahal. An adaptive ensemble classifier for mining concept drifting data streams. Expert Systems with Applications, 40(15):5895-599 -59669, 2013.
[29] Anton Dries and Ulrich Rückert. Adaptive concept drift detection. Statistical Analysis and Data Mining: The ASA Data Science Journal, 2(5-6):3113277 2009.
[30] Tinofirei Museba, Fulufhelo Nelwamondo, Khmaies Ouahada, and Ayokunle Akinola. Recurrent adaptive classifier ensemble for handling recurring concept drifts. Applied Computational Intelligence and Soft Computing, 2021:1-13, 2021.
[31] En Yu, Jie Lu, Bin Zhang, and Guangquan Zhang. Online boosting adaptive learning under concept drift for multistream classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16522-16530, 2024.
[32] En Yu, Yiliao Song, Guangquan Zhang, and Jie Lu. Leapt-t--ddapt: Concept drift adaptation for hybrid multiple streams. Neurocomputing, 496:121-130 2022.
[33] Jiashuo Liu, Tianyu Wang, Peng Cui, and Hongseok Namkoong. On the need for a language describing distribution shifts: Illustrations tabular datasets. Advances in Neural Information Processing Systems, 36, 2024.
[34] Andrés R Masegosa, Ana M Martínez, Darío Ramos-Lópezz Helge Langseth, Thomas D Nielsen, and Antonio Salmerón. Analyzing concept drift: A case study in the financial sector. Intelligent Data Analysis. 24(3):665-8)) 688, 2020.
12

---

[35] Sean Kulinski and David I Inouye. Towards explaining distribution shifts. In International Conference on Machine Learning, pages 17931-17952. PMLR, 2023.
[36] Kailash Budhathoki, Dominik Janzing, Patrick Bloebaum, and Hoiyi Ng. Why did the dis- tribution change? In International Conference on Artificial Intelligence and Statistics, pages 1666-1674, PMLR, 2021.
[37] Haoran Zhang, Harvineet Singh, Marzych Ghassemi, and Shalmali Joshi. " "why did the model fail?": Attributing model performance changes to distribution shifts. arXiv preprint arXiv:2210. 10.709, 2022.
[38] Yiming Xu and Diego Klabjan. Concept drift and covariate shift detection ensemble with lagged labels. In 2021 IEEE International Conference on Big Data (Big Data), pages 1504-1513. IEEE, 2021.
[39] Heng Wang and Zubin Abraham. Concept drift detection for streaming data. In 2015 interna- jionajjoini conference on neural networks (IJCNN). pages 1-9. IEEE, 2015.
[40] Geoffrey I Webb, Roy Hyde, Hong Cao, Hai Long Nguyen, and Francois Petitjean. Characteriz- ing concept drift. Data Mining and Knowledge Discovery, 30(4))964 994,, 2016.
[41] Mauricio A Hernández and Salvatore J Stolfo. Real-world data is dirty: Data cleansing and the merge/purge problem. Data mining and knowledge discovery, 2:9-37, 1998.
[42] Tuomo W Pirinen, Jari Yli-Hietanen, P Pertila, and Ari Visa. Detection and compensation of sensor malfunction in time delay based direction of arrival estimation. In 2004 IEEE International Symposium Circuits and Systems (IEEE Cat. No. 04CH37512), volume 4, pages IV-872. IEEE, 2004.
[43] Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang. Yoon Kim, Yejin Choi, Nouha Dziri, et al. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.08559 2023.
[44] Karan Singhal, Shekoofeh Azizi. Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, pages 1-9, 2023.
[45] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprini arXiv:2204.02311 2022.
[46] Lei Wang, Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):1-26, 2024.
[47] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model 2tt.::330000866444 2023.
based agents: A survey. arXiv preprint
[48] Paulo M Gonçalves Jr, Silas GT de Carvalho Santos, Roberto SM Barros, and Davi CL Vieira. A comparative study on concept drift detectors. Expert Systems with Applications, 41(18):
81 844 1144-81555 8156, 2014.
[49] Jack W Smith, James E Everhart, WC Dickson, William C Knowler, and Robert Scott Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, page 261. American Medical Informatics Association, 1988.
[50] Aishwarya Mujumdar and Vb Vaidehi. Diabetes prediction using machine learning algorithms. Procedia Computer Science, 165:292-221 20999 2019.
[51] Muhammad Azeem Sarwar, Nasir Kamal, Wajeeha Hamid, and Munam Ali Shah. Prediction of diabetes using machine learning algorithms in healthcare. In 2018 24th international conference on automation and computing (ICAC), pages 1-6. IEEE, 2018.
13

---

[52] Albert Bifet, Geoff Holmes. Bernhard Pfahringer, Philipp Kranen, Hardy Kremer, Timm Jansen, and Thomas Seidl. Moa: Massive online analysis, framework for stream classification and clustering. In Proceedings of the first workshop on applications of pattern analysis, pages 44-50. PMLR, 2010.
[53] Robert Cattral and Franz Oppacher, Poker Hand. UCI Machine Learning Repository, 2007. DOI: https://doi.coo http:oogg////...... 22t1100.442///SKWWW
[54] Ryan Elwell and Robi Polikar. Incremental learning of concept drift in nonstationary environ- ments. IEEE transactions on neural networks, 22110):11771151, 2011.
[55] Indre Zliobaite. How good is the electricity benchmark for evaluating concept drift adaptation. arXiv preprint 1rX1v:1301.3524, 2013.
[56] Jock Blackard. Covertype. hmpp://ddi.org/11.2242. 1424442//CSKK.
UCI Machine Learning Repository, 1998.
DOI:
[57] Anjin Liu, Jie Lu, Yiliao Song, Junyu Xuan, and Guangquan Zhang. Concept drift detection delay index. IEEE Transactions Knowledge and Data Engineering, 3585):458544597, 2022.
[58) Fabian Hinder, Valerie Vaquet, Johannes Brinkrolf, and Barbara Hammer. On the hardness and necessity of supervised concept drift detection. In ICPRAM, pages 164-175, 2023.
[59] Vinicius MA Souza, Denis M dos Reis, Andre G Maletzke, and Gustavo EAPA Batista. Chal- lenges in benchmarking stream learning algorithms with real-world data. Data Mining and 2020.
Knowledge Discovery,
[60] Rafael MO Cruz, Robert Sabourin, and George DC Cavalcanti. Dynamic classifier selection: Recent advances and perspectives. Information Fusion, 41:195-216, 2018.
[61] Jinpeng Li, Hang Yu, Zhenyu Zhang, Xiangfeng Luo, and Shaorong Xie. Concept drift adaptation by exploiting drift type. ACM Transactions on Knowledge Discovery from Data, 18 (4):1-22, 2024.
[62] Guang-Bin Bin Huang, Qin- Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: theory and applications. Neurocomputing, 70(1-3)449-5501 2006.
[63] Shuliang Xu and Junhong Wang. Dynamic extreme learning machine for data stream classifica- tion. Neurocomputing, 238:433-444 44943344444444 2017.
[64] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Con- tinual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-10 1040, 2020.
[65] Sebastian Raschka. Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprini 1rXiv:1811.12800 2018
[66] Tania Cerquitelli, Stefano Proto, Francesco Ventura, Daniele Apiletti, and Elena Baralis. Au- tomating concept-drift detection by self-evaluating predictive model degradation. arXiv preprint arXiv:190..88/22 2019.
[67] Markus Hittmeir, Andreas Ekelhart, and Rudolf Mayer. On the utility of synthetic data: An empirical evaluation on machine learning tasks. In Proceedings of the 14th International Conference on Availability, Reliability and Security, pages 1-6, 2019.
[68] Angelos D Keromytis. Characterizing self-healing software systems. 2007.
[69] Debanjan Ghosh, Raj Sharman, H Raghav Rao, and Shambhu Upadhyaya. Self Self-healing systems gystems-survey and synthesis. Decision support systems, +2(4):2164-2188) 2007.
[70] Harald Psaier and Schahram Dustdar. A survey on self-healing systems: approaches and systems. Computing, 91:43-73, 2011.
[71] David Garlan and Bradley Schmerl. Model-based adaptation for self-healing systems. In Proceedings of the first workshop on Self-healing systems, pages 27-32, 2002.
14

---

[72] Omid Gheibi, Danny Weyns, and Federico Quin. Applying machine learning in self- self-adaptive systems: systematic literature review. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 15(3):1-37, 2021.
[73] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
[74] James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N Cohen, and Adrian Weller. Synthetic data-what, why and how? arXiv preprini arXiv:2205.032577 2022.
[75] Gianluca Detommaso, Martin Bertran, Riccardo Fogliato, and Aaron Roth. Multicalibration for confidence scoring in Ilms. arXiv preprint arXiv:2404.0m 04689, 2024.
15

---

Appendix: Self-Healing Machine Learning: A framework for autonomous adaptation in real-world environments
Table of Contents
Extended related work
17
A.1
Comparison other fields
17
A.2
Comparison component level
18
A.3
Unique properties of self-hcaling machine lcarning
19
H-LLM
20
Algorithm and details H-L.LM
20
B.2
Prompt templates used
................................
21
B.2.1
Prompts related diagnosis
21
B.2.2
Prompt templates related to adaptation
23
B
Example outputs of H-LLM
24
B.4
Evaluation strategies of self-hcaling algorithms
27
B.5
Computational notes
27
Case study design
28
Details the experimental setup
................................
28
Details viability studies
28
C.2.1
Viability Study
28
C,2,2
Viability Studies III VI
30
C.3
Other experimental details
32
Extended experiments
D.1
Monitoring
33
D.2
Diagnosis
................................
33
D.3
Adaptation experiment
34
D.4
Effects of Self Healing across corruption levels
34
D.5
Extended benchmarks)
34
D.6
Component wise Ablation Analysis
35
D.7
Model agnosticism
35
Optimal diagnosis
36
16

---

A Extended related work
In this section, we describe and contrast our work with other related areas.
A.1 Comparison to other fields
Concept drift adaptation. Concept drift adaptation algorithms, a key component of self- healing ML systems, primarily handle drifts by re- re-training models on new data [1 or older, pre-trained stored models [i, 6.8) These approaches can be implicit, like continuous retraining, or explicit, based on drift detection in data or model error [5] [3 27]. Drift detection methods compare distributions,
analyze data sequentially, or use statistical process control [59 For instance, the DDM algorithm
23 has in-control, warning, and out-of-control states.
Specialized drift handling. Techniques have been developed for various drift scenarios. For recurring drifts, methods store and reuse historical models 7 19. Streaming data is handled by blind approaches, like sliding windows [10] or adaptive decision trees [Л]. and informed approaches with explicit drift detection 241 Resampling can repair adaptation errors [27] while dynamic classifier selection finds the best model for each input [60]. Methods have been proposed for
robustness to noise [12], specific drift types [6]] and other issues [62 63 Recent work explores understanding distribution shifts through latent variable models [34] and other techniques [35]. Some
adaptive methods of re-training the model also include adding more hidden layer to learner upon
detection of a drift 662 63]. Another area of research closely linked within the field is dynamic
selection which attempts to find the most suitable classifier conditional on the covariates (6)
On "repairing concept drift". There have been other methods propose that implicitly try to adapt by detecting changes 277. However, these adaptations are still based on the observed empirical distributions as opposed to observing the reason for degradation. By periodically sampling the accuracy of inactive classifers, the authors identify cases where change was missed or misclassifed. However, this falls under the broader umbrella of trying out many pre-determined actions without directly reasoning about the reason for model degradation.
Continual learning. One might get the impression that self-healing machine learning might bear close resemblance to continual learning. Continual learning focuses on developing models that learn continuously from a stream of data, acquiring, retaining, and transferring knowledge across tasks over time [64. This contrasts strongly with self-healing machine learning. Below, we outline seven criteria by which self-healing machine learning and continual learning differ.
Differences between continual learning and self-healing machine learning.
1. Objective. The objectives of the two fields are different. Continual learning aims to learn sequentially from a stream of tasks while mitigating catastrophic forgetting. SHML focuses autonomously diagnosing and recovering from performance degrada- tion within single task due to distribution shifts.
2. Knowledge retention. A core goal of continual learning is to preserve previously acquired knowledge while learning new tasks. SHML does not explicitly aim to retain prior knowledge or acquire new knowledge, but rather to maintain stable performance the current task by adapting to the reason for degradation.
3. Stability-Plastiiee being plastic enough to learn new tasks and stable enough to remember old ones. In contrast, there is no such dilemma within SHML.
Dilemma. Continual learning grapples with the trade-off between
4. Task expansion. Continual learning seeks to expand the model's capabilities by in- creasing the number of tasks it can perform. In contrast, SHML operates a single well-defined task ensuring the optimal performance of model, typically by minimiz- ing empirical risk and does not aim to increase the number of tasks. Instead, the focus is on ensuring optimal performance under a single task.
5. Adaptation mechanism. The underlying logic mechanism of adaptation is different. Continual learning typically adapts by modifying model archilecture, updating param- elers via coustrained optimizations tion, using memory replay. In contrast, SHML explicitly adapts by diagnosing the root cause of performance drops and conditioning an adapta- tion action on the basis of that diagnosis. This explicit mechanism which is conditioned on is not a part of continual learning system.
17

---

6. Shift assumptions. Continual learning primarily handles shifts across distinct tasks, where the input or output distribution changes between tasks. In contrast, SHML considers shifts within the same task, where the joint distribution might change.
7. Theoretical formalism. Continual learning often formalized as sequence of con- strained optimization problems to mitigate interference between tasks. In contrast, SHML is formalized as finding optimal policy that can propose actions on the basis of diagnoses.
A.2 Comparison on a component level
Here, we focus on some related work within each sub-component. Table T provides key related work within each column. We do not focus separately on adaptation and testing because adaptation is covered above, whereas testing is simply stage which helps to evaluate proposed actions.
component
Delinition
Methodological contribution
Experimental contribution
Main practical implications
Related work
Monitoring
Eq
n/a
Sec.
63
More robust models againe fulse positvve dri't detection
Xi M
(Sea
45
Diagoosis
Eq
Det
Def
Prop
Sec.
64
Established franework reason about ioky madels
3 53
degrade
Adaptation
Fq
Asmp
Sec.
65
Torgeted adeptation by ideniifynng the mot canse (Soc.
HJ k:
Testing
6.6
Principled framework evaluste
6.
65 55
Self-bealing
New sell healing paradigm
addressing
limitation (See. 5.2 seli-healing system (See.
Table 6: Summary table of self self-healing ML. Use this guiding source navigate the paper. Related work defines the most similar available work within each component
Monitoring. Related work within monitoring largely relates to different statistical techniques for discovering the presence of shifts/drifts or model degradation. We see them integral part of SHML. However, they are also actively used by other adaptation methods to trigger adaptation systems.
Diagnosis. The diagnosis component is core component of SHML. Two primary works are closely related. The first work, "why did the distribution change?" [36]. attempts to factorize the change of the joint distribution into conditional distributions of each variable and attribute some changes to one of the marginals. This is achieved by modeling the change and relationship between variables as causal mechanism. The second work, "why did the model fail?" (37 attributes model performance degradation via causal mechanism. They assume that distribution shifts are induced due to an intervention in the causal mechanism which results in model performance changes, and uses Shapley values to attribute changes to specific distributions. These two methods are fundamentally different from SHML in multiple respects. First and most important, these works do not propose any actions on the basis of these failures or shifts. The primary goal of both works is to understand why distribution has changed or a model has failed, attributing it to a causal mechanism, instead of adapting the model to perform optimally. Second, the theoretical formalism introduced is substantially different and comes with different properties. Both works operate within the directed acyclic graph (DAG) framework, whereas operate under diagnosis component which is defined vector over space of possible reasons. Other key differences relate to the adaptation mechanism, shift assumptions, adaptation assumptions, level of granularity of the diagnosis, level of granularity of the adaptation, or testing.
Recent work has already started coming out understanding distribution shifts [35]. It is known that understanding why a distribution shift happens is important for mitigating that shift [35]. Some other people have looked at modeling shifts via latent variable models without relying access to labels at test time [34] However, as before, these methods do not share the objective of finding optimal actions for adaptation.
Self- helf-healing systems outside MI. Self-healing systems have been proposed outside of machine learning (68472). We view these inspirations for our work but consider them disparate and separate because none them touch upon the core problem of machine learning model degradation, and have not been applied in practice.
18

---

A.3 Unique properties of self-healing machine learning
The core of self-healing machine learning revolves around two primary components: the deployed ML model f and the healing system H. Here, we provide additional clarity on these components and their interactions:
Definition of the Deployed Model The model f represents the deployed machine learning model that we aim heal. is the function that makes predictions on input data and whose performance we're trying to maintain and improve. In our viability studies, we demonstrate this framework using logistic regression models as though the approach generalizes any predictive model.
Relationship Between f and π While is the model making predictions, is the adaptation policy function that determines what actions to take to modify based on the diagnosed reasons for its performance degradation. The healing system H. follows policy to output actions (such as a retrain model or az: remove corrupted features) which are then implemented onto f. Therefore, H. follows policy which helps to determine optimal actions that change/modulate the deployed ML model f.
Practical Implementation In our viability studies with H-LLM, the policy π is instantiated with an LLM (GPT-4) which uses the diagnosed reasons for model failures (also achieved with an LLM) to propose concrete actions. For instance, if f is a diabetes prediction model and π diagnoses that f's performance has degraded due concept drift, π might suggest an action to retrain with more recent data to adjust feature weights.
19

---

B H-LLM
This section provides more details on H-LLM.
B.1 Algorithm and details H-LLMM
The algorithm of H-LLM is presented in Algorithm 1.
Extended discussion.
Monitoring. We use statistical drift detec- tion algorithms to monitor model degradation from & previous time points 22 39 48).
Algorithm 1: H:LLL
Require: :f HD. H.A. HT, T, T,m,
Di- agnosis is triggered if a shift is detected. For our practical implementation, we use the Drift Detection Method, a popular method for binary drift detection classification.
t-1
t* < null while & S T do Hm HM((((;; U:) v.) )-) 8: > t and null then +
II. Diagnosis. Upon detection, H--LLM uses an extractor function & DD* DD transform the dataset information into an information vector This extractor function a mapping from the dataset to information about the dataset. It takes information before the shift happened and calcu- lates summary statistics, such as the mean, aver- age, standard deviation, percentiles, etc., within each column, as well as the performance of deployed model f under various data slices. For instance, this would also involve looping over all variables, binning them into 10 discrete values and calculating the average model performance across each bin. This is done to ensure that the information contained within the information vector are both summary statistics, i.e. how the data has changed, as well as specific performance metrics within data slices. The information is used to generate specific diagnoses as to what has happened. We observe, for instance, that summary statistics are extremely helpful if there are any larger deviations from average, the diagnosis module within H-LLMM picks up on these clues. This information is provided as textual information to the next step which is the diagnosis phase.
if null and t - t² Detection Window then
1' t
V E((x,)) CE,
for i = to k do
2, ~ I(+1v)
Č + [z: )
for j 1 to m do
0j ~ 1(-1)
â" - arg min Hr(s DDe
f + fā
t + null t f+
The information vector used as a textual representation within the next LLM call to generate concrete hypotheses/ / diagnoses about the reason for the f failure. This is where additional context could be added, if available, such as the presence any particular exogeneous events that could have affected model performance and could guide the diagnosis search. In the future, we envision that the module
additional context could be acquired by the system itself. This is used in a chain-of-thouge with self-reflection where & candidates for degradation are generated along with associated scores. We employ different "diagnosis" modules within H.LLMM. For instance, there a specific diagnosis module that only attempts find which covariates responsible for degradation. The system level instruction could be as follows: "Find covariates that are responsible for the model degrading". However, we also supplement this with more broader reasons for degradation, such as "Find and hypothesize reasons that could have resulted in model degradation, given the information provided". We provide three prompt templates used to hypothesize issues in Section B.2.I We sample such prompts m times using MC sampling. The chain-of-thouge and self-reflection is implemented by calling H-LLM multiple times re-consider the evidence and hypotheses. Table 2 illustrates diagnoses generated by H-LLM.
III. A Adaptation Conditioned on the empirical diagnosis distribution Č. H-LLM generates m candidate adaptation actions {a ~ I(-1) via CoT-based MC sampling. Specifically, we focus on three kinds of adaptation actions.
Generie adaptation actions
20

---

Adaptation actions by removing corrupted data
Adaptation actions by training multiple models for subsets of the data
This is reflected in three different prompt templates in Appendix B.2.2
Generic adaptation actions. The first attempt find generic adaptation actions that the diagnosis module suggests on the basis of the identified evidence. These are often quite generic, for instance, "add covariates that could control for the seasonality".. In many such cases, within the confines of our experiments, we do not have the ability to resolve the issues on the basis of the proposed solutions. Therefore, we add more directly actionable adaptation actions that are also attempted by H-LLM after the generic adaptation actions have been attempted.
Adaptation actions by removing corrupted data. Another concrete adaptation action is that instruct H-LLM to hypothesize specific data slices that might have been corrupted. This could be, for instance, biologically implausible values (negative insulin, age 200, implausible hbalc levels), mismatches (c.g. height, weight do not match BMI), sudden shifts the data (ages change from averages of 30 to 60), and other. The adaptation module then proposes which data slices to remove to achieve superior performance. These suggested data slices are then removed and re-trained in the next batch.
Adapaation actions by training multiple models. The final concrete adaptation action is to propose specific data slices where the model might have drifted within that slice. This is done because instead of global drifts, models sometimes drift locally and require complete re-training of the new dataset.
Example outputs of such strategies are presented in Appendix B.3 We note, however, that, in reality, there might be many possible adaptation actions, such as re-training the model on combinations of old and historical data, re-using old models, re-using parts of old models, creating custom ensembles, changing models altogether, changing hyperparameters or adding regularization terms, building different models for different samples based on their difficulty, switching between symbolic and predictive ML models in the face of high uncertainty, and many more. Our approach is to introduce only the primary few ways with the hope of extending this in the future.
As before, because the actions sampled from 1 textual representations, function to execute each on f.
an interpreter
IV. Testing. The sampled actions evaluated on empirical dataset (Def. 77. and the empirically optimal action a" arg minje[m] R(a) is implemented. Limited access to the shifted DGP compli- cates evaluating R(a). but can be approximated with empirical data Desst by using a backtesting window, continuously incoming data, or historical data. In all of our experiments, we use a backtest- ing window. However, other strategies could be attempted. The different strategies are explained in greater detail in Appendix B.4
Goal. This procedure aims approximate the optimal action (Def. 8 We remark that there might be better adaptation policies that could be suggested on the basis of evidence. Likewise, there might be better diagnosis modules available. We see H-LLM a first attempt integrate self-healing into ML.
B.2 Prompt templates used
The following are some of the primary prompt templates used within H-LLM.
B.2.1 Prompts related to diagnosis
Given the folloving information:
Data before the shift: (x.before describe )))
Data after the shift: {x_after describe
Context: {context)
Model performance across each covariate before the shift: eerariate_pefoommacce beforr)
- Model performance across each covariate after the shift: {
21

---

You know for fact that the model has degraded. Analyze the covariates and think why.
10
II
Review each existing covariate and provide a hypothesis on whether it might have changed and resulted in the model underperforming Provide evidence for each hypothesis and the strength of belief for each covariate.
12
13
Format your output as follows: Covariate: <covariate>; Hypothesis: ... Evidence: Strength of belief:
14
15
16
After reviewing all the covariates, assign confidence score for each covariate indicating your confidence level that the covariate has issues. Use the following confidence levels: extremely confident, confident, somewhat confident, unsure, completely unsure. Only use 'extremely confident' if you have overwhelming evidence for your decision. Prioritize making more confident beliefs. Avoid being uncertain. Use the available inputs well the data to make the best possible decision. Your goal to correct while reducing entropy the probabilities (be confidently correct).
17
Code Listing 1: Generie diagnosis prompt
Given the following information:
Data before the shift: (x.bofore.describe ())
Data after the shift: (x_after (escribe
Context: [context}
Model performance across each covariate before the shift: { wew.aatteeerrooomceeeeeee
Model performance across each covariate after the shift: { covariate_perfommmoo.de
You know for a fact that the model has degraded. Analyze the covariates and think why.
10
11
Then, hypothesize {n} possible covariates or combinations covariates that might have changed and resulted in the model underperforming. Each possibility should be mutually exclusive. For example, [X1] is one possibility, [X2] is another, and [X1, x2 a third.
12
Code Listing 2: Generic diagnosis prompt for searching combinations of covariates responsible for degradation
Given the folloving information:
Data before the shift: (x_before describe }
Data after the shift: {x_after describe
Context: {context)
covariates combinations of covariates that might have changed resulted model underperformance: Coovariate_uuesees
Initial hypotheses
Summarize the provided hypotheses and assign probabilities to each hypothesis such that the total probability sums 100%.
10
Your probabilities should be reflective of the evidence and data. Uniform probabilities (10% each) implies no knovledge. 100% probability covariate implies certain belief. Prioritize
22

---

making more confident beliefa. Avoid being uncertain. Use the available inputs well the data to make the best possible decision. Your goal is to be correct while reducing entropy of the probabilities (be confidently correct).
II
Format each hypothesis and its probability follows: Hypothesia: [<covariatel>, <covariate2>, ...;; Probability: probability>
14
Code Listing 3: Diagnosis probability prompt
B.2.2 Prompt templates related to adaptation
Suppose the following hypothesized issues in the dataset: { issues)
Data before the shift: (x_before describe Data after the shift: {x_after describe
Suggest {self. .n) possible reasons why the model might have failed the basis of the issues presented. These reasons should hypotheses that might have resulted in the degradation model if such hypotheses turn out to be true. These hypotheses also have to be likely on the basis of the issues provided. These hypotheses should be specific track down specific changes within the data that could have resulted in the model degradation.
the
the data itself. The goal is to
Format your output follows:
11
Hypothesis: <>; Evidence:
12
13
Code Listing 4: Generic adaptation prompt
f nun
Suppose the following issues in the dataset: {issues}
Data before the shift: (x_before (escribe Data after the shift: {x_after describee
Suggest fself.n) possible subgroups that if removed could result better performance for the model.
The subgroups can be single (e.g. X > x) but could also be multiple combinations (e.g. X and Y < y)
Code Listing 5: Subgroup adaptation prompt
Suggegttt solutions based on removing data *** task funn
Suppose the following issues in the dataset: (issues)
Data before the shift: (x_before describe Data after the shift: {x_after. ())
Suggest (self.n) possible subgroups that might need re- training. That is, fitting a separate model on these subgroups might result in superior performance.
The subgroups can single (e.g. X x) could also be multiple combinations (e.g. x and Y < y)
Code Listing 6: Subgroup retrain prompt
23

---

B.3 Example outputs of H-LLM
Covariate: HbA1c; Hypothesis: The distribution of HbÅic has shifted after the shift, which might have resulted in the model underperforming: Evidence: The mean of HbA1c has slightly increased from 6.999 to 5.730, and
standard deviation increased from 0.492 to 0.505. The model performance across different ranges of HbA1c has significantly dropped after the shift; Strength of belief: Extremely Confident
also
Covariate: FastingGlucose; Hypothesis: The distribution
FastingGlucose has shifted after the shift, which might have resulted in the model underperforming: Evidence: The mean of FastingGlucose has slightly increased from 99.639 to 100.519, and the standard deviation 180 The model performance across different ranges of FastingGlucose has significantly dropped after the shift; Strength of belief: Extremely Confident
slightly decreased from 15.341
Covariate: Age; Hypothesis: The distribution
Age has shifted after the shift, which might have resulted in the model underperforming: Evidence: The mean of Age has slightly increased from 49.721 to 49. 49.900, and the standard deviation has slightly increased from 12.066 12.310. The model performance across different ranges Age has significantly dropped after the shift; Strength belief Extremely Confident
Covariate: BMI; Hypothesis: The distribution of BXI has shifted after the shift, which might have resulted in the model underperforming: Evidence: The mean of BMI has slightly decreased from 25.075 to 24.985, and the standard deviation has slightly decreased from 4.015 to 3.903. The model performance across different ranges of BMI has significantly dropped after the shift; Strength of belief: Extremely Confident
10 Covariate: BloodPressure: Hypothesis: The distribution of BloodPressure has shifted after the shift, which might have resulted in the model underperforming; Evidence: The mean BloodPressure has slightly increased from 119.417 to 120.106, and the standard deviation has increased from 14.177 to 15.205. The model performance across different ranges of BloodPressure has significantly dropped after the shift; Strength of belief: Extremely Confident
11
12 Covariate: Cholesterol; Hypothesis: The distribution of Cholesterol has shifted after the shift, which might have resulted in the model underperforming; Evidence: The mean Cholesterol slightly decreased from 201.108 to 198.927, and the standard deviation has slightly increased from 39.557 to 209 The model performance across different ranges of Cholesterol has significantly dropped after the shift; Strength of belief: Extremely Confident
13
Covaris
14 Covariate: Insulin; Hypothesis: The distribution of Insulin has shifted after the shift, which might have resulted in the model Insulin has slightly
underperforming: Evidence: The mean increased from 86. to 84.925, and the standard deviation has alightly decreased from 45. 090 to 45. The model performance across different ranges of Insulin has significantly dropped after the shift; Strength of belief: Extremely Confident
15
16 Covariate: PhysicalActivity. Hypothesis: The distribution of PhysicalActivity has shifted after the shift, which might have resulted in the model underperforming: Evidence: The mean of
24

---

PhysicalActivity has slightly decreased from 3. 3.080 to 3. 020 and the standard deviation has slightly increased from 0.994 to 1.014. The model performance across different ranges PhysicalActivity has significantly dropped after the shift; Strength of belief: Extremely Confident.
19 #### Generated probability outputa based on the above
20
Prompt for getting these hypothesses:
23 и*.
24 Your goal to provide a summary of probabilities on likelihood of each of the covariates resulting in the model failing
27 Outputs:
28
29 Hypothesis: [HbAic]; Probability: 12.5%
Hypothesis:
[Fastingdlucose); Probability: 12.5%
31 Hypothesis:
[Age]; Probability: 12.5%
32 Hypothesis:
[BMI]; Probability: 12.5%
33 Hypothesis:
[BloodPreasure]; Probability: 12.5%
Hypothesis: [Cholesterol]; Probability: 12.5%
Hypothesis: [Insulin]; Probability: 12.5%
36 Hypothesis:
[PhysicalActivity)] Probability: 12
Code Listing 7: Output for guesses which covariates have shifted. This example showcases that when there is little evidence that any specific covariate has shifted more than the others
1. Issue: Increase in standard deviation; Evidence: The standard deviation for most the variables has increased in the new dataset, indicating increased variability in the data; Confidence:
8
2. Issue:
values for most of the variables have changed, which could indicate a shift the population being studied; Confidence: 7
Change in
values; Evidence:
The
3. Issue: Change in minimum and maximun values; Evidence: The minimum and maximum values for most of the variables have changed, which could indicate outliers Confidence:
a change the range of data;
4. Issue: Change in quartile values; Evidence: The 25%, 50%, and 75% quartile values for most of the variables have changed, indicating change in the distribution of the data; Confidence: 5. Issue: Negative values
PhysicalActivity; Evidence: The minimum values for Insulin and Physicalactivity are negative, which is not possible in real-vorld acenario and indicates data errors; Confidence: 10
Insulin
6. Issue: Change in distribution
data; Evidence: The changes in mean, standard deviation, and quartile values suggest change the distribution of the data, which could affect the model's
performance; Confidence:
7. Issue:
Increase in age range; Evidence: The minimum and maximum age has increased, indicating a broader age range the new dataset; Confidence:
8. Issue:
Decrease in BMI; Evidence: The mean BMI has decreased in the
new dataset, which could indicate change in the health status of the population being studied; Confidence: 6
10 9. Issue: Increase in Blood Pressure; Evidence: The mean Blood Pressure has increased in the new dataset, which could indicate change in the health status of the population being studied; Confidence:
25

---

10. Issue: Decrease in Cholesterol; Evidence: The mean Cholesterol has decreased in the dataset, which could indicate change in the health status the population being studied; Confidence:
12
Code Listing 8: Generic issue response which identifies overall issues within the dataset.
u.n
1. Subgroup: Individuals with age 85; Reason: The maximum age has increased in the new dataset, which could be due to outliers or the data.
2. Subgroup: Individuals with age 12; Reason: The minimum age has decreased in the new dataset, which could be due to outliers or errors in the data.
3. Subgroup: Individuals with Insulin < 0; Reason: Negative values for Insulin are not possible in world scenario and indicate data errors.
4. Subgroup: Individuals with PhysicalActivity < 0; Reason: Negative values for PhysicalActivity are not possible in real-world scenario and indicate data subgroup: Individuals with BMI 12 Reason: The minimum decreased in the new dataset, which could be due to outliers or errors in the data.
6. Subgroup: Individuals with BloodPressure 70.5; Reason: The minimum Blood Pressure has decreased in the new dataset, which could due to outliers or errors in the data.
7. Subgroup: Individuals with Cholesterol < 66.3; Reason: The minimun Cholesterol has decreased in the new dataset, which could be due to outliers errors in the data.
8. Subgroup: Individuals with FastingGlucose maximum FastingGlucose has increased in the new dataset, which could be due to outliers or errors in the data.
154; Reason: The
10 9. Subgroup: Individuals with HbAic < 4; Reason: The minimum HbAic has decreased in the dataset, which could be due to outliers errors the data.
11 10. Subgroup: Individuals with BMI > 39.6; Reason: The maximum BMI has increased in the new dataset, which could be due to outliers or errors in the data.
Code Listing 9: Example response about which subgroups to remove
1. Subgroup: Individuals with age 85; Reason: The maximum age has
increased in the new dataset, indicating broader age range.
2. Subgroup: Individuals with age 12; Reason: The minimum age has decreased in the
dataset, indicating broader age range. 3. Subgroup: Individuals with BMI 12.83; Reason: The minimum BMI has decreased in the new dataset, indicating change in the health status of the population.
4. Subgroup: Individuals with BMI 37.07; Reason: The maximum BMI has increased in the dataset, indicating change in the health status the population.
5. Subgroup: Individuals with Blood Pressure 16; Reason: The maximum Blood Pressure has increased in the new dataset, indicating change in the health status of the population.
6. Subgroup: Individuals with Blood Pressure 70.49; Reason: The minimum Blood Pressure has decreased in the new dataset, indicating a change in the health status of the population.
7. Subgroup: Individuals with Cholesterol < 44.64; Reason: The minimum Cholesterol has decreased in the dataset, indicating change the health status of the population.
8. Subgroup: Individuals with Cholesterol > 347.08; Reason: The maximum Cholesterol has increased in the new dataset, indicating change in the health status of the population.
26

---

10 9. Subgroup: Individuals with Insulin < 87; Reason: The minimum Insulin has decreased in the dataset, indicating data I1 10. Subgroup: Individuals with PhysicalActiváty < -0.30; Reason: The minimum PhysicalActivity has decreased in the new dataset, indicating a data error.
Code Listing 10: Example response about which subgroups to retrain the model on represents the new distribution, one might ask: how can we test models on this new distribution? The primary alternative used in our experiments a backtesting window which we define formally below.
B.4 Evaluation strategies of self- self-healing algorithms
Self-healing relies on a testing phase, i.e. the ability to test whether the proposed actions perform well on test dataset. However, given that the distribution has shifted and the historical data longer
Definition 3 (Backtesting Window). Let (Ptte be a sequence of probability measures X' x V, and suppose a distributional shift occurs at time t" E TT. i.e., Pt. + Pr-11 Let > t be the time at which the self-healing system detects the shift. The backtesting window is the time interval [t' ,t] satisfying the following properties:
vt [t' .t] (xr, Per. vt [t' [T] (Xr, ye) 4 P
We notice that the backtesting window is unique property that arises uppon sudden shifts in the data generating process. Specifically, because we assume only two data generating processes and transition between them at time point i, then all points l: where k > L will be from the new DGP and all points à will be from the old DGP. Since drift detection algorithm requries some time to detect the drift, by the time drift has been detected, we have some collected data from the new distribution which we call the backtesting window. We can therefore optimize our actions on this specific window of the dataset.
Clearly, this does not hold when the assumptions about the nature of the shift change. such case, we could always use continuously incoming streaming data. Upon the arrival of each new batch, we can test each proposed action and validate consistently upgrading and using the actions that perform well on the most recent batch of data. This strategy assumes that the labels are almost immediately available prediction time. If not, another strategy employed could be to test such actions on the mot recent available data with labels.
Other approaches could include generating synthetic data to imitate the new shift with labels using historical data by de-biasing it. However, these are experimental approaches which need further validation.
B.5 Computational notes
Computational overhead. SHML methods have larger overhead than reason-agnostic approaches due to the self-healing system (LLLM pipeline) identifying model failure reasons. Practically, it takes 20-40 seconds to implement full pipeline and correct a model upon drift detection. This overhead is negligible for real-world systems given the benefits. Overhead may vary across systems.
Sample efficiency. No differences exist as failure detection doesn't depend on sample size, but on self-healing pipeline complexity.
27

---

C
Case study design
Code can found at: http://ww..a.ooaaaaaaaaaaaaaaeea Hee. or https:// github. http:////wwadeecaaaaaaaaaa/a///a SeaSSSeffHHaaliae ine.__ng____...
C.1 Details the experimental setup
Experimental setup. To evaluate the performance of self-healing systems, we require to manipulate the data generating process (DGP) and ask "what-if" questions. Real-world datasets, while valuable, do not offer control over the DGP and come with pre-embedded biases that can implicitly affect detection systems 7 In contrast, by using synthetic data to control the DGP, we can run controlled in silico experiments and perform viability studies [74]. Furthermore, the overwhelming majority of
model adaptation methods are designed for tabular data (refer to Sec. and Sec. A which includes
our benchmarks (see Sec. [..I) Therefore, we simulate a diabetes prediction task [19 We perfectly mimic the introduced setup Sec. [3.]] Our goal is to predict the presence of diabetes Yi } } at each time point & for a set of n. observations, generated according to (changing) pre-specified DGP log P(Y.-0)
P()) = a + EkeK Xt, + Es, where includes relevant covariates
such Age or BMI, Bt, are time-varying covariates and E oO, is noise component.
We generated synthetic data for the diabetes prediction task. Each feature sampled from a normal distribution with specified parameters:
Hemoglobin Ale (HbAlc) levels are sampled from normal distribution: HbAlc N 055.7.
Fasting Glucose levels are sampled from a normal distribution: Fasting Glucose (100,, 15?).
Age is sampled from a normal distribution: Age (50, 12
Body Mass Index (BMI) sampled from a normal distribution: BMI ~ (225
Blood Pressure sampled from a normal distribution: Blood Pressure ~ N(120, 15²).
Cholesterol levels are sampled from a normal distribution: Cholesterol (20000 403).
Insulin levels are sampled from a normal distribution: Insulin ~ (8855,555)
Physical Activity is sampled from a normal distribution: Physical Activity ~ / N3,, 12).
The observations X are constructed as a matrix where each row is an instance of the generated features. The outcomes then determined by running the model through a logistic regression and obtaining a binary outcome value.
C.2 Details viability studies
C.2.1 Viability Study I
Viability Study I. To simulate covariate shift and introduce data corruption, we follow these steps:
1. Generate two datasets with different coefficients and noise parameters:
The first dataset with ni [0.3, 0.0075.0 -0.01, 0.05, 0.04, -0.03, -0.02, -0. and noise 6 ~N 0.22).
100, 000 samples, coefficients 31
The second dataset with n2
100, 000 samples, coefficients 82
0.000,,002, 0.00,,,00.00000000..00000 -2), and noise e2 (O 0.22). 2. Split the first dataset into training and testing sets, using 70/30 split.
[-0.3,-0.007, -0.0, -0.015,
3. Combine the testing set of the first dataset with the entire second dataset form the complete testing set. The second testing set therefore contains shift where the transitions between the DGPs happen.
4. In addition the shift the DGP, we introduce outliers in the second dataset by multiplying selected features by outlier factor that control. By default, it is set to
28

---

5. This outlier factor corrupts the number of columns corrupted by k, and corrupts percentage of values within the column, denoted as 7.
6. The shift index is determined as the starting point of the second dataset in the combined testing set.
7. We measure and report the performance of the model during the second data generating process.
Summary of parameters:
71 = 100, 000: Number samples in the first dataset.
n2 100, 000: Number samples in the second dataset.
31 [0.3, 0.0005,00005.. -0.01, 0.05, 0.04, 00.03, 0.02,-0.111 Coefficients for the first dataset.
B2 1-003,-0.0075, 0.2, -0.05, 00.015, 0.00,,,0002,-10000 Coefficients for the second
dataset.
El ,O, 0.2 Noise for the first dataset.
E2 0.23): Noise for the second dataset.
Seed for reproducibility: 42.
Proportion of outliers introduced: 20%.
Features corrupted varies.
Viability Study I. Summary of the benchmarks. Below describe the key benchmarks.
Benchmark 1. New model retraining. We use the Drift Detection Method (DDM) to monitor changes in the data distribution and retrain the model when a drift is detected. The procedure includes:
1. Split the test data into multiple batches.
2. Train the model on the initial training dataset.
3. For each batch in the test set:
Predict the outcomes and calculate the accuracy.
Update the drift detector with the prediction error (1 accuracy).
If drift is detected:
Retrain the model on the most recent batch.
Benchmark 2. Ensemble method. This algorithm uses an ensemble of models to improve robustness against data shifts. It combines the predictions of multiple models, each trained on different segments of the data. The procedure involves:
1. Initialize ensemble with single model trained the initial training dataset.
2. Split the test data into multiple batches.
3. For each batch in the test set:
Aggregate predictions from all models the ensemble, weighted by their current
accuracies.
Make final predictions based on the weighted aggregation.
Calculate the accuracy and update the drift detector with the prediction error. If drift is detected:
Train new model on the current batch and add it to the ensemble.
Update the weights of all models based on their accuracies.
This method maintains diverse set of models that adapt to different aspects of the data distribu- tion, enhancing overall performance and stability.
Benchmark 3. Partial updating. The model is retrained using sliding window of the most recent data batches. This allows continuous adaptation to recent changes in the data distribution. The steps are:
29

---

1. Split the test data into multiple batches.
2. Train the model on the initial training dataset.
3. Maintain a buffer store the most recent batches.
4. For each batch in the test set:
Predict the outcomes and calculate the accuracy.
Update the buffer with the current batch.
If the buffer exceeds predefined size (window size), remove the oldest batch.
Retrain the model using the data in the buffer.
Our method. H-LLM. In this example, we use H-LLM to identify corrupted columns and values and identify whether they need removal. The overall setup is as follows:
1. Split the test data into multiple batches.
2. Train the model on the initial training dataset.
3. Maintain buffers to store the most recent batches and backtesting window.
4. For each batch in the test set:
Predict the outcomes and calculate the accuracy.
Update the buffers with the current batch.
Update the drift detector with the prediction error.
If drift is detected:
Use the self-healing mechanism to inspect the most recent and previous batches.
Propose multiple adaptation strategies
Select the best adaptation strategy on backtesting window.
Retrain the model on the inspected and backtesting data recover from the detected
drift.
In all cases, the optimal strategy was removing a corrupted batch of data, where the amount of corrupted values or their extent varied.
Comments on the experimental setup of viability study The goal of this setup to showease that blindly retraining the model or using pre-determined actions is not necessarily optimal. In this case, the strategy required is to understand that the model requires full re-training and some values have been corrupted which require careful dealing, such adjustments removal.
C.2.2 Viability Studies III VI
Viability Study III. We employ the Drift Detection Method (DDM) and vary the sensitivity parameter indicated on the x-axis. We then calculate the recovery time how much time it takes detect the shift-, as well as the iost-intervention accuracy. As discussed in the main paper, this is purely determined by the DDM. For each detected drift, we fully run H-LLM detect issues and propose adaptation strategies that are tested on a backtesting window. If none of them beat the performance of the current model, the existing model f deployed.
Viability Study IV. We evaluate how well self- self-healing systems identify the root causes of problems.
We corrupt proportion of observations (corruption coefficient) by multiplying their values by factor (outlier factor) and see if the H-LLM detects issues related to these factors. We output probability distribution over diagnoses of which variable is corrupted. Knowing the corrupted variable, we measure the difference between the distributions using KL.-Divergence, with lower values indicating better matches between true and estimated corruption. uniform diagnosis baseline represents random guessing. Here is an example of what it means for the "true probabilities" to be corrupted when the corrupted column is "Age".
true_ cite-_pobbbbittie. "HbAic's ; 0, "FastingGlucose" 0,
A'Age 1,
BBHI :
30

---

BloodPressure ,,
Cholesterol 0,
IIsulin'' 0,
"PhysicalActivity" 0}
Code Listing 1 11: An example of true corrupted probabilities
Recall that H-LLMM produces normalized probability guesses, shown in Sec. B.3 Therefore, the obtained predicted guesses of which variable is corrupted in this setup looks as follows:
probbbiiteebb
(' Age : 0. 0212
"HbAlc's 0.125,
"FastingGlucose" : 0. 125, 'BMI 0.125,
"BloodPressure 0. 125,
"Cholesterol"; 0. 125,
'Insulin'; 0.5
PhyssccllAttvvity : 0.125
Code Listing 12: An example of predicted corrupted probabilities
When the corruption coefficient is higher, the output looks as follows:
Seclliteedprobabilit
A'Age' ": 0 0.4,
"HbAlc'; 0.2,
"Fastingdlucose 15,
'BXI 0.05,
"BloodPressure' 0.05,
"Cholesterol 0.05,
IInsulin's 0.05,
Phyysicalcttivity : 0.05}
Code Listing 13: An example of true corrupted probabilities
Therefore, the KL divergence computed between these two probability distributions. The KL the highest when the outputted probability distribution is uniform (first example) and the lowest when it perfectly matches the reference/true probability distribution. It has been shown that with certain techniques, LLMs can generally output calibrated confidence scores probabilities 775
The reason why the KL-divergence decreases because the predicted probabilties put greater relative value on the true corrupted value (i.e. the "Age" column in this example) as (i) the outlier factor increases and as (ii) the percent of values corrupted increase.
Viability Study V. We study the sensitivity of SHML adaptation policies by examining how well actions perform based (i) the number of corrupted values and (ii) the size of the backtesting dataset. Fig. G shows this relationship. The corruption coefficient is described in the overall experimental setup. The size of the backtesting window is the size of the dataset used to evaluat the proposed actions. Recall that H-LLM has three adaptation actions in place: (i) generic; (ii) filtering corrupted data slices; and (iii) training slice-specife models (Appendix [B. For this experiment, we focus on actions proposed by the second adaptation strategy: filtering corrupted data slices. Each adaptation action an identified data slice by H-LLM that might be corrupted, the removal which might improve performance. The following is example of proposed adaptation actions by the removal of the following queries (each query a separate candidate adaptation action):
[ (FFastingluuos
376. 145108',
IInslln > 320. 642677'
.HbAic > 21. 55395594
'Age 187 807319'''
'BMI > 93. 93.998780*
'BloodPressure 452. $52.899287',
'Cholesterol > 667.675355.
PPysicallctiviit > 11.314583',
"HbAic 25.544)
(FastingGlucose 376. 145108))
10
"(Age > 187.805319) & (BMI 93.998780)',
(BloodPrressure > 452. 8992877 & (Cholesterol > 757. (67.5))
(Insulin 320. 642677) & (PhysicalActivity 11.314583)
31

---

(HbA1c > 21. 5539666
(Fastingdlucose 376. 145108) & (Age
187 .0.3319)'
14
(BMI > 93. 93.998780)
(BloodPressure 452. 899287) (Cholesterol
757 757.675365)',
15
"IInullin 320 920.642677)
& (PhysicalActivity 11. 314583) & (HbA1c
21. 21.553946)
(FastingGlucose
376. 145108)
(Age 187 887.81) & (BMI
93 998880'
17
(BloodPressure 452.899287) & (Cholesterol 677.555355) & (Inaulin
> 320 320.674277)',
(PhysicalActivity > 11.314583)
(HbA1c > 21. 553946)
FastingGlucose 376. 145108)
19
(Age >
(BMI 93 998780) (BloodPressure
>
452. 899287) *]
Code Listing 14: Proposed adaptation actions by removing candidate corrupted slices
Such actions are proposed for each range values corrupted and evaluated accordingly.
Viability study VI. We study the importance the testing component (Eq. 7 by evaluating H-LLM suggested actions with and without the testing phase (backtesting window) and comparing their accuracies. Fig. 7 shows this relationship. The action with the backtesting window is the action which has received the highest empirical performance
the backtesting window. In contrast, the action proposed by "no backtesting window" is the action that is selected as the most likely one by H-LLM without any empirical validation. "Most likely" implies that after few iteration loops, this was the action that was listed as the first action to perform. This showcases the usefulness of having way to filter out actions with some specific actions. We mimic the setup from study IV where each action is specific subgroup to filter out to achieve better performance due to the corrupted nature of the data.
C.3 Other experimental details
We note that all experiments were performed using two compute resources: a server with NVIDIA RTX A4000 GPU and 18-Core Intel Core 19-109,, well an Apple MI Pro 32GB RAM. We exemplify H-LLM with GPT-4 via an API.
32

---

D Extended experiments
This section provides few additional experiments or more detail regarding the experiments presented in the main paper.
D.1 Monitoring
Average Recovery
Average Recovery Time
Setup. We vary the warm-star criterion within drift detection methods to evaluate the recovery time and post-intervention accuracy of HM..MM. The warm start parameter is the minimum number of samples required to conclude that a drift has been detected and trigger re- re-training or self- healing.
x 10
100
200
Warm-Start
Discussion. Fig. showcases the relationship between the warm- sarm-start
Rost-intervention Accura
parameter and the average recovery tiem and cost-intervention accuracy. You see the massive increase in average covery time that jumps when the warm-start is set relatively high threshold. This results from a drift detection algoritihm detecting a false positive drift just before
Post-incerveniinn
Accuracy
0.8
0.7
0.6
0.5
the actual drift. However, given the wwarm-star parameter, there significant delay in re-triggering the self-healing system. This suggests self-healing systems benefit from lower warm-start parameters in case the drift detection algorithms are sensitive to false positives. This corresponds with relative drop in the pnstiintervention accuracy because of the longer time it took to trigger self-healing.
200
Warm-Stare
Figure 8: Adaptation strate- gies of different methods in response three shifts.
Takeaway. Self- Self-healing systems benefit from lower warm-start parameters in case drift detection systems are sensitive to false positive trifts.-interventoon accuracy with smaller thresholds.
D.2 Diagnosis
Setup. In this experiment, instead of corrupting a single variable which responsible for model degradation, corrupt n variables to evaluate how
Quality of Diagnosis based on corrupted columns
TL-Divergence
well H-LLM can diagnose multiple corrupted values at once. With each
corrupted columns, the true corrupted probability changes. For instance, there are four columns and there is single corrupted column, the true cor-
ruption vector is O, OJ. If there are four corrupted columns, then it is [0.25, 0.25, 0.25, 0.25]. We use these probabilities and compare them to the corruption probabilities outputted by H-LLM. This is shown in Fig. 9
0
8 # columns that are corrupted
2 3 4
Figure The qualtiy diagnosis based on n columns. Lower is better
Discussion. This showcases that the more columns corrupted, the better the predictive diagnosis becomes. For instance, once all columns are corrupted, H-LLM outputs a uniform diagnosis because has no information given the evidence observed. This exactly corresponds to the true corruption probability, outputting KL of O. We notice that the KL generally decreases with the number of corrupted columns for this reason.
Takeaway. Greater uncertainty results more uniform diagnosis. However, less uncertainty can make it difficult to directly pinpoint the exact cause, causing more uncertainty.
33

---

D.3 Adaptation experiment
This section expands on the adaptation experiments by providing more variables and values by corruption coefficient and the number of columns corrupted.
Table 7: Accuracy based on the number of corrupted columns, where 5% of values given selected
column are corrupted on a shifted dataset with number of corrupted values. Higher is better. (with corruption coefficient of 0.05)
8
No retraiingg
a.43 0.02
0.04
0.48 0.02
0.64 0.02
0.45 a 0.02
0.45 0.02
0.45 4 0.02
0145 a 0.02
Partially Uptazing
a. 0.00
0.71 0.02
0.70 0:02
0.69
0.68 0.02
067:20002
0.65
0.54 0.06
New maining
0.71
Ensemble Method
0.71 4.00
0.00
0.00
0.00
0.08 0.00
0.0720002
0.60
0.50 0.02
H--LMM
0.35 0.01
0.3 =000
0.90
0.07
0.54 1:4=0000 0.02
0.79
0.77 0.02
0.68 0.02
Table 8: Accuracy based on the number of percent of corrupted value within given column (with three corrupted columns with three corrupted columns)
0.01
0,02
0.05
0.1
0.2
0.3
0.5
No retraining
0.43 2 0.00
0.02
0.48 0.02
0.45 a 0.02
0.46
0.48 0.02
0.49 1 0.03
Putially Uplating
(.74 = 0.03
0.72
0.70
=:00=00:2 0:02
0.66 + 0.02
0.62 4 000
0.57 0:02
0.52 + 0.03
New model
0.77
0.04
0:02
0
0.61
0.03
RELLM
0:00
0.0
0.00
0.82 a 0.00
0.70
0.7
0.52 0.00
D.4 Effects of Self-Healing across corruption levels
We systematically analyze how self-healing effectiveness varies with corruption levels across five datasets (Airlines, Poker, Weather, Electricity. and Forest Type). For each dataset, we vary both the corruption value t and the number of corrupted columns k, measuring accuracy with and without the self-healing mechanism. Figure[[O]soor severity. Specifically, as either 7 or l increases, the gap between baseline and self-healed performance widens. This pattern holds consistently across all datasets, though with varying magnitudes. These results demonstrate that self-healing becomes more crucial as data degradation becomes more severe, providing safety mechanism for maintaining model performance under challenging conditions.
that self-healing's 's impact grows with corruption
Airlines
Covtype
Elec
Accuracy C 0.7
Poker
Weather
Accuracy
0.7
Accuracy
0.7
Accuracy
0.7
Accuracy
0.7
0.6
0.6
0.6
0.6
0.6
0.5
0.5
0.5
0.5
0.5
2
6
2
6
# Compupted Cols
Compupted Cols
Corrupted Cols
# Commupted Cols
Corrupted Cols
Airlines
Covtype
Elec
Poker
Weather
Accuracy
0.7
Accuracy
0.7
Accuracy
00.
Accuracy
0.7
Accuracy
0.7
0.6
0.6
0.6
0.6
0.5
0.5
0.5
0.5
0.5
10a
10
10
10a
Corruption Value
Corruption on Value
Corruption Value
Corruption Value
Corruption Value
Self- Self-healing
No self-healing
Figure 10: Effects of self-healing for five datasets and the corruption value. Self-healing consistently identifies corrupted columns at test time. This typically becomes more important as the corruption level increases (either by corruption value or number of corrupted columns). Baseline is not implementing a self-healing mechanism upon drift detection.
vary the number of corrupted columns
D.5 Extended benchmarks
We extend our comparison on the diabetes prediction task to include additional adaptation methods and adaptive algorithms. Table results from ten different approaches, including standard adaptations (no retraining, partial updates, new model training, ensemble methods), streaming-specinne algorithms (ADWIN Bagging, Hoeffding Tree), and our SHML approach.
34

---

Method
Adaptations
Algoritiums
SHML Self-Healing MI.
No retraining
Partially updating
New model
Ensemblee method
Airstream
ADWIN
Hoefiding
Adaptive
Adaptive
training
Ragzing
Accurncy
0.16
0.13
0.12
0.12
0.13
0.30
0.30
0.62
0.11
0.76 0.88
Table 9: Accuracies of various adaptations on the original diabetes dataset setup in the paper.
The results show that while specialized streaming algorithms (e.g.. Hoeffding Tree at 0.70 accuracy) outperform basic adaptations, they still fall short of SHML's performance (0.76 accuracy).
D.6
Component-wise Ablation Analysis
To understand the importance of each SHML component, we conduct an ablation study by system- atically removing each component and observing the impact. Table I0 shows the results of this analysis.
Ablation
Accurucy
Takeaway
Baseline sell healing)
Accuracy without velf-healing
Full full sell-helling)
76
Self Sell-heeling improves accurscy over baseline
No monitoring
52
behuvior. Monitoring required to tigger the SHML sysem. H. LLM was triggered and actions were pruposed.
No diagnosis
52
Diagnosis required for proposing sensible actions. Defaults sensical actions. they Actions chesen but not tested against empiniall data, A subopcimal action was chosen.
actions
Actions
implemented
proposed.
No testing
62
Table 10: Ablation study results for H-LLM. We systematically remove one component of the system and inspect its outputs. The takeaway represents our qualitative evaluation.
The ablation reveals that each component is crucial for effective self-healing. Removing monitoring (52% accuracy) prevents the system from triggering adaptation. Without diagnosis, the system proposes non-sensical actions, leading to baseline performance. Removing action generation or testing similarly degrades performance bascline levels, though testing removal shows slightly better performance (62%) as some reasonable actions are still attempted, albeit without proper validation.
This analysis empirically validates framework's design, showing that effective self-healing requires all four components working in concert.
D.7 Model agnosticism
We evaluate SHML's effectiveness across ten different ML models to demonstrate its model-agnostic nature. Table [] shows results for models ranging from simple (c.g., Decision Trees) to complex XGBoost,, comparing various adaptation strategies. SHML consistently outperforms bascline approaches across all model types, with improvements ranging from 11 percentage points (Naive Bayes) to 31 percentage points (LDA). This consistent improvement demonstrates that SHML's benefits are not tied to any particular model architecture but rather stem from its ability to reason about and address degradation causes.
Menhod
DecisionIree
KNN
LDA
LoginicRegression
MLP
Naivellayes
Perceptron
RandomFirest
SGD
XGBooott
Raveline (No retmaningg
0.63 + 0.05
0.51 a 0.
0.47 0.08
0.49 a 0.02
0,3 a 0.04
0.5 à 0.03
0.49 a 0.01
0.6 a 0.05
0.47 a 0.03
0.67 a 0.05
Sliding Window
0.63 4 0.05
0.51 0.00
0.47
0.03
0.49 0.02
0.66 0.03
0.5 0.03
0:49
0.01
0.70 = 0.05
0.47 0.03
0.67 0.05
Drift
0004
0.5
Ensemble with DDM
0.63
0.00
0.47 0.03
0.99
0.65 0.05
0.5 0.03
0.09 0.01
0.65 = 0.07
0.47 0.03
0.67
R-LLM
0.70 0.64
0.73
0.65 0.77 0.04
0.76 0.84
0.78 = 0.15
6,22 a 0.02
0.68 = 0.09
0.72 0.04
0.75 a 0.04
0.71 6.4
Table II: Comparison of various methods across different ML models on the weather dataset (sctup
above), where features are corrupted at test time. Results show mean accuracy standard deviation.
35

---

E
Optimal diagnosis
Here, we prove that under the stated assumptions, the optimal diagnosis has zero entropy. the optimal diagnosis Ç has zero entropy, i.e. HI((*) 0.
Proposition 3. Under Assumption
/
Proof. By Definition
2
arg min (to)
(12)
(=A(Z)
As A is finite, we write the expected value as follows.
E (1>
Σ ((a)
(13)
anA
By Assumption | this can be rewritten as:
Σ R(a) Σ [<(a=))<(())
(14)
acA
We change the order of summation arrive at the following.
E{(=) E R(a)m(al=))
(15)
se2
aEA
The inner sum can now be rewritten as an expectation.
Σ
(()))))
(16)
zeZ
Thus we can rewrite the minimization problem as follows.
Ç* arg min Σ (())))
(17)
(<)(2) se2
Let 2" Z such that
[[]()) 2 arg min E (f= [[(a)]
(18)
20
Then
2 Σ ((z) (()))
362
222
(19)
2e2
Therefore
=²= (z*)
(20)
and by the definition of entropy and (=*)* get
H((")) = 0
(21)
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper presents a novel paradigm called self-healing machine learning.
This is established in Section and Sec. 4 where we develop the theoretical underpinnings of this field. We also make significant practical contributions by proposing the first-ever self-
We discuss the positive impacts of this technology in the introduction and the discussion section, where argue it can have transformative effects in a variety real-world situations. We clearly show the viability of self elf-healing
healing algorithm presented Sec. S
machine learning Sec. 6
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Self-healing machine learning inherently assumes that diagnoses can be performed well and that actions can be proposed on the basis of these diagnoses. This poses challenges we discuss in Sec. [5.1 We attempt overcome these limitations by using language models and incorporating their unique properties into H-LLM. We discuss further challenges of self self-healing systems in the discussion section (Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
37

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In order to establish the relationship between diagnosis and the actions, we
define the certainty diagnosis (Def. [ and the optimal diagnosis (Def. 2 Furthermore, we assume independent actions (A (Assumption []. Under this, we establish two propositions
about the entropy of an optimal diagnosis (Proposition) [ and its existence (Proposition 2
One of the proof' is in the main paper and the other one can be found in Appendix E
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the required information to reproduce our algorithm H-LLM (Appendix. B as well as the full detail on experiments, including dataset generation, param- eters, etc. (Appendix. C). We provide additional appendix section with supplementary experiments that can aid the reviewers where we discuss the setups within each experiment (Appendix. D) The code can be found at: https: com wwwpccww.oppcccccccc
iusiuuuureuabSSs. Healing ML or htttps://github htttcchu...om/aaaaaaeeeeeeeeaa haap://aaddrrrchaaaaaaaaaa htt.::/hhaaarraaaaaaaaaaaaaaaaaa _eaaling_MM
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
38

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide full experimental details, where all access to data and code is fully available. Code can be found at: httpp:://githbb http://hhw...cooppaaasssaaaaaa/a
Healing_Mi ML or
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/pullle wttpcccpppppccceeeeeeeeee...eeee httpe//ggg.eee//eebebbiisssssss /Coo///oooiisssseeeeeee ssiooiicccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All details are provided in Appendix. C The experimental setting is presented in the core the paper to a level of detail that necessary to appreciate the results and make sense of them. Full details provided in the appendix.
Guidelines:
39

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All appropriate experiments report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, all relevant information is provided.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the httpurippp.ccppupc///eeii
NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes.
40

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes. We discuss this in Sec. 7 To reiterate, we believe our work can have significant positive effects on multiple areas within AI and have substantial practical impli- cations, as discussed in the Discussion section. This includes having immediate practical benefits in industries where model degradation is common, such as medicine, finance, pre- dictive policing. IoT data streams, and more. We further hope our work spurs substantial developments self-healing theory. We also discuss that the unique improvements in systems that can employ self-healing could also be misused by agents for other purposes, such as using self-healing for surveillance systems or other ethically ambiguous technolo- gies. The broader impact also subsumes future work in this area. We hope that potential direction for future work is building theory around optimal diagnoses, optimal adaptation strategies, as well as scaling larger algorithms.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any data or models with high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
41

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
Justification:
We cite and refer all appropriate codebases that employed as benchmarks.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The core assets, including the framework and the algorithm, are clearly described.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
42

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
43


Reviews:
[{"review_id": "lBgl2N9pKj", "reviewer": "Reviewer_LnCp", "summary": "Model performance degradation on unseen data is a classic problem. Existing approaches solve the problem through a deterministic strategy: change model, retraining, etc. This paper proposes an adaptive way to decide the action after model degradation automatically and introduces a self-healing framework. The evaluation thoroughly analyzes the intuition of SHML and its limitations.", "questions": "Please refer to the weaknesses", "limitations": "Limitations are well-discussed in the main text.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Autonomous healing for model performance degradation is an important problem.\n2. The automatic adaptation idea is novel and interesting.\n3. The evaluation is strong and thorough, which covers the details to help readers further understand the scope and the logic of the proposed method.", "weaknesses": "1. The writing is hard to follow in section 3. What is the practical meaning of f? How is it related to the policy? \n2. The assumption of the requirement for optimal adaptation actions is strong. Is there any practical case to support this assumption?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LnCp,\n\nThank you for your thoughtful feedback on our work on self-healing machine learning. We appreciate your recognition of the importance and novelty of our approach. We'll address your concerns in two parts, corresponding to both weaknesses.\n\n---\n\n# (A) Clarity of Section 3\n\nThe goal of Section 3 is to describe how a deployed ML model ($f$), such as a logistic regression classifier, can be \"healed\" with a self-healing system ($\\mathcal{H}$). In this case, *healing* refers to restoring the performance of $f$ after it drops. \n\nA natural question is: how do we know what actions we should take to heal $f$ once it drops in performance? We say that $f$ should be healed by the healing-system $\\mathcal{H}$ which follows a policy $\\pi$. The policy $\\pi$ outputs actions $a$ (such as $a_1$: retrain a model or $a_2$: remove corrupted features) which are then implemented *onto* $f$. Therefore, $\\mathcal{H}$ follows a policy $\\pi$ which helps to determine optimal actions $a$ that change/modulate the deployed ML model $f$.\n\n\nWe will improve the paper's writing in the following ways:\n\na) We will add a clear definition of $f$ at the beginning:\n\n> $f$ represents the deployed machine learning model that we aim to heal. It is the function that makes the predictions on input data and whose performance we're trying to maintain and improve. \n\nIn our viability studies, a logistic regression model represents $f$.\n\nb) We'll clarify the relationship between $f$ and $\\pi$.\n\n> While $f$ is the model making predictions, $\\pi$ is the adaptation policy --- a function that determines what actions to take to modify $f$ based on the diagnosed reasons for its performance degradation.\n\nc) We will include an illustrative example.\n\n> For instance, if $f$ is a diabetes prediction model and $\\pi$ diagnoses that $f$'s performance has degraded due to concept drift, $\\pi$ might suggest an action to retrain f with more recent data or to adjust feature weights.\n\nd) We will link the theory of the policy $\\pi$ to the experiments.\n\n> In our viability studies with $\\mathcal{H}$-LLM, the policy $\\pi$ is instantiated with an LLM (GPT-4) which uses the diagnosed reasons for model failures (also achieved with an LLM) to propose concrete actions.\n\ne) We will explain how the policy impacts SHML:\n\n> Self-healing ML is formalized as \"an optimization problem over a space of adaptation actions.\" This means we aim to find the optimal actions to take each time the model $f$ degrades. These actions are chosen by the policy $\\pi$ of the self-healing system $\\mathcal{H}$ (Fig. 3). For instance, two different policies $\\pi_1$ and $\\pi_2$ might propose different actions to take in response to $f$'s performance drop. \n\n---\n\n# (B) Requirement for optimal adaptation actions\n\nYou're correct that the assumption of optimal adaptation actions would be very strong. Optimal adaptation actions would assume that we always pick the most *optimal* action in any situation. However, this is unrealistic --- we rarely *really* know what is the optimal action to take to improve a model's performance because there are many possible reasons which could have led to the decrease in performance. So, achieving model optimality is often impossible.\n\nHowever, we believe there may have been a misunderstanding. Our framework doesn't require optimal actions. Rather, we aim to make more informed decisions to try to *approximate* optimal actions (which are never known in practice). We illustrate this with the model degradation example in **Sec. 3.2**.\n\nTo improve the clarity, we will make the following changes:\n\na) We will add a paragraph (lines 154-156) explaining that SHML approximates an ideal rather than achieving or assuming perfect optimality.\n\n> In real-world ML problems, it is often impossible to determine the optimal action due to the complexity of the problem. SHML attempts to approximate an ideal optimal adaptation strategy instead of achieving perfect optimality. This is done by selecting actions to take by taking into account diagnosis information instead of relying on deterministic actions (such as model retraining).\n\nb) In Section 5.2, we'll add this practical example.\n\n> Consider a scenario where both data corruption and concept drift occur simultaneously. A traditional method might simply retrain the model on new data (potentially incorporating corrupted values). In contrast, SHML would diagnose both issues and suggest a two-step adaptation strategy: (a) clean the corrupted data and (b) retrain the model on the drift-adjusted datasets. In our experiments, this has improved the model by about 18% in model accuracy, despite not necessarily being the theoretically optimal action (Sec. 6.1).\n\nc) We'll expand the discussion section with:\n\n> While SHML attempts to find optimal adaptations, it does not theoretically guarantee that the adaptations are indeed optimal. While we see this as a substantial improvement over reason-agnostic methods, future research could explore how to obtain theoretical guarantees of optimality within self-healing machine learning. \n\n---\n# Thank you\n\nYou have helped us improve our paper. Given these changes, we hope you consider revising your assessment of the paper's impact and the corresponding evaluation for NeurIPS. ☺️", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qi14A6is03", "reviewer": "Reviewer_4TYN", "summary": "The paper presents a self-healing framework for machine learning models called Self-Healing Machine Learning (SHML). Unlike previous methods, SHML autonomously diagnoses the causes of model degradation and suggests corrective actions based on these diagnoses. The authors formalize SHML as an optimization problem, aiming to minimize expected risk by adapting to changes in the data-generating process (DGP).\nA theoretical framework for self-healing systems is introduced, exemplified by H-LLM, which leverages large language models for self-diagnosis and self-adaptation. Empirical analyses of SHML's components demonstrate its potential and effectiveness.\nThe paper underscores the importance of ensuring optimal performance in algorithms used in high-stakes applications. By enabling systems to autonomously adapt to new environments, SHML aims to advance self-healing systems, benefiting both the machine learning community and society. The theoretical framework lays the groundwork for developing optimal adaptation and diagnosis methods. The authors hope this work will stimulate further theoretical developments and encourage the adoption of self-healing systems in critical fields such as medicine and finance.", "questions": "1. Is the diagnostic step continuously operational during the entire period of model usage?\n2. How does the performance of the framework compare to benchmarks when applied to large-scale models?", "limitations": "Authors mention a limitation in section \"SHML’s success relies on accurate root cause identification and finding effective adaptation policies which could pose challenges in some complex, real-world settings\" (Sec. 5.1).\nThis could be further explored.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "From the perspective of originality, this paper lowers the barrier for others to implement adaptation actions in machine learning models. Its importance lies in addressing the growing challenge of maintaining machine learning models, especially given their increasing usage. A framework capable of diagnosing drift or degradation and autonomously solving these issues is crucial. This paper effectively addresses this need, outperforming existing approaches in both presentation and results.\nThe organization of the paper is good, with a well-structured presentation. The detailed inclusion of examples, definitions, assumptions, theoretical explanations, and results in the main text and appendix is thorough and effective (but the appendix is probably required to really understand)", "weaknesses": "The main weakness of this paper is its tendency to use overly long sentences and complex phrases, which can hinder readability and clarity. Another significant weakness lies in the viability section; many crucial details have been relegated to the appendix. This makes it difficult to fully understand and trust the framework based on the main text alone. Additionally, while the concept of \"self-healing machine learning\" is compelling, the section dedicated to it could be more concise and focused. Overall, improving the clarity of these sections would greatly enhance the paper's impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 4TYN,\n\nThank you for carefully reading our paper. We're glad that you think we address the problem of maintaining machine learning models autonomously and that this lowers the barriers for others. We respond to each point below.\n\n---\n# (A) Moving information from the Appendix to the main text\n\nIn the camera-ready version, we'll use the additional page to expand the viability section in the main text. We'll include more details on the prompts and input/output examples of $\\mathcal{H}$-LLM. We will also highlight which design choices are most important to consider such that adopters can more easily replicate and build their own self-healing systems.\n\n---\n# (B) Improving the clarity of Section 3 on self-healing ML\n\nWe agree with you that the main section could be more concise and focused. To improve clarity, we will make the following changes:\n\na) We'll start with a concise overview of SHML: \n\n> Self-healing machine learning is a framework for autonomously detecting, diagnosing, and correcting performance degradation in deployed ML models. It aims to maintain model performance in changing environments without constant human intervention.\n\nb) In the box \"Self-Healing Machine Learning in a nutshell.\", we will add a more intuitive explanation at the beginning. \n\n> SHML contains four components: (a) **monitoring**: continuous assessment of model performance; (b) **diagnosis**: identification of root causes for performance degradation; \\(c) **adaptation**: suggesting possible corrective actions to take in response to degradation, and (d) **testing**: empirically evaluating the effect of actions on the model's performance. After these steps, the best action is implemented on the ML model. This is illustrated in Fig. 2.\n\nc) We will contrast SHML with traditional approaches at the end of Sec. 3.3 (lines 152-154):\n\n> The primary insight of SHML is that the best action to take in response to model degradation depends on the reason for that degradation. This contrasts with standard approaches, which often assume the best approach is independent of the degradation reason. For example, a standard drift adaptation method might continuously retrain the model, which could be suboptimal if the new dataset is corrupt.\n\nWe will also make other smaller changes, such as add a clear definition of $f$ at the beginning, clarify the relationship between $f$ and the policy $\\pi$, and include an illustrative example.\n\n---\n# \\(C) Question: Is the diagnostic step continuously operational?\n\nThe diagnostic step is not continuously operational. It's triggered only when the monitoring component detects performance degradation, as shown in Figures 2 and 3. This design choice is due to the computational intensity of the diagnostic step. In our implementation of $\\mathcal{H}$-LLM, we perform multiple language model calls to hypothesize possible model failures, propose actions and implement them. In practice, we find this loop takes about 20 seconds to a minute to finish. \n\nContinuous diagnostics might be feasible in two scenarios:\n\na) Batch prediction settings: If your model makes predictions in large batches (e.g., every 15 minutes) rather than continuously (e.g., every second), a diagnosis step could potentially run for each batch, even without detected degradation.\n\nb) Future research could explore novel architectures where LLM-based monitoring systems run concurrently and continuously with monitoring. However, to the best of our knowledge, no such systems currently exist.\n\n**Actions taken**: We will clarify when the diagnosis is triggered in the main paper more clearly and expand on when it might be continuously operational.\n\n---\n# (D) Question: performance with large-scale models\n\nThe SHML framework is agnostic to the model $f$ (L139-141 in the main text). Therefore, self-healing ML shows superiority over benchmarks whenever understanding the reason for degradation is important, *regardless of the learner*. We can intuitively explain this with an example: suppose that a new batch of data is corrupt. Using a reason-agnostic approach, such as retraining your model on new data, is suboptimal regardless of the model used. This is because the optimal action requires to remove the corrupted data and then retrain the model on the de-corrupted data.\n\nThat said, **we've conducted an additional viability study** comparing SHML's performance against other adaptation methods **for 10 popular ML models** commonly used in practice. We include models used for large-scale industry applications, e.g. XGBoost or Random Forest. We simulate real-world degradation by corrupting data at test time and evaluating the performance of each adaptation approach. The results are included in **Table 4** in the response pdf. **Takeaway**: We show that $\\mathcal{H}$-LLM outperforms other adaptation methods. This showcases that self-healing ML can benefit any downstream ML model whenever understanding the reason for degradation is important (such as test time corruption).\n\n\n---\n# (E) Limitations\n\nIn the camera-ready version, we will expand our limitations.\n\n- We will move some challenges of building self-healing systems from **Section 5.1** (L206-216) to the discussion to highlight limitations.\n- We will expand on the challenges in diagnosis (L208-212) by giving an illustrative example.\n- We will expand on the challenges in adaptation (L213-216) by explaining when root cause identification might be easy/difficult.\n- We will outline research directions that can help overcome said challenges.\n\n---\n# Thank you\n\nYou have helped us improve our paper. Given these changes, we hope you consider revising your assessment of the paper's impact and the corresponding evaluation for NeurIPS. ☺️", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WUmUNyKiKZ", "reviewer": "Reviewer_vweT", "summary": "This paper proposes a new concept of self-healing machine learning, or SHML. The idea is based on understanding and addressing the reasons of performance drops in ML systems, thereby going beyond most common approaches that are labelled as reason-agnostic. The approach is based on a pipeline well illustrated in Fig 2 that includes four steps: monitoring, diagnosis, adaptation and testing. Formal definitions are provided. Experimental support is provided to demonstrate some aspects of the new idea and its potential advantages.", "questions": "1. Is it possible to define a medium to large set of conditions in known benchmarks that could be use to extensively test the SHML?\n2. One challenge in a system with multiple sequential steps is that the failure of one component affects all the following. E.g. failure in diagnosis will affect adaptation and testing.  How can we assess how well the system work when failure in one component compromises the correct functioning of others?", "limitations": "I find that the limitation part (before the conclusion and in 5.2)  is too short . Yes, I agree that the correct identification of the root causes of performance drop is challenging in real work scenarios, but the issue needs to be expanded and discussed more. For example, why is it difficult? Are some reasons that are more difficult than others to identify? Monitoring could also be challenging. Adaptation could be challenging. \n\nThe impact is sufficient covered.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I have identified the following strength: \n- The paper starts from an interesting assumption that the reasons for performance drop should be considered and investigated. \n- The paper is well organised and easy to read. \n- The method is described with precise formalism.", "weaknesses": "I have identified the following weaknesses: \n1. While the main assumption is move on from reason-agnostic methods to a system that attempts to understand the reasons, the paper does not offer a good classification of such reasons, mainly because it main a meta-level approach, in which reasons are to be learned. This could be less effective than pre-defining monitoring and diagnosis methods according to known causes.  \n2. As a follow-up from the previous point, the diagnostic part that generates hypotheses via a LLM may or may not be correct and the adaptation may or may not be effective. I find it hard to get a feeling of how well the system would work on a large set of benchmarks and problems from the evidence and experiments presented in the paper. \n3. Most of the interesting details of the paper are actually in the appendix.\n4. I disagree that this is the first non-reason-agnostic method, as the authors overview and list other approaches that do focus on the causes of performance drop. I agree that this may be the first approach to tackle the issue comprehensively. Nevertheless, the novelty, knowledge gap, and improvements of the approach with respect to existing approaches is not well summarised in the introduction/related work, which is rather short and it refers to the appendix for further detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear R-vweT,\n\nwe're glad you think our paper is important and easy to read. \n\n---\n# (A) Classification of reasons for performance drops\n\nYou're right that we don't provide an exhaustive classification of reasons for performance drops. We aim SHML to be a meta-level approach which flexibly adapts to various scenarios.\n\nA meta-level approach offers two main advantages: 1) It allows the system to adapt to unforeseen scenarios and new types of model degradation not captured by pre-defined lists. 2) It flexibly generates multiple hypotheses to explain the same model degradation, as different explanations may account for the same issue (illustrated in **Table 2** of the main paper).  Also note that SHML could incorporate pre-defined causes into the framework, as these represent deterministic if-else diagnosis functions and would result in a less complex policy function $\\pi$ than explored in the paper.\n\n**Actions taken**: Include the classification of performance drops in the discussion.\n\n---\n\n# (B) How well the system works on large set of benchmarks\n\nWe introduce self-healing ML as a framework for understanding model degradation. Our goal is to demonstrate its viability rather than provide extensive benchmark comparisons.\n\nThat said, to address your concern directly, we've conducted additional viability studies:\n\n- We compare $\\mathcal{H}$-LLM, with baseline approaches on **five real-world datasets** by introducing corruption at test time (**Table 1 in supplementary pdf**). **Takeaway**: $\\mathcal{H}$-LLM improves upon reason-agnostic baselines when the reason for performance drop can be diagnosed.\n- We show how the importance of self-healing ML varies with the problem difficulty (**Fig. 1** in the response pdf). **Takeaway**: The effects of self-healing are greater when the data corruption levels are higher.\n\n**Actions taken**: We will include a summary of this discussion in the main paper.\n\n---\n# \\(C) The effectiveness of each component\n\nWe emphasize that we see robust studies of each component as non-trivial research directions, as our paper primarily aims to introduce SHML. That said, we address your concern by performing a qualitative ablation study. **Setup**: We run $\\mathcal{H}$-LLM and iteratively remove one component from the system, inspecting its output. This is presented in **Table 3 in the response pdf**.\n\n**Takeaway**: We illustrate that *all components are required for the SHML system to work*.\n\n---\n# (D) Moving items from the appendix to the main text\n\nIn the camera-ready version, we'll use the additional page to expand the viability section in the main text. We'll include more details on the prompts and input/output examples of $\\mathcal{H}$-LLM.\n\n---\n# (E) Is self-healing the first reason-agnostic method?\n\nYou're correct that there have been previous papers attempting to identify reasons for model degradation, discussed in **Section 2** (L83 - 90 and L105-120). To clarify our contribution, we've created a comparison table highlighting the key differences between SHML and existing approaches:\n\n| Approach                                    | Diagnosis    | Adaptation Action |\n|-|-|-|\n| Concept drift adaptation [1,2]              | n/a        | Fixed             |\n| Specialized drift handling [3,4]            | n/a        | Fixed             |\n| Distribution change attribution [5]         | Fixed     | n/a             |\n| Model failure attribution [6]               | Fixed     | n/a             |\n| Dynamic classifier selection [7]            | n/a        | Fixed          |\n| SHML (Our approach)                         | Variable     | Variable          |\n\n[1] Gama et al. (2004), [2] Lu et al. (2018), [3] Goncalves et al. (2013), [4] Alippi et al. (2013), \n[5] Budhathoki et al. (2021), [6] Zhang et al. (2022), [7] Cruz et al. (2018)\n\n**Actions taken:** We will highlight that SHML is the first framework where the diagnosis and adaptation are not fixed in advance and where the diagnosis informs the adaptation. \n\n---\n# (F) Defining conditions in known benchmarks that could be used to extensively test SHML\n\nWe agree that having an extensive set of known conditions could be extremely useful for evaluating SHML systems that would likely gain a lot of traction in the ML community. We think there are multiple conditions that could be used to simulate real-world conditions in existing benchmarks (such as data corruption, systematically changing the DGP, introducing external shocks such as covid-19) which would require significant testing and validation. We see this as promising future work.\n\n---\n# (G) Failures in sequential systems and how that affects self-healing ML\n\nYou're right that failure in one component can affect the entire system's performance.\n\n**a) Component sensitivity analysis**. To provide initial insights, we refer to the qualitative ablation study presented earlier in our rebuttal. The study showcased that each component working is required for the system's overall performance. \n\n**b) Built-in safeguards**. SHML includes mechanisms to mitigate cascading failures with the testing component (step 4). Actions that do not improve performance over baseline are discarded. \n\nWe see designing fail-safe self-healing systems as a promising research direction.\n\n---\n# (H) Expanding limitations\n\nIn the camera-ready version, we'll expand the discussion of the limitation sections.\n\n1. We'll add a paragraph discussing specific challenges with root cause identification (expanding on and moving items from section 5.1 lines 208 - 2012)\n\n2. We'll add a paragraph discussing the challenges with choosing an appropriate action (expanding on and moving items from section 5.1 lines 213-216)\n\n3. We'll add a challenge explaining when understanding the root cause is easy/difficult based on our experience.\n\n---\n# Thank you\n\nYou have helped us improve our paper. Given these changes, we hope you consider revising your assessment of the paper's impact and the corresponding evaluation for NeurIPS. ☺️", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b0D6ggBgGd", "reviewer": "Reviewer_xkmy", "summary": "The paper titled \"Self-Healing Machine Learning: A Framework for Autonomous Adaptation in Real-World Environments\" introduces the concept of self-healing machine learning (SHML). This framework aims to address performance degradation in machine learning models due to distributional shifts. Unlike traditional concept drift adaptation methods, SHML focuses on diagnosing the reasons for model degradation and proposing targeted corrective actions. The paper presents a theoretical foundation for SHML, an algorithm (H-LLM) that uses large language models for diagnosis and adaptation, and empirical evaluations demonstrating the effectiveness of the approach.", "questions": "1. Can you provide more detailed results and analysis of SHML in a wider range of real-world environments beyond the simulated diabetes prediction task?\n2. How does SHML compare to other state-of-the-art concept drift adaptation methods in terms of computational overhead and sample efficiency?\n3. Can you elaborate on the specific contributions of each component of the SHML framework to the overall performance improvement?", "limitations": "The authors acknowledge the limitations of SHML, including the challenges in accurately diagnosing the root causes of performance degradation and the potential computational overhead of the approach. They also discuss the need for sufficient parallel processing capacity to handle the increased demands of multiple diagnostic and adaptation actions. The broader societal impact of SHML, including potential misuse in high-stakes applications, is also briefly addressed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The SHML framework is a novel approach to handling model degradation by diagnosing and addressing the root causes, rather than using reason-agnostic methods.\n2. The paper provides a solid theoretical foundation for SHML and demonstrates its practical viability through empirical evaluations.\n3. The concepts are well-explained, and the structure of the paper is logical and easy to follow. Figures and tables enhance the understanding of the proposed methods.\n4. SHML has significant potential in high-stakes applications where maintaining optimal model performance is critical, such as healthcare and finance.", "weaknesses": "1. While the paper demonstrates promising results, the empirical evaluation is limited to a simulated diabetes prediction task. Additional experiments in diverse real-world environments would strengthen the claims.\n2. The comparison with existing concept drift adaptation methods is not exhaustive. A broader set of baseline comparisons would provide a clearer picture of the advantages and limitations of SHML.\n3. The paper lacks detailed information on the implementation of H-LLM, especially regarding the practical challenges of deploying such a system in real-world scenarios.\n4. Although the authors mention the availability of the code upon acceptance, more details on the experimental setup and data used would help in reproducing the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer xkmy,\n\nThank you for taking the time to review our paper. We're happy you think that SHML is a novel approach with a solid theoretical foundation and that has significant potential in high-stakes applications. \n\nTo address your concerns, we've expanded our evaluation with **five new datasets**, **additional SHML effectiveness evaluation**, added a study with **six new benchmarks**, evaluated SHML across **10 different ML models**, and performed  **a qualitative ablation study**. We hope this information provides reasons to consider increasing your score. Our responses (A-H) follow:\n\n---\n# (A) More datasets\n\nWhile our paper's main contribution is the SHML theoretical framework, with viability studies as proof-of-concept, we agree more real-world examples improve the paper. **We've added five new datasets** common in concept drift adaptation, varying corruption levels at test time to evaluate adaptation methods. Please find this in **Table 1 in the response pdf**. \n\n**Takeaway**: $\\mathcal{H}$-LLM improves upon reason-agnostic baselines in five real-world datasets when the reason for performance drop can be diagnosed, even in test-time scenarios. This highlights the potential of self-healing ML.\n\nWe'll include these results with more experimental details in the updated paper.\n\n---\n# (B) Greater empirical analysis of self-healing ML\n\nTo address your concern about limited empirical analysis, **we've conducted an additional viability** study across the five datasets. We vary problem difficulty (corruption level) at test time and evaluate self-healing benefits (**Fig. 1 in the pdf**).\n\n **Takeaway**: $\\mathcal{H}$-LLM improves performance relative to baselines across five real-world datasets, with greater self-healing effects at higher corruption levels.\n \nWe'll include these results with more experimental details in the updated paper.\n\n---\n# \\(C) Limited benchmarks\n\nWe emphasize that our viability studies aim to show that any reason-agnostic strategy will perform poorly whenever it is important to understand the reason for performance degradation. This is because such strategies do not directly address the root cause.\n\nThat said, we've extended our evaluation benchmarks. **We've run an additional viability study with five benchmarks and four adaptive algorithms** common in concept drift adaptation (**Table 2 in the response pdf**). \n\n**Takeaway**: $\\mathcal{H}$-LLM consistently outperforms adaptation methods and adaptive algorithms that fail to address test-time corruption, demonstrating self-healing ML's feasibility in high-stakes environments where understanding model degradation reasons is important.\n\nAdditionally, we run one more viability study to evaluate whether these gains are consistent across different ML models. We show that $\\mathcal{H}$-LLM consistently outperforms other adaptation methods across 10 different ML models (shown in **Table 4 in the response pdf**), highlighting the model-agnostic nature of the framework.\n\n---\n# (D) Implementation of $\\mathcal{H}$-LLM\n\nWe're glad you're interested in $\\mathcal{H}$-LLM's practical implementation. Besides the main text, implementation details are in the appendix: components (**Appendix B.1.**), prompt templates (**Appendix B.2**), outputs (**Appendix B.3**) and viability studies (**Appendix C.2.**). That said, we acknowledge it might be difficult to find this information.\n\n**Actions taken**: Move input/output examples and prompt structures from appendix to main text.\n\n---\n# (E) Practical challenges of deploying self-healing systems\n\nWe're excited that you're interested in the practical deployment of self-healing systems. We answer this exact concern in **Section 5.1.** titled \"unique challenges of building self-healing systems\". We agree that the naming of this section could be improved.\n\n**Actions taken**: We will rename section 5.1. to *Practical challenges of deploying self-healing systems* to improve clarity. We will also expand the discussion in the appendix.\n\n---\n# (F) More details on experimental setup and data used\n\nIn the camera-ready version, we'll use the additional page to move some of the experimental details and dataset information described in **Appendix C** to the main paper.\n\n---\n# (G) SHML comparison to other concept drift adaptation methods\n\n**Computational overhead**. SHML methods have larger overhead than reason-agnostic approaches due to the self-healing system (LLM pipeline) identifying model failure reasons. Practically, it takes 20-40 seconds to implement a full pipeline and correct a model upon drift detection. This overhead is negligible for real-world systems given the benefits. Overhead may vary across systems.\n\n**Sample efficiency**. No differences exist as failure detection doesn't depend on sample size, but on self-healing pipeline complexity.\n\n**Actions taken**: Add an appendix discussing computational details for SHML systems.\n\n---\n# (H) Contribution of each component\n\n**We've performed an additional qualitative ablation study**, running a self-healing ML system and iteratively removing one component, inspecting its output (**Table 3 in supplementary pdf**).\n\n**Takeaway**: This study illustrates that all components are required for the self-healing ML system to work. Having said this, we see the robustness of self-healing ML as a big research agenda for future work and hope this spurs research in this area. \n\n---\n# Thank you\n\nWe believe these changes should greatly enhance the paper's contribution and improve its clarity. In addition to the clarifications which will be included in the camera-ready version, we have added **five new datasets**, **additional SHML effectiveness evaluation**, evaluation across **10 ML models**, a study with **six new benchmarks**, and **a qualitative ablation study**.\n\nThank you for your help.\n\nIf we addressed your concerns, we hope you consider significantly revising your assessment of the paper's impact and the corresponding evaluation for NeurIPS. ☺️", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful and positive feedback!

We are encouraged by the unanimous recognition of our self-healing ML framework's importance and novelty. The reviewers consistently described our work as "important" (**R-LnCp**, **R-4TYN**, **R-xkmy**) with a "novel and interesting" approach (**R-LnCp**). Our paper had a "well-structured presentation" (**R-4TYN**) and was considered as "well organised and easy to read" (**R-vweT**), with the structure "logical and easy to follow" (**R-xkmy**). Reviewers mentioned that our paper is "outperforming existing approaches in both presentation and results" (**R-4TYN**). We would like to note the consensus on two key themes we saw across the reviews:

- **Innovative and impactful approach**. Reviewers noted our work "lowers the barriers for others to implement adaptation actions in machine learning models" (**R-4TYN**) and has "significant potential in high-stakes applications where maintaining optimal model performance is critical, such as healthcare and finance" (**R-xkmy**). "Rather than using reason-agnostic methods" (**R-xkmy**), we show that "the reasons for performance drop should be considered and investigated" (**R-vwet**). In this way, our paper addresses "the growing challenge of maintaining machine learning models, especially given their increasing usage" (**R-4TYN**).

- **Robust theoretical foundation**. The paper provides a "solid theoretical foundation" (**R-xkmy**) and the method "is described with precise formalism" (**R-vweT**). This is complemented by empirical evaluations demonstrating its practical viability (**R-xkmy**). Reviewers saw our evaluation as "strong and thorough, which covers the details to help readers further understand the scope and the logic of the proposed method" (**R-LnCp**). Reviewers appreciated the "detailed inclusion of examples, definitions, assumptions, theoretical explanations, and results in the main text and appendix" which are "thorough and effective" (**R-4TYN**).


---
# Information in the supplementary pdf
We provide in total five new viability studies. That said, we would like to note that we see the primary contribution of our paper as formalizing self-healing machine learning which researchers and practitioners can use to build their own self-healing systems. 

We provide the following information in the supplementary pdf.
- **Table 1.** Additional viability studies involving **five real world datasets**. The datasets cover a wide variety of setups: Airlines (Bifet et al., 2010), Poker (Cattral et al. 2007), Weather (Elwell & Polikar, 2011), Electricity (Zliobaite, 2013), Forest Type (Blackard, 1998). We simulate real-world unexpected degradations by corrupting features at test time and evaluating models for different number of corrupted features and corruption values. **Takeaway**: $\mathcal{H}$-LLM better adapts at test time to issues that require reasoning about the structure of the data generating process across five datasets. This showcases the need for using self-healing systems in real-world environments.
- **Figure 1**. **Empirical insights** into the effect of self-healing on downstream accuracy for each of the **five datasets**. We systematically vary the corruption value and the number of corrupted columns and quantify the accuracy with and without triggering a self-healing mechanism. **Takeaway**: We find that the effect of applying a healing mechanism is largest when there is greatest corruption levels in the dataset. Furthermore, we find that applying a healing mechanism consistently improves downstream accuracy in the presence of data corruption. This showcases the importance of healing mechanisms in test-time environments.
- **Table 2**. Additional viability study involving **more benchmarks** on the original diabetes prediction task. We benchmark against an additional adaptation methods and four other adaptive algorithms. Results shown on the original diabetes setup described in the paper. We do not vary corruption values for space reasons. **Takeaway**: The additional benchmarks are unable to cope with adaptations when it requires reasoning about the structure of the data generating process.
- **Table 3**. **Qualitative ablation study** results for $\mathcal{H}$-LLM. We systematically remove one component of the system and inspect its outputs. **Takeaway**: All four components are required for self-healing to work. Removing any component results in poorer adaptations.
- **Table 4**. Additional viability study **evaluating each adaptation action across 10 different ML models**. We follow the same setup as in Table 1 by corrupting values at test time and varying the underlying model used. **Takeaway**: We show that self-healing is agnostic to the kind of model used. Self-healing ML can benefit any downstream ML model whenever understanding the reason for degradation is important (such as test time corruption).

---
# Thank you
The review has been extremely productive. We thank everyone for their help in shaping the paper to be in better form.

Meta-review: This paper introduces a new approach to the problem of drift adaptation in ML models called Self-Healing Machine Learning (SHML). The strengths of the paper include: (1) high novelty in addressing an important problem - the SHML approach to drift adaptation starts from the assumption that drift adaption must be driven by the reasons (root causes) of the shift in the first place, in contrast to prior work on this problem; and (2) significant technical contributions - a solid theoretical framework is provided, along with compelling proof of principle experimental results to demonstrate the viability of the approach (particularly with the expanded comparative results that show the approach to outperform existing "reason-agnostic" approaches); The paper opens up several interesting possibilities for future research.

The main weaknesses of the paper relate to various problems with aspects of the presentation, including such issues as aspects of the approach being unclear, the fact that in other places the information needed to properly understand aspects of the work are buried in the supplementary material, that the paper could do a better job of motivating the approach and highlighting its novelty from prior work upfront, and that the writing could be made more concise in some places. The authors in their rebuttal propose changes that largely address these deficiencies and correct various reviewer misconceptions and concerns caused by these problems. Please include all of these changes in the final version of the paper, along with the expanded comparative performance results mentioned above.
Decision: Accept (poster)