# End-to-end lifecycle machine learning framework for predictive maintenance of critical equipment

Jeremie Marchand, Jannik Laval, Aicha Seklouli-Sekhari, Vincent Cheutet, Jean-Baptiste Danielou

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Jeremie Marchand, Jannik Laval, Aicha Seklouli-Sekhari, Vincent Cheutet, Jean-Baptiste Danielou. End-to-end lifecycle machine learning framework for predictive maintenance of critical equipment. Enterprise Information Systems, 2025, 19 (1-2), 10.1080/17517575.2024.2448008. [hal-04990296v2](https://hal.science/hal-04990296v2)

HAL Id: hal-04990296

https://hal.science/hal-04990296v2

Submitted on 14 May 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# End-to-end lifecycle machine learning framework for predictive maintenance of critical equipment

Jérémie Marchand<sup>a,b</sup>, Jannik Laval<sup>a</sup>, Aicha Sekhari<sup>a</sup>, Vincent Cheutet<sup>a</sup> and Jean-Baptiste Danielou<sup>b</sup>

<sup>a</sup>INSA Lyon, Université Lumière Lyon 2, Université Claude Bernard Lyon 1, DISP UR4570, Villeurbanne, France; <sup>b</sup>INEO Nucléaire, Lyon, France

## ARTICLE HISTORY

Compiled May 14, 2025

## ABSTRACT

Industry is moving towards the adoption of data-based predictive maintenance (PdM). Machine Learning (ML) methods have emerged as a promising tool in PdM applications to prevent failures in equipment. The use of ML has been demonstrated in its ability to design and experiment, but obstacles appear when it comes to approaching the industrialization stage and operate in the long term. Many ML models are developed assuming a stationary data stream, which often does not hold true in real-world scenarios. Especially in the industry where non-stationary environments are predominant.

This paper presents a solution for long-term management of industrial data-based ML systems for PdM. It focuses on model maintenance, emphasizing dynamic self-updating to address concept drift. **The framework ensures expected performance throughout the lifecycle, particularly during usage and maintenance phases.** It features dual-level drift detection, drift severity quantification, human expertise integration, and end-to-end lifecycle management. An experiment with a multi-classification ML model for PdM failure detection was conducted. Results show the framework's ability to detect equipment behavioral changes through data drift identification. It also allows incorporation of new failure types without compromising detection of known failures after retraining. This framework contributes to developing trustworthy, effective AI systems for practitioners

## KEYWORDS

predictive maintenance; machine learning; end-to-end lifecycle; drift; non-stationary

## 1. Introduction

Equipment maintenance is an essential support function as it helps ensuring equipment reliability, service availability, and human safety. In the industry, corrective maintenance and preventive maintenance are predominant. The former aims to carry out

Email: jeremie.marchand@equans.com

Email: jannik.Laval@univ-lyon2.fr

Email: aicha.sekhari@univ-lyon2.fr

Email: vincent.cheutet@insa-lyon.fr

Email: jean-baptiste.danielou@equans.com
urgent interventions that occur after a system failure, while the latter aims to sched-
ule inspections at regular intervals to prevent potential failures.

Recently, a new maintenance strategy has emerged, known as predictive mainte-
nance (PdM). This strategy has received increasing attention in the industry, espe-
cially in the era of Industry 4.0, where systems are equipped with sensors that generate
real-time data streams, facilitating continuous monitoring, and prediction of its future
conditions. Techniques for PdM can be divided into three categories: knowledge-based,
model based and data-based approaches (Zhang, Yang, and Wang 2019).

Data-driven methods have been employed to enhance the accuracy of PdM require-
ments for various machines and operations (Coandă, Avram, and Constantin 2020).
Within Zero Defect Manufacturing, data-driven methodologies have proven to be more
effective than traditional analytical solutions, making them a viable alternative (Car-
valho et al. 2019). Data-driven PdM strategies have become increasingly important in
both industry and academia. A data-driven approach leverages available data to un-
derstand failure patterns and predict potential issues. Methods within the data-driven
approach can be categorised into either stochastic modeling or Machine Learning (ML)
(Davari et al. 2021; Le Nguyen 2023).

This paper is focused on the latter, where ML has gained importance. The common
practice is to collect a certain volume of data and then train a model on this batch of
data in multiple iterations until it converges. After training and validating the model,
it is deployed in production to make failure predictions.

However, many existing ML models are developed assuming a stationary data
stream, which often does not hold true in real-world scenarios, especially in the PdM
context. This leads to ML models becoming outdated over time due to unpredictable
changes in data distribution (Dreyfus et al. 2022). This is true in non-stationary envi-
ronment where dynamic phenomena are frequent (Ditzler et al. 2015). This dynamic
nature can arise from external or internal factors. Dynamic changes in data distribu-
tion can lead to a decline in model accuracy or even unexpected equipment failures,
posing significant risks to equipment maintenance outcomes (Hurtado et al. 2023).
Consequently, the model performance may be unexpected when dealing with input
samples lying outside the instance space covered by the training examples (Marcus
2018). This phenomenon is called drift and often occurs in a dynamic data stream in
a non-stationary environment. Drift is any change in the joint probability distribution
of input variables and targets (Gama et al. 2014). Using static approach, ML models
remain unchanged and cannot adapt to the evolving data patterns (Bayram, Ahmed,
and Kassler 2022). This drift requires re-training or updating the ML model on new
data. This leads to an effort to collect new data and revalidate the ML model.

A largely developed approach to tackle this issue has been in proposing models that evolve with new data input, like continuous or reinforcement learning (Ditzler et al. 2015), (Hurtado et al. 2023). These approaches have demonstrated some interests on several occasions by allowing an adaptation of the models when drift appears. Nevertheless, in this paper, we decide to have an other point of view on this issue by claiming that **the ML system has to be maintained over time**. Consequently, this study propose to adopt a systemic and lifecycle-based vision on PdM software, moving from a traditional ML model-centric view to a ML system-centric view.

With this consideration, one can notice that, while significant effort has gone into
developing data-based models for industrial applications, the maintenance over time of
these ML systems has received less attention (Dreyfus et al. 2022). Properly defining
when and how to measure these drifts and how to use the data for model updates
is essential to maintain data-based system. Neglecting these considerations can lead
to model prediction error, which could have severe consequences depending on the
industry's critical nature.

This paper focuses on this aspect of ML system maintenance in operational con-
dition, highlighting the importance of dynamic self-updating in the face of drift for
continuous PdM. It provides a **framework to ensure the expected performance
and behavior of ML systems throughout their lifecycle**, with a particular fo-
cus on the usage and maintenance phases. This will help practitioners in building
trustworthy and effective AI systems applied on critical equipment maintenance.

The paper is organised as follow. Section 2 discusses literature related to the area of ML model and system lifecycle. Section 3 introduces our concept for a new process supporting the end-to-end lifecycle of ML systems for PdM. This framework was employed in the experimentation presented in Section 4, followed by a detailed discussion of the results. Finally, Section 5, provides a summary of the paper and a discussion of future work.

## 2. Related Work

2.1. *PdM characteristics and problem specifications*

Regarding NF EN 13306 standard, the PdM is a “Condition-based maintenance car-
ried out following a forecast derived from repeated analysis or known characteristics
and evaluation of the significant parameters of the degradation of the item. In (Saley
et al. 2022), the authors present a detailed overview of PdM in the nuclear sector.
Although the application of ML methods in this field is still in its infancy, they are set
to improve reliability and confidence in critical equipment. This requires an in-depth
understanding of the behaviors of the equipment being monitored and the environ-
ments in which they operate.

The nature of manufacturing systems faces ever more complex and dynamic behaviors (Wuest et al. 2016), physical systems evolve over time, and the environments in which equipment operates are non-stationary. Various factors introduce dynamic changes (Zhou et al. 2023):

* Either due to an extrinsic cause of the system: for example, a change in operating mode to increase production in the factory, replacement of a sub-equipment in the system, seasonality, etc.

* Either due to an intrinsic cause of the system: ageing of system components (tool wear) or even ageing of the sensors observing the system (need to recalibrate sensors, etc.).

Therefore, physical systems follow a lifecycle, reflecting the continuous adjustments
and transformations driven by these dynamic influences.

All of these imply that the data distributions change over time. This shift in data
distribution create challenges for pre-trained models as their parameters are not ac-
customed to handling changes in the underlying data distribution. It may lead to
performance degradation during the system lifecycle (Bayram, Ahmed, and Kassler
2022). The cause of this phenomenon is often referred as **concept drift** (Widmer and
Kubat 1996; Moreno-Torres et al. 2012; Gama et al. 2014). The primary cause of a
model deterioration over time lies in the breach of the assumption that both the train-
ing data and the input data during inference originate from identical distributions.
For example, if a ML model only sees data on a few system failure modes, it will
fail to recognise other rarer failure modes that did not appear in the training data. Another example, is when functional changes and improvements are implemented in the systems, subsequently altering their behavior and the data patterns they generate.

As a consequence, changes in data distribution over time can be classified as follows (Lu et al. 2018; Webb et al. 2015):

* Sudden drifts: the change occurs abruptly.

* Incremental drifts: the drift increases incrementally over a period of time.

* Gradual drifts: the drift fluctuates over a period of time.

* Reoccurring/Recurrent drifts: the new drift occurs temporarily, then recedes after a period of time.

Therefore, it is challenging to envision updating a ML model dedicated to PdM with a significant delay, as it could affect the final production. An inefficient maintenance strategy can significantly impact the total operational costs of an industrial system (Na et al. 2021). PdM requires human-in-the-loop intervention on two essential levels. Firstly, it is essential for assessing the quality of the predictions generated. Secondly, it plays a crucial role in preventing automatic decision-making, thus guaranteeing essential transparency. This human involvement is particularly important in the context of safety-critical systems.

The concept of non-stationary environment is well-known in the literature (Sugiyama and Kawanabe 2012; Trojanowski and Michalewicz 1999; Rangachari and Loizou 2006; Sayed-Mouchaweh and Lughofer 2012) and covers a wide field of applications: electromechanical, chemical, thermal, medical, energy, etc. (Mouchaweh 2014), but its specific impact on the end-to-end ML lifecycle in the industry is less commonly addressed. In (Achouch et al. 2022) devoted to the challenges of PdM, authors explain that there are three challenging steps after developing a failure prediction model: integration, monitoring and updating. These challenges are addressed in this work with the notion of ML system lifecycle.

## 2.2. Overview of process lifecycle for ML projects

The literature related to the area of ML system lifecycle, has been studied from diverse perspectives in numerous fields. This section provides an overview of similar research and prepares for the proposed framework to be introduced.

Considering the standards aspect, DIN 92001 addresses the ML lifecycle based on the ISO 12207 standard which is dedicated to software lifecycle processes. The DIN 92001-2 standard mentions a list of AI quality requirements but does not go into details. In the ISO 22989 standard, only aspects of concepts and terms are mentioned. It is supplemented by the very recent ISO 23053/2022 standard (June 2022) which addresses the AI system lifecycle about the stages of data acquisition, data preparation, modelling, verification and validation, model deployment and operation but remains very generic. The ISO 23053/2022 standard presents the definitions of the different steps but does not go into the details on how to guarantee the quality of the model and make it operational throughout the end-to-end lifecycle: on the definition of the objectives, on the validation of the model quality assurance, on the detection and the monitoring of the drifts related to the input data and the models performance, on the way of how the re-training is driven.

Due to the lack of guidance through standards and development process lifecycle for ML applications, many project organizations use alternative processes that are closely related to data science and data mining, such as TDSP (The Team Data Science Pro-
cess)<fnref n="1" />, KDD (Knowledge Discovery in Databases) (Fayyad, Piatetsky-Shapiro, and Smyth 1996), SEMMA (Sample, Explore, Modify, Model and Access) developed by the SAS Institute and CRISP-DM (CRoss Industry Standard Process for Data Mining) (Wirth and Hipp 2000). The KDD and CRISP-DM methodologies existed since the 2000s. They are data mining oriented and despite minor differences they share a common representation of different stages of realization integrating feedback loops between stages.

In fact, CRISP-DM became the de facto industry standard process lifecycle for data mining in practice, with an expanding number of applications (Studer et al. 2021; Plotnikova, Dumas, and Milani 2020). However, CRISP-DM focuses on data mining and does not take into account the ML particularity of using an inference (Hanzelik, Kummer, and Abonyi 2022). Indeed, data mining extracts patterns and knowledge directly from data, while ML application consists in two steps: train a ML model on data, and then apply the resulting inference model on new data. In addition, CRISP-DM is too far away from contemporary software engineering practices, and contains little information on code deployment or collaboration within a cross-disciplinary team (Jackson, Yaqub, and Li 2019). The TDSP method introduced in 2016 by Microsoft (Coandă, Avram, and Constantin 2020), is the first method to introduce the particularity of ML modeling. However it does not go into detail on the monitoring and performance tracking part.

Breck et al. include this monitoring phase and go further by introducing 28 specific tests (Breck et al. 2017) to quantify issues in the ML pipeline to reduce the technical debt (Sculley et al. 2015) of ML applications. An internal study at Microsoft on ML projects challenges, suggests nine different ML workflow stages: model requirements, data collection, data cleaning, data labelling, feature engineering, model training, model evaluation, model deployment and model monitoring (Amershi et al. 2019).

The ML model has to be adaptable to a changing environment otherwise the performance of the model will degrade over time. This requires a permanent monitoring and maintaining of the ML model after deployment. An extension of CRISP-DM, which can take into account the adaptation due to data drifts, is advocated by Baier et al. (Baier, Jöhren, and Seebacher 2019). When incorporating new stakeholder requirements, it is necessary to conduct a fresh round of experimentation. This will involve the introduction of both the 'plan model' and the update plan steps, as elaborated in (Weber et al. 2019).

In their research, Studer et al. (Studer et al. 2021) proposed a process lifecycle designed for the development of a ML model deployed and maintained as part of a product or service. This process lifecycle is called CRISP-ML(Q) for 'CRoss-Industry Standard Process model for the development of ML applications with Quality assurance methodology'.

In (Lavin et al. 2022), the author introduces the Technology readiness levels for machine learning systems (MLTRL). They explain that in the level 9, so-called deployment, there is a significant need to monitor the current version and explicitly consider improvements for the next version. Monitoring data quality, drift and data creep is crucial. No AI system can be reliably deployed without thorough testing. Drift can have notable consequences and has become a fundamental consideration within PdM (Zenisek, Holzinger, and Affenzeller 2019). Industry-specific publications on drift

<footnote n="1">¹Microsoft. Accessed: Oct. 20, 2022. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/data-science-process/overview</footnote>
management are beginning to emerge (Kermenov et al. 2023; Zhou et al. 2023; Dreyfus et al. 2022).

As a result, the notion of monitoring data in end-to-end ML process lifecycle only arises at the beginning of the 2017s, the notion of maintenance appeared even later from the 2020s. The risk of not maintaining the model is the degradation of the performance over time which leads to false predictions and could cause errors in subsequent systems.

Algorithms for novelty detection in data streams generally decompose in two phases, namely offline and online (Faria et al. 2016). They indicate that in the offline phase, a set of labeled examples is used to induce a classifier. These labeled examples represent the known concepts about the problem. While the training process is usually conducted offline, inference very often has to deal with real-time constraints (Fischer et al. 2020). Traditionally in data mining, already collected data is processed in an offline mode. For instance, predictive models are trained using historical data given as a set of pairs (input, output). Models trained in such a way can be afterwards applied for predicting the output for new unseen input data (Gama et al. 2014). ML models for pedestrian protection and lane keeping assistant are trained using offline training data (dataset), then the ML system operates using online data as input (Riccio et al. 2020). In their work, (Zhang et al. 2020) explain that, at the very beginning, a prototype model is generated based on historical data, before deploying the model online.

There are tools and related literature to implement part of the model management system (MMS) (Bachinger, Kronberger, and Affenzeller 2021). But the current literature on MMS is still mainly focused on collaboration and tracking of ML experiments. There are various tools associated with the ML Operations (MLOps) paradigm (Symeonidis et al. 2022; Hewage and Meedeniya 2022), including those designed for versioning models and data like MLFlow and DVC, as well as tools dedicated to monitoring models, such as Arise<fnref n="2" /> and Fiddler³.

Concerning the issues of deployment and monitoring of ML models in operation, few papers go in that direction (Velox (Crankshaw et al. 2014), Clipper (Crankshaw et al. 2017), Prezel (Lee et al. 2018), Ease.ml (Aguilar Melgar et al. 2021)). These papers mainly focus on software architecture stack.

Practitioners have recognised the issue of performance degradation in ML models over time. They have gained increasing interest the MLOps to streamline deployment, monitoring, and retraining of ML models. However, practitioners have not incorporated mechanisms to adapt to this non-stationary environment.

## 2.3. Related Work Synthesis

This section presents a synthesis and critical analysis of the literature outlined in Figure 1. Traditionally, while the training process is usually conducted offline with historical data, inference very often has to deal with real-time constraints on new unseen input data.

In a traditional machine learning (TML) approach, it is generally assumed that the data used for training and the data used in production follow a stationary probability distribution (Sugiyama and Kawanabe 2012). It is commonly assumed that once a model has been trained to achieve the required quality, it is ready to be deployed and used without further updates or retraining (Vela et al. 2022). From a traditional and

<footnote n="2">²https://arize.com/
³https://www.fiddler.ai/</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1. TML lifecycle versus non-stationary environment issues</figcaption></figure>

macroscopic point of view, the lifecycle of ML system can be summed up and divided into two parts (Palli et al. 2022), namely offline and online. Following the retrieval of historical data, the project begins with the development and training of an 'offline' ML model, referred to as the BUILD. For PdM, these data can consist of physical measurements from sensors and failures from computerised maintenance management system (CMMS). Then, ML model is used in production on dynamic real-time data 'online mode', referred to as the OPERATE phase.

Despite the new arrivals input data, the deployed model remains constant, and its parameters do not change, even if the data characteristics have deviated significantly from those used during model training. This is the offline ML paradigm. One contributing factor is the inadequacy of training data to fully capture the complexity of the problem space. Another factor is the dynamic and evolving nature of the system environment, which undergoes continuous changes (Elwell and Polikar 2011). In a non-stationary environment, the ML models become obsolete over time (Dreyfus et al. 2022). It is necessary to update the model. This involves integrating new data from sensors, as well as taking into account online feedback from the field. But TML is not proactive, as it stops learning at the end of the training process (Le Nguyen 2023). In TML, the only possibility is to systematically create a new BUILD to maintain the model. The outputs of ML models are continuously used as a service to optimise production, detecting and anticipating failures that could impact this production. It is therefore difficult to envisage updating a ML model dedicated to PdM with a significant delay. A new BUILD phase involves waiting for data collection, rebuilding a new model offline, and then deploying it online. Furthermore, the economic cost associated with a new BUILD phase is not negligible. To minimise the delay between the BUILD and OPERATE phases, a new DEPLOY phase is emerging, driven by the adoption of the MLOps concept (Testi et al. 2022). However, the MLOps is still in its early stages and is currently in a messy phase (Ruf et al. 2021).
While academic research has focused on creating and optimizing the performance of ML models on static data or offline environments (Fujii et al. 2021), less attention has been paid to the deployment (Paleyes, Urma, and Lawrence 2022) and maintenance over time of ML system (Hanzelik, Kummer, and Abonyi 2022; Sculley et al. 2015). The observation is also corroborated in the field through surveys conducted with practitioners attempting to implement ML based solutions (Mäkinen et al. 2021). In the literature, there are interesting but highly disparate building blocks. What is lacking is a methodological and well-equipped framework for a lifecycle-oriented management, without the need to perform a complete BUILD phase each time a model degrades in performance. In recent ML process lifecycle, such as CRISP-ML(Q), monitoring and maintenance seem obvious. Similarly, in MLOps, continuous training has garnered increasing interest. But the consideration of a non-stationary environment and inherent drift appears less understood.

As far as we could push our investigations and our research efforts, no comprehensive framework has been identified in the literature that anticipates and integrates the maintenance of ML system over the long term, particularly for PdM. Our framework proposes a systemic view of the long-term maintenance of ML models dedicated for PdM. The proposed framework integrates the online domain knowledge from human feedback and defines a mechanism to detect and qualify a drift. This framework also includes an online strategy for retraining and validating ML models.

# 3. ML lifecycle framework for PdM

Our proposed framework is developed to encompass the end-to-end ML system lifecycle in an industrial context, specifically dedicated to PdM. This framework consists of four main phases: Build, Deploy, Operate and Maintain. Figure 2 provides an overview of the key steps and modules involved. The BUILD and DEPLOY phases are addressed to explain the overall approach of our framework. Our research is centered on the downstream phases of developing and validating offline ML models: the OPERATE and MAINTAIN phases. In these stages, our framework addresses and manages issues related to a dynamic environment over time, within which the equipment operates. Consequently, our framework highlights the importance of dynamically self-updating the ML model in the face of drift detection.

## 3.1. Phase BUILD

The objective of this phase is to build the ML model. A series of experiments is carried out to build and validate a ML model based on the historical data available. To begin this phase, it is crucial to take into account the distinct requirements outlined by the relevant use case, its domain, and the stakeholders involved. The initial step is to define the business requirements, while keeping in mind the objectives of the intended use cases. This very first step is called ‘Plan Model’ by (Weber et al. 2019).

During this phase, data scientists work on an integrated development environment to experiment and build a ML model that best generalises the available data history. Moreover, this environment is not directly linked to the real-time data flow, especially in a factory context where data is not readily accessible.

* Model Training / Offline Learning:
  This phase covers exploratory data analysis and data preparation (dataset con-
<figure><img src="image_4.png" /><figcaption>**Figure 2.** The proposed ML lifecycle framework for PdM</figcaption></figure>
struction). Next comes the modeling and evaluation stage, an iterative process involving model selection and hyper-parameter optimization, enabling the final model to be generalised (Spjuth, Frid, and Hellander 2021). At the end of this experimentation phase, ML artifact and non-ML artifact are produced (Santhanam 2020). ML introduces unique artifacts with unique characteristics (Paleyes, Urma, and Lawrence 2022): datasets, ML models and the derivated code. This encompasses the code for data preparation and feature engineering, responsible for crafting the dataset. The dataset is then utilised by the ML training code, which includes ML algorithm, hyperparameters, and other components, to generate the ML model (Kreuzberger, Kühl, and Hirschl 2023).

To build the ML model, the available historical data provides a partial picture of the equipment's behavior over its entire lifecycle. The knowledge space, or in other words, the space of input data available during training, does not represent the possible evolutions that the equipment may undergo (Hurtado et al. 2023; Dreyfus et al. 2022; Bayram, Ahmed, and Kassler 2022).

*   **Increase Input Space Test:**

    During the BUILD phase, a test phase was proposed for inclusion. The idea is to evaluate the model robustness (Tambon et al. 2021) through 'adversarial test inputs' and 'metamorphic tests' (Riccio et al. 2020), as well as using digital twins or (digital shadows) to expand the initial knowledge space (Zhong et al. 2023). This result of a simulation of a non-stationary environment before the ML model is deployed and used in production. This sub-module will be the subject of future work outside the scope of this paper.

## 3.2. Phase DEPLOY

During this phase, the code generated in the BUILD phase is packaged using MLOps concept. Practitioners face challenges at each stage of the ML deployment workflow, some of which are shared with running regular software services, and some are unique to ML (Paleyes, Urma, and Lawrence 2022).

*   **Packaging:**

    The code used for experiments in BUILD phase, usually in the form of notebooks, is not directly deployable in production. There is a step to transform the code from experiments into reusable modules or standalone scripts to facilitate their use and maintainability (Symeonidis et al. 2022). The aim is to design an automated pipeline using DevOps core principle which is called DevOps pipeline or CI/CD pipeline, CI for 'Continuous Integration' and CD for 'Continuous Delivery' (John, Olsson, and Bosch 2021). The CI/CD pipeline is the top-level component of continuous integration, delivery, and deployment. It consists of series of processes that automatically run tests and deploy application.

Compared to DevOps, MLOps completes additional testing procedures on data, and models validation. In addition to CI and CD, MLOps introduces also a new practice known as 'continuous training' and 'continuous monitoring', which aims to automatically and continuously retrain the model when data drift or when the model performance is degrading (Ruf et al. 2021). These elements are more detail in OPERATE and MAINTAIN phases, sections 3.3 and 3.4. In (Kreuzberger, Kühl, and Hirschl 2023), authors talk about 'ML workflow pipeline code' and 'model serving code'. A ML pipeline refers to a set of interconnected processes and operations (or a list of sequential transformations),
used to transform raw data into a useful model. It involves a series of steps that
typically starts with data acquisition, preprocessing, feature engineering, train-
ing, and evaluation. The pipeline ensures that the processing steps applied to the
data are clearly outlined, ordered, and the parameters applied are recorded. This
helps to maintain order, replicability, and ultimately makes it easier to apply the
model to new data sample.

These best practices ensure the quality assurance of the code delivered to pro-
duction, and reduce the technical debt (Sculley et al. 2015). They also speed up the
transition phase (packaging of the solution) for eventual design a ML model (Ruf et al.
2021) from a new BUILD phase.

3.3. *Phase OPERATE*

At the end of phases in offline environment, the ML model is the result of learning in a
partial knowledge space at a given time ‘t’. This model is then applied to new dynamic
data. Therefore, when the data deviates from this knowledge space during equipment
operation (in a non-stationary environment context with concept drift) (Fernandes,
Corchado, and Marreiros 2022; Zenisek, Holzinger, and Affenzeller 2019), the ML
model will produce low-quality predictions, leading to a decline in model performance.
All phenomena linked to the organization are considered ‘organizational’ drift, and will
require manual training by the operator. All unforeseen phenomena (ageing, unusual
breakdowns, sudden events) are considered ‘unplanned’ drift.

In both cases, the information system embedding the ML model must integrate
new components (Lu et al. 2018). These modules are a drift detection mechanism, a
decision-making procedure, an embedded update module and a mechanism to confirm
that the new model is working properly. Compared to the existing framework in lit-
erature, the following described modules are innovative. The drift detection module
is explained below, the other modules are part of MAINTAIN in section 3.4 and are
linked with physical equipment in section 3.5.

*   **Model Serving:**
    The active ML model (ML inference) in production is used on a constant real-time stream of new data to generate predictions (Fischer et al. 2020). If no drift is detected, the predictions are sent or incorporated into a dedicated decision-support tool, such as for PdM software.

*   **Drift Detection:**
    There are different types of data drift, depending on the type of change in this
    joint probability distribution and a causal relationship, between inputs and tar-
    gets (outputs). This module enables the detection of performance drift in the
    model (commonly referred to as ‘concept drift’, related to the target to be pre-
    dicted ‘y’), and the detection of input data drift (related to features ‘f(x)’, com-
    monly referred to as ‘data drift’, ‘covariate shift’ or ‘virtual drift’) (Bayram,
    Ahmed, and Kassler 2022). Data drift occurs when there is a change in the dis-
    tribution of input data, while the relationship between inputs and outputs (e.g.,
    machine failure) remains unchanged. For example, aging vibration sensors may
    produce noisier data, shifting the vibration data distribution without altering
    the relationship “high vibration = failure.” Concept drift happens even when
    the distribution of input data (like temperature and vibration) remains stable.
    Instead, the relationship between this data and the output (failure) changes. For
instance, a machine may start to fail for reasons unrelated to vibration characteristics, making existing vibration data insufficient for predicting failure. Data drift and concept drift can occur simultaneously, but not for the same reasons. If new causes of failure arise (concept drift), it may also lead to changes in certain input data distributions (data drift). For example, if bearing wear becomes a major cause of failure, it might result in altered readings from vibration or temperature sensors. Thus, both concept drift (changing failure causes) and data drift (shifting data distributions) could be observed.

Detecting drifts has traditionally been tackled as a supervised task, where labeled data is consistently employed to validate the trained model (Sethi and Kantardzic 2017). While these methods have proven to be effective in identifying drifts, they are not always practical. The strategies demand full and rapid access to the true labels, which is an unreasonable assumption in most real-world applications (Bhattacharya 2022). There is also the problem of delayed arrival of the labels (Le Nguyen 2023). The decision making applications includes diagnostics. The ‘ground truth’ is usually delayed, i.e. the true answer whether the decision was correct becomes available only after certain time (Žliobaitė 2010). It is therefore important to measure both data drift and model performance drift in this module. Based on the literature, this taxonomy of drift detection methods, coupled with the decision component of our framework, is proposed in Figure 3.

<figure><img src="image_5.png" /><figcaption>Figure 3. Drift Detections Taxonomy and associated Decision</figcaption></figure>

The output drift detection monitors the output of ML model. If the prediction performance decreases, it is considered as a drift. Thus, the appropriate approach is the error rate-based drift detection. A comparison is done between a performance reference (during the initial training) and the real time performance. To perform this calculation, labeled data from ground truth feedback is required. This feedback is not always immediate. So the output drift detection can be referred to as ‘Delay Drift Detection’.
Error rate-based methods determine drift indirectly as they presume that the changing distributions cause the significant changes of the performance metric. On the other hand, distribution-based drift detectors deal with the drift problem from the root sources, where they monitor the distribution drift. In the case where a concept drift is confirmed, an update of ML model is proposed to Human Decision.

Regarding the input drift detection, the aim is to estimate the data distributions' similarity for each feature used during the model training. The similarity is calculated between two different time-windows (historical data used in initial model training and real-time data). Input drift detection, referred to as 'Immediate Drift Detection,' can also be extended to unsupervised drift detection methods, where changes in data distribution are identified without relying on labeled data, enhancing the detection process in scenarios with limited or no labeled information. For this part different unsupervised algorithm can be used. This requires several steps:

* Step 1: Define a reference window over past data.

* Step 2: Define a window over the most recent data.

* Step 3: Choose a dissimilarity measurement. In the 'Error rate-based' approach, drift detection relies on the prediction error of the ML model's output. It involves comparing the prediction error rate values between a reference window (step 1) and new data from step 2. In the 'Data Distribution-Based' approach, drift detection is performed using the input features of the ML model, leveraging unsupervised methods. This involves comparing the feature distributions between the reference window (step 1) and the new incoming data (step 2). Metrics such as the Wasserstein distance can be employed to measure differences between probability distributions. Additionally, statistical tests like the Kullback-Leibler divergence or the Kolmogorov-Smirnov test can be used to evaluate distribution similarity. Other unsupervised algorithmic methods may also be applied to enhance the drift detection process.

* Step 4: A dissimilarity significance test. In this step the statistical significance of the change observed in step 3, or the p-value is evaluated (Lu et al. 2018). This step is used to calculate the drift severity.

This gives the ability to zero manufacturing industry to prevent and anticipate any issue by generating an alert to factory production validated by 'Human Decision' sub-module. In the framework, the unsupervised data drift detection module can operate solely, however this configuration limits the information provided to maintenance operators. In such cases, the maintenance operators do not receive the exact nature of the failures, but rather an alert with no further information. This is why a two-tier approach is advocated: an unsupervised layer using data drift detection and a supervised layer with human feedback in the form of ground truth to identify in the future prediction the nature of failure.

The drift detection enables predictions and data to be controlled and supervised. This element combines to the main 'MAINTAIN' module guarantees model performance over time.
### 3.4. Phase MAINTAIN

The purpose of this step is to update the ML model in case of confirmation of a drift detection, and a mechanism to validate the new models trained. This update is confirmed by a human-in-the-loop control points decision. Integrating the human dimension into decision-making is important for industries with demanding safety-critical systems, such as the nuclear industry.

* Drift severity:

The notion of drift severity may have different names in the literature (drift rate, drift magnitude, ...) (Webb et al. 2015). Drift severity is simply the distance between the concepts at the start and end of the drift period. For drift detection based on error rate, the magnitude is the degree of decrease in prediction performance measures (Lu et al. 2018). For data distribution, the step 4 of data drift detection is used as a basis. A threshold is defined to confirm the data drift detection. The drift severity is expressed as a percentage relative to the threshold of the method used to assess dissimilarity. For a statistical test, the p-value is used as a reference for drift severity. The null hypothesis is rejected with increasingly restrictive p-values, starting from a p-value threshold reference inferior than 0.05.

$$ \text{Drift severity}_{\text{p-value}} = \left( \frac{\text{threshold}_{\text{ref}}}{x} \right) \times 100\% $$

The distance (or divergence) between probability distribution can be used as a reference for drift severity. For example, in the case of Wasserstein distance, the distance threshold reference can be set to 0.1.

$$ \text{Drift severity}_{\text{distance}} = \left( \frac{x}{\text{threshold}_{\text{ref}}} \right) \times 100\% $$

In these equations, x is the dissimilarity measure at time t.

If minor drift is detected (Drift severity ∈ (100, 150)), a new ML model is automatically re-trained. The initial ML model is still used in operation until the decision to replace it is finalised (in 'Shadow Model Management' and 'Human Decision' modules). In the event of a major drift, a 'Human Decision' is required to identify the cause (e.g. inconsistent sensor data, etc.). Subsequently, the choice can be made either to proceed with on-board re-training ('New model learning') or either to initiate a complete 'BUILD' experimentation phase afresh.

* New Model Learning:

The non-stationary changes are too numerous to undertake a new BUILD phase at each time, which would take too long. Local relearning, directly at the edge of the equipment through a 'New Model Learning' module, is preferred. This module is dedicated to retrain the ML model while considering the new data. These new data are composed of new input data from sensors and corresponding ground truth feedback on prediction. The new inference (or new ML model) is generated using the 're-training pipeline' code packaged during the DEPLOY phase. For (Dreyfus et al. 2022) the notion of 'Updating System' is introduced. In (Habibullah, Gay, and Horkoff 2023), authors have conducted interviews and a survey to understand how non-functional requirements (NFRs) for ML sys-
tems are perceived among practitioners. From both industry and academia, the
‘retrainability’ has emerged as one of the important NFRs. Different adaptive
learning strategies can be envisaged: either full retraining or incremental learn-
ing (Le Nguyen 2023; Khamassi et al. 2018). Adaptive learning algorithms can
be seen as advanced incremental learning algorithms that are able to adapt to
the evolution of the data-generating process over time (Gama et al. 2014).

* Shadow Model Management:

The new inference is compared in parallel with the active inference used in
operation to monitor whether the new model produces better results (Lavin
et al. 2022), (Treveil and Dataiku Team 2020). The word ‘Shadow’ refers to a
technique that involves employing real time production data on a new version of
the ML model without the model actually returning the response or prediction
to clients or other systems. Instead, the old version of the model continues to
provide responses or predictions, and the outcomes of the new version are simply
captured and stored for analysis purposes. ML models are versioned and stored
in a database for eventual version roll-back (Baier, Jöhren, and Seebacher 2019).

* Human Decision:

In this step, the updated ML model is validated by a human-in-the-loop control
points decision. After a last validation done by a human, if the new model meets
the expectations, it replaces the old model in production. Otherwise, if it is not
sufficient, an in-depth design phase is initiated in offline mode. During this new
BUILD phase, new experiments, modeling and further optimization of hyper-
parameters will be carried out. Subsequently, a new deployment is initiated and
expedited, leveraging the best practices of MLOps (Testi et al. 2022). At this
stage, the team in charge of monitoring the model may also decide to raise an
alert about a risk of failure linked to the detected drift given by the Drift Detector
module (‘Update Equipment’ arrow in Figure 2 and 4 ). This human checkpoint
is important, particularly for industries with demanding safety-critical systems,
such as the nuclear industry. But for less sensitive industries, it is possible to
dispense with human intervention by automating the decision.

**3.5. Links with physical equipment**

Use and maintenance mechanism of ML models and ML system at the PdM software
level are illustrated on the left Figure 4. On the right, the classic equipment lifecycle it
represented. Informations are exchanged between software and equipment operational
side. The right-hand ‘Factory Production Equipment Level’ is made up of three stages:

(1) **Operational production**: use of equipment under normal operating conditions (e.g. production of chemical raw materials at equipment output).

(2) **Production Control**: monitoring the production output of the equipment. In this stage, the predictive decision aids provided by the PdM software are taken into account.

(3) **Maintenance**: maintenance is carried out on the equipment.

Our framework also takes into account feedbacks and requests from ‘Factory Produc-
tion’:

* Human Decision forced re-train: At any time, the operator using the equipment may decide to re-train the ML model in the event of anticipated and known changes on the production or equipment side. These changes may be an increase
<figure><img src="image_6.png" /><figcaption>**Figure 4.** ML and equipment lifecycle, with a particular focus on the OPERATE and MAINTAIN phases of ML models</figcaption></figure>

in production, maintenance on the equipment, reuse of the ML model on other similar equipment, etc. Before re-training, the user must select a date range representative of the new behavior from the historical data available online.

* **Human feedback:** The operator of the equipment in production can qualify the failure prediction that has been communicated. This interaction with human illustrates the principle of Active Learning (Settles 2009). The importance of using 'external feedback' to update models, if available, is explained in (Faria et al. 2016). This qualification of the prediction (by human feedback) is capitalised in the MAINTAIN module to evolve the ML model and take ground truth into account. This feedback can either confirm that the fault is correct or indicates an error in the 'prediction' that has been communicated (this is known as a 'false positive', or an incorrect fault qualification). In both cases, this information is stored in a database to expand the knowledge space. If the ML model produces inaccurate predictions, a new training is initiated, and the resulting model is concurrently compared ('shadow management') for validation by the team responsible for maintaining the PdM software. The implementation of the human-in-the-loop mechanism at this stage allows for continuous improvement of the ML model based on ground truth.
## 3.6. Synthesis

Industrial scenarios mainly take place in non-stationary environments. This dynamic nature can result from external or internal factors. However, most ML models are trained offline and remain static.

To address these issues, a framework is introduced. Our framework is developed to detect intrinsic drift in the monitored system, such as the aging of system components. Furthermore, in cases of organizational dysfunction (lack of communication regarding changes) between the operational side in charge of factory production and the maintenance side in charge of equipment maintenance, the framework enables detection of extrinsic system drift. It assesses supervised drift, in obtaining ground truth feedback with a delay, and unsupervised drift, enabling the prompt detection of changes in equipment behavior. It embeds a mechanism for automatically retraining and competing with the newly updated models. It also allows for on-demand retraining initiated by humans through a production-side factory interface. For example, retraining is initiated in response to a change (maintenance intervention, production change, etc.). It integrates and capitalises on feedback from the field (ground truth), in order to correct and improve our model in production. It also accelerates the deployment phase by advising the use of MLOps best practices. In the extreme case where no failure data is available, our framework allows the integration of a model learned solely on normal operations. Then, it gradually incorporates ground truth feedback to predict failure cases while alerting on any behavior drift through input drift detection.

# 4. Experimentation

## 4.1. Experimentation purposes

This section presents an experimental implementation of our framework, specifically focusing on the detection aspect of PdM. The experiment aims to evaluate the robustness and flexibility of the proposed framework in addressing the specific challenges of fault detection in PdM. While this study concentrates solely on the detection component, it provides a solid foundation for future extensions towards prognostic capabilities. The primary objective of this experimental phase is to validate our proposed framework at Technology Readiness Level (TRL) 4, which corresponds to component validation in a laboratory environment. This crucial step in the development process aims to demonstrate the fundamental functionality of our approach under controlled, simulated conditions that closely approximate real-world scenarios. This experiment presents the non-stationary behavior of equipment by simulating a new, unknown failure. Simultaneously, it substantiates the validity of our framework regarding the long-term maintenance of a ML system, achieved by detecting and effectively managing drifts.

Wear and tear or fatigue failure are phenomena that occur over the long term. Retrieving industrial data containing failures is a dual challenge, primarily due to the heterogeneous nature of the data sources and, more importantly, because the objective is precisely to prevent such failures. Providing a reasonable testbed for drift detection on time series similar to those from industrial condition monitoring, is a non-trivial task (Zenisek, Holzinger, and Affenzeller 2019). To demonstrate the effectiveness of the proposed method, experimental results are provided using a simulated industrial equipment. By simulating a model of the equipment using its physical representation, it becomes possible to adjust the parameters, allowing for the generation of simulated
failure scenarios. This experiment facilitates the management of different drift dynamics, which can be sudden, incremental, gradual, or recurrent. It demonstrates that our framework supports the detection of drift in two different ways: one for detecting a data drift in input and another for identifying a concept drift based on ground truth feedback. Data drift is detected by monitoring and calculating the drift in the ML model input. Concept drift is identified by monitoring and calculating the offset in the ML model output prediction based on ground truth feedback. More precisely this experimentation allows to detect and manage a concept drift evolution. This accelerates the updating of the ML model without the need for a lengthy BUILD phase. In this experiment, our primary emphasis is on validating a specific aspect of the framework, specifically the phases related to OPERATE and MAINTAIN, while the other components will be the subject of future work.

## 4.2. Experimentation presentation

For this experimentation, a digital model is created and based on a triplex pump in Matlab Simscape model<fnref n="4" />, illustrated in Figure 5. The use of a digital model overcomes the limitations of a specific spatial and temporal scale. The simulation of degraded operating states shows the non-stationary nature of the equipment over time (Zhong et al. 2023). In this physical model, a CAT Pumps model 1051 is simulated<fnref n="5" />. Three plungers are attached to a single crankshaft with 120 degrees out of phase crankpins. The Simscape model of the pump can be configured to model degraded behavior due to seal leakage, blocked inlets, bearing wear, and broken motor windings. The digital model is used to generate training data for the ML model in offline and is used to simulate online data for the ML model deployed.

### 4.2.1. Phase BUILD

A history of flow measurement and failures linked to a blocked inlet fault is available during the BUILD phase. These data are communicated to the data scientist teams in offline mode through csv files. The pump output flow is the result simulated by 100 progressive blocking factor parameters variations. Each simulated mesure is done during 2 secondes period with a frequency of 1000 Hz. The total raw dataset size is equal to 400000 rows. Each mesure is separated by a time interval of 4 hours to avoid a large quantity of data and to anticipate the measurements with a future IoT sensor. The features ‘X’ are based on the output flow spectrum where flow statistic (mean, variance, kurtosis, etc.) and common signal characteristics (rms, peak2peak, etc.) are computed. Condition indicators are extracted from the flow signal every 2 seconds, resulting in one row of data per 2-second interval. Following this extraction process, the final dataset is compiled, incorporating these condition indicators derived from the flow signal. This extraction process yields a final dataset comprising 400 rows. The dataset is divided into training and testing sets using an 80/20 ratio, with 80% of the data allocated for training and the remaining 20% reserved for testing. Two scenario, with differents feature list, are created to represent the lifecycle that a ML model can have during the BUILD phase, as follows:

* Scenario 1: pMid, pHigh, pKurtosis, qMean.

<footnote n="4">⁴https://github.com/mathworks/Simscape-Triplex-Pump/tree/master</footnote>

<footnote n="5">⁵https://www.catpumps.com/products/pumps/15-frame-plunger-pump-1051</footnote>
<figure><img src="image_7.png" /><figcaption>Figure 5. Triplex Pump simulated</figcaption></figure>

* Scenario 2: fPeak, pLow, pMid, pHigh, pKurtosis, qMean, qVar, qSkewness, qKurtosis, qPeak2Peak, qCrest, qRMS, qMAD, qCSRange.

Scenario 1 provides the features list used for the initial ML model design in the first BUILD phase. This is followed by the DEPLOY, OPERATE and MAINTAIN phases. At the end of this lifecycle instance, a second design is decided. Scenario 2 provides the features list used for a second design of ML model during the second BUILD phase.

The target 'y' is composed of 0 and 1 values for respectively no failure and failure linked to a blocked inlet. From this knowledge, a ML (Multi class default detection) model is learned on Python with scikit-learn ML library using the algorithm 'Random-ForestClassifier'. Performance validation is evaluated in terms of the F1 score. A cross validation has been performed for feature selection, to optimise the hyper-parameters of the ML model and to test its generalization property to unseen data. At this step, the F1 score ML model performance is perfect and equal to 1 as illustrated in the first row 'BUILD phase (Offline train)' of Table 1. As a result, by monitoring the pump output pressure, this ML model can detect which parts of the pump are failing based on the learned knowledge space available at time t.

### 4.2.2. Phase DEPLOY

The ML model and the training code are packaged and put into production to operate prediction ('served'). In this stage, unstructured codes from data scientist experimentation in BUILD phase are transformed to structured and reusable modules. Furthermore, all necessary modules for monitoring and maintaining the ML model are created. DevOps ('CI' and 'CD') and MLOps ('continuous training' and 'continuous monitoring') good practices are not detailed in this paper. In this step, it is assumed
that all the code pipelines are implemented, along with processes that automatically run tests and deploy the application.

### 4.2.3. Phase OPERATE

At this stage, the ML model resulting from the BUILD phase operates on online data. These online data are simulated with the Matlab Simscape digital model. At the beginning when everything goes well, the ML model identifies the weak signals of a blocked inlet fault or does not notice any failures. Then, a new leak seal failure on the plunger is simulated by modifying progressively the orifice area parameter. The input data drift detector is done with a Kolmogorov-Smirnov (KS) test between train reference value and new data stream value. The KS test compares the empirical cumulative density functions F_P and F_Q. P is the training data (reference set) on which the ML model was trained and Q is the data on which the ML model is performing predictions (inference set).

$$KS(P, Q) = \sup_{x}(Fp(x) - Fq(x))$$

The scipy.stats.ks_2samp python function has been used to perform this test with default parameters settings. The null hypothesis is that the two distributions are identical. This test is done for each feature and for each new input data stream. In the decision-making process, a more stringent significance level by setting a p-value of 0.01 is opted rather than common used p-value equal to 0.05. The null hypothesis is rejected if the p-value falls below 0.01. Selecting this lower p-value threshold effectively reduces the risk of making a type I error, occurring when mistakenly rejecting a true null hypothesis. A p-value of 0.01 reflects a preference for conservatism in drawing conclusions, ensuring strong evidence against the null hypothesis before making any decisive claim.

An alert is sent to the factory production by respecting the MAINTAIN treatments modules (Drift severity, Humain decision). Then, maintenance is done on the equipment side and confirms the presence of a failure. This failure is qualified as 'leak seal failure' by human feedback experience. In parallel, no fault detection is performed by the initial classifier, which is unaware of the equipment's change in stationary. The physical measurement conditions for a 'leak seal failure' failure were not learned by the initial ML model.

If the ML model is left in its current state, the ML model performance will deteriorate. If this failure happens again (under similar conditions), it won't be able to detect the leak seal failure. The ML model performance decline is detected by the drift detection using performance based indicators.

For online data simulation, slightly different initial conditions are performed to ensure the robustness of the new ML model and get as close as possible to the real conditions of defect occurrence. This procedure is repeated for other new type of failure (single failure or multiple failures combined together).

### 4.2.4. Phase MAINTAIN

The severity of data drift is calculated with Drift severity<sub>p-value</sub> = (0.01/x) × 100%. An additional treatment is added to prevent false alarms generation, the cumulative sum of detections over an observation period (here each new measurement generated by simulating a new fault) is calculated. This cumulative sum of detection must be >
1. The number of features 'drifted' is communicated to the 'Human Decision'. If at least, one data drift is detected and confirmed (by several consecutive drifts for the same feature), an alert is set to the factory production to indicate a change in the equipment behavior.

The severity of concept drift is detected by a abrupt change of F1_score performance. In this case, maintenance is carried out on the equipment, human feedback is collected and added to the knowledge space. A 'new model learning' is generated using the same code on this expanded knowledge space. The ML model retraining process integrates both historical and newly acquired data to enhance predictive accuracy. The training dataset comprises:

* Historical data: Previously collected and validated information.

* New data: Recent sensor inputs coupled with their corresponding ground truth labels, which serve as feedback on the model's predictions.

Furthermore, we adopted a full retraining approach as our adaptive learning strategy. This method involves rebuilding the entire model using the comprehensive dataset, allowing for a thorough incorporation of new patterns and relationships in the data. Our retraining process focuses on a specific subset of data centered around human feedback, rather than including all historical data. This targeted approach significantly reduces computational time, effectively addresses concept drift by focusing on data points likely to represent new patterns, and avoids issues related to imbalanced datasets common in predictive maintenance scenarios. The initial model is compared with the new model in 'Shadow Model Management', confirming that retraining yields to better results. Following human validation, the new model is put into production and used for future predictions.

### 4.3. Results

The ML model performance online is tested for each occurrence of a new type of failure. The emergence of a new failure leads to a concept drift evolution. Retraining online following the detection of the drift allows the ML model to regain good performance.

Retraining is conducted by incorporating data resulting from the detection of a single fault. This accounts for 4 new rows in the dataset, corresponding to the occurrence of the fault at different time points (4 different instances separate of 4 hours). Retraining is carried out on little data to get as close as possible to reality, where few fault observations are available (unbalanced classification issue (Palli et al. 2022)). Tests after retraining are performed on 20 new rows of the dataset that were not used in the initial training to avoid any issue related to data leakage. Thus, the quality of the ML model relearning is tested for non-failure instances across twenty new time points. For new faults, it is tested across twenty new instances of the same fault occurrence. Tests are carried out on more data to validate the robustness of our results. To ensure the reliability of our results considering the non-deterministic nature of the RandomForestClassifier model, the experiment is rerun twenty times and averaged the outcomes.

Table 1, presents a comprehensive overview of the model's performance metrics for each identified concept drift evolution. This data illustrates the effectiveness of our framework in maintaining model accuracy across various drift scenarios. In most of cases, online retraining significantly enhances the ML model's performance. Without retraining, the initial ML model fails to detect new fault types, resulting in an F1_Score of 0 for these novel failures. This outcome confirms the presence of concept drift evo-
<table><caption>**Table 1.** Result of Scenario 1 before or after re-training with features: pMid, pHigh, pKurtosis, qMean.</caption><thead><tr><th>Event</th><th>ML Model and data used</th><th>F1_Score</th></tr></thead><tbody><tr><td>BUILD phase (Offline train)</td><td>Offline Original Model (train with None and Blocking data)</td><td>1</td></tr><tr><td rowspan="2">OPERATE phase (Online)</td><td>Online Original Model on new None failure data</td><td>1.0</td></tr><tr><td>Online Original Model on new Blocking failure data</td><td>0.95</td></tr><tr><td>New Failure (concept evolution)</td><td>Online Original Model on new Leak failure data</td><td>0</td></tr><tr><td rowspan="4">Re-train (with Leak failure data)</td><td>Online New Model on new None failure data</td><td>1.0</td></tr><tr><td>Online New Model on new Blocking failure data</td><td>0.87</td></tr><tr><td>Online New Model on new Leak failure data</td><td>0.91</td></tr><tr><td>Online Original Model on new Bearing failure data</td><td>0</td></tr><tr><td rowspan="3">Re-train (with Bearing failure)</td><td>Online New Model on new None failure data</td><td>1.0</td></tr><tr><td>Online New Model on new Blocking failure data</td><td>0.87</td></tr><tr><td>Online New Model on new Bearing failure data</td><td>0.88</td></tr><tr><td>New Failure (concept evolution)</td><td>Online Original Model on new simultaneously Leak and Bearing failure data</td><td>0</td></tr><tr><td rowspan="3">Re-train (with Leak/Bearing failure)</td><td>Online New Model on new none failure data</td><td>1.0</td></tr><tr><td>Online New Model on new Blocking failure data</td><td>0.87</td></tr><tr><td>Online New Model on new Leak/Bearing failure data</td><td>0.99</td></tr><tr><td>New Failure (concept evolution)</td><td>Online Original Model on new simultaneously Leak and Blocking failure data</td><td>0</td></tr><tr><td rowspan="3">Re-train (with Leak/Blocking failure)</td><td>Online New Model retrain on new None failure data</td><td>1.0</td></tr><tr><td>Online New Model on new Blocking failure data</td><td>0.87</td></tr><tr><td>Online New Model on new Leak/Blocking failure data</td><td>0.43</td></tr></tbody></table>

lution, necessitating model updates. A comprehensive retraining process, incorporating newly labeled faults, enables the detection of similar defects while preserving the model's performance for known failure types and non-fault cases. Post-retraining, the F1_Score consistently achieves or exceeds 0.87, with one notable exception: the simultaneous Leak/blocking fault, where retraining alone proves insufficient (F1_Score: 0.43). To address this challenge, the 'Human Decision' sub-module initiates a new, BUILD phase (Scenario 2). This phase culminates in an expanded feature set, increasing the total to 14 features. This scenario employs the same 'RandomForestClassifier' ML algorithm but incorporates 10 additional features: fPeak, pLow, qVar, qSkewness, qKurtosis, qPeak2Peak, qCrest, qRMS, qMAD, and qCSRange. These enhancements aim to improve the model's predictive capabilities by capturing a wider range of signal characteristics.

Table 2 presents the performance metrics following the implementation of Scenario 2, which incorporates an expanded set of 14 features in the new BUILD phase. The results demonstrate a marked improvement in the model's predictive capabilities, with F1-Scores now reaching acceptable levels across all fault categories, including those previously challenging to detect.

<table><caption>**Table 2.** Scenario 2 after a new BUILD with features: fPeak, pLow, pMid, pHigh, pKurtosis, qMean, qVar, qSkewness, qKurtosis, qPeak2Peak, qCrest, qRMS, qMAD, qCSRange.</caption><thead><tr><th>Event</th><th>ML Model and data used</th><th>F1_Score</th></tr></thead><tbody><tr><td>New BUILD phase (Offline train)</td><td>Offline Model (train with None, Blocking and Leak/Blocking data)</td><td>1</td></tr><tr><td rowspan="2">OPERATE phase (Online)</td><td>Online Model on new None failure data</td><td>1</td></tr><tr><td>Online Model on new Blocking failure data</td><td>0.99</td></tr><tr><td></td><td>Online Model on new Leak/Blocking failure data</td><td>0.78</td></tr></tbody></table>

It also becomes crucial to accelerate the BUILD, DEPLOY phases. The use of a set of best practices and tools proposed by the MLOps concept seems to be a promising starting point (Kreuzberger, Kühl, and Hirschl 2023).

Previous results validate concept drift detection and the contribution of online re-learning. However, this concept drift is detected with a delay due to feedback from the field. Following table 3 results validate the detection of data drift, which provides a more immediate warning of a change in stationarity.
<table><caption>**Table 3.** Input data drift detection</caption><thead><tr><th>Online simulated X(t) data</th><th>number of 'drifted' features at t</th><th>number of 'drifted' features at t+5</th><th>number of 'drifted' features at t+10</th><th>feature impacted</th></tr></thead><tbody><tr><td>Online simulated X(t) none failure</td><td>0</td><td>0</td><td>0</td><td>none</td></tr><tr><td>Online simulated X(t) new Blocking failure</td><td>0</td><td>1</td><td>1</td><td>qMean</td></tr><tr><td>Online simulated X(t) new Leak failure</td><td>1</td><td>5</td><td>10</td><td>qMean</td></tr><tr><td>Online simulated X(t) new Bearing failure</td><td>2</td><td>10</td><td>23</td><td>pMid / pHigh / qMean</td></tr></tbody></table>

For this data drift detection test the scenario 1 is performed, during the BUILD phase in offline, only none failure and blocking failure data are trained to compute the ML model. Then in online new data are simulated. First, new none failure then new blocking failure data are simulated. No data drift is detected in accordance with learnt input space in BUILD phase. Secondly, the apparition of a leak failure then a bearing failure is simulated. Data drift detection is confirmed in both cases. Note that bearing failure detection is more sensitive and more features are impacted.

As described in section 4.2.3, a KS test is done for each feature. Reference used is the X\_train values use to learn the ML model. Comparison is made on each new value representing a new instant t.

For severity of data drift, the cumulative sum of detection > 1 prevents false alarms generation (for example from a previously learnt failure: blocking failure).

Another advantage is that failure data is not always available. This 'drift detection on input data' module allows the deployment of a ML model without waiting for specific failure cases to be available. It enables an initial fault detection by learning the normal operation of the equipment in the initial ML model.

## 4.4. Threats to validity and limitations

A strength of our experimentation is the use of simulation to overcome the obstacle of time and the lack of available data on long-term wear phenomena. However, several factors present some potential threat to the validity of the study. As accurate as simulations can be, they often simplify complex systems. Sample data from simulations may not be representative of real equipment. The number of simulations conducted and the degradation progression to reproduce the wear case may not faithfully reflect actual wear behavior. Access to heterogeneous data sources is simplified in our experiment. It is necessary to develop standardized interfaces to integrate various types of sensors and systems, along with the implementation of a data abstraction layer to harmonize formats and protocols. In addition, physical measurement sensors may be limited by the compatibility of communication protocols (TCP/IP, OPC UA, MQTT, etc.). Data quality can also be affected by losses due to problems with computer networks, electromagnetic compatibility (EMC), etc. To address network communication problems, it is essential to establish data storage and synchronization mechanisms to manage network outages. For data quality and sensor malfunctions, it is necessary to implement algorithms to detect and correct data anomalies, including data cleaning, imputation, and outlier detection techniques to handle noise and missing values in operational data. Redundancy mechanisms must be employed, along with cross-validation of sensors to ensure reliability. Additionally, calibration procedures and preventive maintenance of measurement equipment should be introduced
to maintain accuracy over time. An imbalanced dataset can affect the performance of ML models and drift detection methods. It might be beneficial to further tailor the drift management strategy to address this specific issue in more detail. In this paper, our framework is focused on supervised or semi-supervised ML models. The impact for unsupervised ML models and also concept drift handling with limited label availability are not addressed. There are certain limits to the dynamic interaction between processes. This pertains specifically to the sequencing of information flow exchanges between modules. Additionally, it concerns the method used to validate the contract interface between each sub-module.

Despite these potential threat to the validity and limitations, the completeness of your framework has been demonstrated.

## 5. Conclusions and Future Works

This paper presents a framework aimed at encompassing the end-to-end ML system lifecycle in an industrial context, dedicated to predictive maintenance of critical equipment. Our focus has been on the maintenance aspect of ML models, particularly on the detection of drift associated with the non-stationary environmental context in which the equipment operates. To achieve this objective, a drift detection mechanism has been introduced at both the input and the output stages of ML model used in production, with unsupervised data drift detection on one side and concept drift detection incorporating human ground truth on the other, along with a mechanism for retraining and competing with the newly updated models. A new metric for quantifying drift severity has been introduced to guide this decision-making process. Furthermore, the retraining mechanism incorporates domain knowledge online, gathered from human feedback, also known as ground truth. A human-in-the-loop control points decision has also been introduced to account for the specificities of industries equipped with safety-critical systems, such as the nuclear industry. The framework is applicable to other domains than PdM, and it would be interesting to test it on other use cases and adapt the modules accordingly. If the new use case remains with supervised learning, attention should be paid to the mechanism that guarantees feedback from the field. In the case where the data are predicted for the next n-step values with a regression method, this feedback may simply be the actual sensed values. In the case of unsupervised learning, since human feedback or ground truth does not exist, drift detection needs to be adapted by focusing only on the data drift in the ML model input.

This framework as a whole provides a good structural mindset for decomposing the problem into sub-elements, and the existing dependencies within these sub-elements have been demonstrated.

This work highlights some of the challenges to be faced in the future. The first one consists in understanding changes in the input space over time to manage stability and to bring explainability to the ML model evolution. This will give the maintainer answers about the evolution of the equipment. Approaches such as incremental learning for the 'new model learning' module is a path to explore (taking into account the stability-plasticity dilemma (Gama et al. 2014)). A comparison with the full retraining approach should also be conducted to better understand its limitations. Indeed, the experiment began by simulating failures using a physical model. It would be interesting to explore whether the evolution of drift detection over time can explain these variations and identify the underlying physical phenomenon. The second challenge will focus on the DEPLOY module and explore whether current MLOps practices are suit-
able for deploying and managing models close to the equipment, i.e. in the context of
Edge computing. The third challenge concerns the management over time of configura-
tions and versions of ML models from online retraining processes (MAINTAIN phase)
or the development of a new design (BUILD phase). This also involves incorporating
online elements that facilitate human decision-making, such as specific indicators that
improve the analysis and understanding of changes over time. The fourth challenge
focus on the impact of the feedback loop on the BUILD phase. More precisely, when
arriving at this phase after a severe drift, the data scientist have access to a lot of
knowledge on the ML models (the previous iterations) and the errors of these ML
models (drift). As a consequence, how can this phase be improved with all this knowl-
edge ? The fifth challenge investigates the optimal balance between automation and
human oversight. Studies on the effects of imperfect labeling by maintenance oper-
ators are underway, exploring various degrees of accuracy to refine the framework's
resilience against human-induced variability. Finally, the last challenge is to progres-
sively elevate the maturity of our framework, with different pilot projects that broaden
the diversity of use cases, with the aim of achieving TRL 7. This level corresponds to
the demonstration of a system prototype in an operational environment, marking a
significant leap in the practical applicability of our research.

Acknowledgment

For the purpose of Open Access, a CC-BY public copyright license has been applied
by the authors to the present document and will be applied to all subsequent versions
up to the Author Accepted Manuscript arising from this submission.
## References

Achouch, Mounia, Mariya Dimitrova, Khaled Ziane, Sasan Sattarpanah Karganroudi, Rizck Dhouib, Hussein Ibrahim, and Mehdi Adda. 2022. "On predictive maintenance in industry 4.0: Overview, models, and challenges." *Applied Sciences* 12 (16): 8081.

Aguilar Melgar, Leonel, David Dao, Shaoduo Gan, Nezihe M Gürel, Nora Hollenstein, Jiawei Jiang, Bojan Karlaš, et al. 2021. "Ease. ml: a lifecycle management system for machine learning." In *Proceedings of the Annual Conference on Innovative Data Systems Research (CIDR), 2021*, CIDR.

Amershi, Saleema, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. "Software engineering for machine learning: A case study." In *2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)*, 291–300. IEEE.

Bachinger, Florian, Gabriel Kronberger, and Michael Affenzeller. 2021. "Continuous improvement and adaptation of predictive models in smart manufacturing and model management." *IET Collaborative Intelligent Manufacturing* 3 (1): 48–63.

Baier, Lucas, Fabian Jöhren, and Stefan Seebacher. 2019. "Challenges in the Deployment and Operation of Machine Learning in Practice." In *ECIS*, Vol. 1.

Bayram, Firas, Bestoun S Ahmed, and Andreas Kassler. 2022. "From concept drift to model degradation: An overview on performance-aware drift detectors." *Knowledge-Based Systems* 245: 108632.

Bhattacharya, Pratyusha. 2022. "Concept Drift Detection and adaptation for machine learning." Master's thesis.

Breck, Eric, Shanqing Cai, Eric Nielsen, Michael Salib, and D Sculley. 2017. "The ML test score: A rubric for ML production readiness and technical debt reduction." In *2017 IEEE International Conference on Big Data (Big Data)*, 1123–1132. IEEE.

Carvalho, Thyago P., Fabrízzio A.A.M.N. Soares, Roberto Vita, Roberto da P. Francisco, João P. Basto, and Symone G.S. Alcalá. 2019. "A systematic literature review of machine learning methods applied to predictive maintenance." *Computers and Industrial Engineering* 137.

Coandă, P., M Avram, and V Constantin. 2020. "A state of the art of predictive maintenance techniques." In *IOP Conference Series: Materials Science and Engineering*, Vol. 997, 012039. IOP Publishing.

Crankshaw, Daniel, Peter Bailis, Joseph E Gonzalez, Haoyuan Li, Zhao Zhang, Michael J Franklin, Ali Ghodsi, and Michael I Jordan. 2014. "The missing piece in complex analytics: Low latency, scalable model management and serving with velox." *arXiv preprint arXiv:1409.3809*.

Crankshaw, Daniel, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. 2017. "Clipper: A {Low-Latency} online prediction serving system." In *14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17)*, 613–627.

Davari, Narjes, Bruno Veloso, Gustavo de Assis Costa, Pedro Mota Pereira, Rita P Ribeiro, and João Gama. 2021. "A survey on data-driven predictive maintenance for the railway industry." *Sensors* 21 (17): 5739.

Ditzler, Gregory, Manuel Roveri, Cesare Alippi, and Robi Polikar. 2015. "Learning in nonstationary environments: A survey." *IEEE Computational Intelligence Magazine* 10 (4): 12–25.

Dreyfus, Paul-Arthur, Antoine Pélissier, Foivos Psarommatis, and Dimitris Kiritsis. 2022. "Data-based model maintenance in the era of industry 4.0: A methodology." *Journal of Manufacturing Systems* 63: 304–316. https://doi.org/10.1016/j.jmsy.2022.03.015.

Elwell, Ryan, and Robi Polikar. 2011. "Incremental learning of concept drift in nonstationary environments." *IEEE transactions on neural networks* 22 (10): 1517–1531.

Faria, Elaine R, Isabel JCR Gonçalves, André CPLF de Carvalho, and João Gama. 2016. "Novelty detection in data streams." *Artificial Intelligence Review* 45: 235–269.

Fayyad, Usama, Gregory Piatetsky-Shapiro, and Padhraic Smyth. 1996. "The KDD process
for extracting useful knowledge from volumes of data. Commun.” *ACM* 39 (11): 27–34.

Fernandes, Marta, Juan Manuel Corchado, and Goreti Marreiros. 2022. “Machine learning techniques applied to mechanical fault diagnosis and fault prognosis in the context of real industrial manufacturing use-cases: a systematic literature review.” *Applied Intelligence* 52 (12): 14246–14280.

Fischer, Lukas, Lisa Ehrlinger, Verena Geist, Rudolf Ramler, Florian Sobieczky, Werner Zellinger, and Bernhard Moser. 2020. “Applying AI in practice: key challenges and lessons learned.” In *International Cross-Domain Conference for Machine Learning and Knowledge Extraction*, 451–471. Springer.

Fujii, Tiago Yukio, Victor Takashi Hayashi, Reginaldo Arakaki, Wilson Vicente Ruggiero, Romeo Bulla Jr, Fabio Hirotsugu Hayashi, and Khalil Ahmad Khalil. 2021. “A digital twin architecture model applied with MLOps techniques to improve short-term energy consumption prediction.” *Machines* 10 (1): 23.

Gama, João, Indrê Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. “A survey on concept drift adaptation.” *ACM computing surveys (CSUR)* 46 (4): 1–37.

Habibullah, Khan Mohammad, Gregory Gay, and Jennifer Horkoff. 2023. “Non-functional requirements for machine learning: Understanding current use and challenges among practitioners.” *Requirements Engineering* 28 (2): 283–316.

Hanzelik, Pal Peter, Alex Kummer, and Janos Abonyi. 2022. “Edge-computing and machine-learning-based framework for software sensor development.” *Sensors* 22 (11): 4268.

Hewage, Nipuni, and Dulani Meedeniya. 2022. “Machine learning operations: A survey on MLOps tool support.” *arXiv preprint arXiv:2202.10169* .

Hurtado, Julio, Dario Salvati, Rudy Semola, Mattia Bosio, and Vincenzo Lomonaco. 2023. “Continual learning for predictive maintenance: Overview and challenges.” *Intelligent Systems with Applications* 20251.

Jackson, Stuart, Maha Yaqub, and Cheng-Xi Li. 2019. “The agile deployment of machine learning models in healthcare.” *Frontiers in big Data* 1: 7.

John, Meenu Mary, Helena Holmström Olsson, and Jan Bosch. 2021. “Towards mlops: A framework and maturity model.” In *2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)*, 1–8. IEEE.

Kermenov, Renat, Giacomo Nabissi, Sauro Longhi, and Andrea Bonci. 2023. “Anomaly detection and concept drift adaptation for dynamic systems: a general method with practical implementation using an industrial collaborative robot.” *Sensors* 23 (6): 3260.

Khamassi, Imen, Moamar Sayed-Mouchaweh, Moez Hammami, and Khaled Ghédira. 2018. “Discussion and review on evolving data streams and concept drift adapting.” *Evolving systems* 9: 1–23.

Kreuzberger, Dominik, Niklas Kühl, and Sebastian Hirschl. 2023. “Machine learning operations (mlops): Overview, definition, and architecture.” *IEEE access* .

Lavin, Alexander, Ciarán M Gilligan-Lee, Alessya Visnjic, Siddha Ganju, Dava Newman, Su-
joy Ganguly, Danny Lange, et al. 2022. “Technology readiness levels for machine learning
systems.” *Nature Communications* 13 (1): 6039.

Le Nguyen, Minh Huong. 2023. “Online machine learning-based predictive maintenance for the railway industry.” PhD diss., Institut Polytechnique de Paris.

Lee, Yunseong, Alberto Scolari, Byung-Gon Chun, Marco Domenico Santambrogio, Markus Weimer, and Matteo Interlandi. 2018. “{PRETZEL}: Opening the black box of machine learning prediction serving systems.” In *13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)*, 611–626.

Lu, Jie, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. 2018. “Learning under concept drift: A review.” *IEEE transactions on knowledge and data engineering* 31 (12): 2346–2363.

Mäkinen, Sasu, Henrik Skogström, Eero Laaksonen, and Tommi Mikkonen. 2021. “Who needs MLOps: What data scientists seek to accomplish and how can MLOps help?” In *2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN)*, 109–
112. IEEE.

Marcus, Gary. 2018. "Deep learning: A critical appraisal." *arXiv preprint arXiv:1801.00631* .

Moreno-Torres, Jose G, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. 2012. "A unifying view on dataset shift in classification." *Pattern recognition* 45 (1): 521–530.

Mouchaweh, M Sayed. 2014. “Apprentissage dynamique dans un environnement non-stationnaire.” *Editions Techniques de l’ingénieur* .

Na, He, Jiang Zhong, Wang Jian, Sun Minghe, and Xie Guanghe. 2021. “Maintenance optimisation and coordination with fairness concerns for the service-oriented manufacturing supply chain.” *Enterprise Information Systems* 15 (5): 694–724.

Paleyes, Andrei, Raoul-Gabriel Urma, and Neil D Lawrence. 2022. “Challenges in deploying machine learning: a survey of case studies.” *ACM computing surveys* 55 (6): 1–29.

Palli, Abdul Sattar, Jafreezal Jaafar, Heitor Murilo Gomes, Manzoor Ahmed Hashmani, and Abdul Rehman Gilal. 2022. “An Experimental Analysis of Drift Detection Methods on Multi-Class Imbalanced Data Streams.” *Applied Sciences* 12 (22): 11688.

Plotnikova, Veronika, Marlon Dumas, and Fredrik Milani. 2020. “Adaptations of data mining methodologies: a systematic literature review.” *PeerJ Computer Science* 6: e267.

Rangachari, Sundarrajan, and Philipos C Loizou. 2006. “A noise-estimation algorithm for highly non-stationary environments.” *Speech communication* 48 (2): 220–231.

Riccio, Vincenzo, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. “Testing machine learning based systems: a systematic mapping.” *Empirical Software Engineering* 25 (6): 5193–5254. https://doi.org/10.1007/s10664-020-09881-0.

Ruf, Philipp, Manav Madan, Christoph Reich, and Djaffar Ould-Abdeslam. 2021. “Demystifying mlops and presenting a recipe for the selection of open-source tools.” *Applied Sciences* 11 (19): 8861.

Saley, Amaratou Mahamadou, Jérémie Marchand, Aicha Sekhari, Vincent Cheutet, and Jean-Baptiste Danielou. 2022. “State-of-art and maturity overview of the nuclear industry on predictive maintenance.” In *IFIP International Conference on Product Lifecycle Management*, 337–346. Springer.

Santhanam, Peter. 2020. “Quality management of machine learning systems.” In *Engineering Dependable and Secure Machine Learning Systems: Third International Workshop, EDSMLS 2020, New York City, NY, USA, February 7, 2020, Revised Selected Papers 3*, 1–13. Springer.

Sayed-Mouchaweh, Moamar, and Edwin Lughofer. 2012. *Learning in non-stationary environments: methods and applications*. Springer Science & Business Media.

Sculley, David, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. “Hidden technical debt in machine learning systems.” *Advances in neural information processing systems* 28.

Sethi, Tegjyot Singh, and Mehmed Kantardzic. 2017. “On the reliable detection of concept drift from streaming unlabeled data.” *Expert Systems with Applications* 82: 77–99. https://doi.org/https://doi.org/10.1016/j.eswa.2017.04.008.

Settles, Burr. 2009. “Active learning literature survey.” .

Spjuth, Ola, Jens Frid, and Andreas Hellander. 2021. “The machine learning life cycle and the cloud: implications for drug discovery.” *Expert Opinion on Drug Discovery* 16: 1071–1079. https://doi.org/10.1080/17460441.2021.1932812.

Studer, Stefan, Thanh Binh Bui, Christian Drescher, Alexander Hanuschkin, Ludwig Winkler, Steven Peters, and Klaus-Robert Müller. 2021. “Towards CRISP-ML (Q): a machine learning process model with quality assurance methodology.” *Machine learning and knowledge extraction* 3 (2): 392–413.

Sugiyama, Masashi, and Motoaki Kawanabe. 2012. *Machine Learning in Non-Stationary Environments Introduction to Covariate Shift Adaptation*. The MIT Press.

Symeonidis, Georgios, Evangelos Nerantzis, Apostolos Kazakis, and George A Papakostas.
2022. "MLOps-definitions, tools and challenges." In *2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC)*, 0453-0460. IEEE.

Tambon, Florian, Gabriel Laberge, Le An, Amin Nikanjam, Paulina Stevia Nouwou Mindom, Yann Pequignot, Foutse Khomh, Giulio Antoniol, Ettore Merlo, and François Laviolette. 2021. "How to certify machine learning based safety-critical systems? A systematic literature review." *Automated Software Engineering*.

Testi, Matteo, Matteo Ballabio, Emanuele Frontoni, Giulio Iannello, Sara Moccia, Paolo Soda, and Gennaro Vessio. 2022. "MLOps: A Taxonomy and a Methodology." *IEEE Access*.

Treveil, Mark, and The Dataiku Team. 2020. *Introducing MLOps How to Scale Machine Learning in the Enterprise*. O'Reilly.

Trojanowski, Krzysztof, and Zbigniew Michalewicz. 1999. "Evolutionary algorithms for non-stationary environments." In *Proc. of 8th workshop: Intelligent information systems*, 229-240. Citeseer.

Vela, Daniel, Andrew Sharp, Richard Zhang, Trang Nguyen, An Hoang, and Oleg S. Pianykh. 2022. "Temporal quality degradation in AI models." *Scientific Reports* 12. https://doi.org/10.1038/s41598-022-15245-z.

Webb, Geoffrey I., Roy Hyde, Hong Cao, Hai-Long Nguyen, and François Petitjean. 2015. "Characterizing concept drift." *Data Mining and Knowledge Discovery*.

Weber, Christian, Pascal Hirmer, Peter Reimann, and Holger Schwarz. 2019. "A New Process Model for the Comprehensive Management of Machine Learning Models." In *International Conference on Enterprise Information Systems*, .

Widmer, Gerhard, and Miroslav Kubat. 1996. "Learning in the presence of concept drift and hidden contexts." *Machine learning*.

Wirth, Rüdiger, and Jochen Hipp. 2000. "CRISP-DM: Towards a standard process model for data mining." In *Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining*, Vol. 1, 29-39. Manchester.

Wuest, Thorsten, Daniel Weimer, Christopher Irgens, Klaus-dieter Thoben, Thorsten Wuest, Daniel Weimer, and Christopher Irgens. 2016. "Machine learning in manufacturing: advantages, challenges, and applications." *Production & Manufacturing Research* 3277: 1-23. https://doi.org/10.1080/21693277.2016.1192517.

Zenisek, Jan, Florian Holzinger, and Michael Affenzeller. 2019. "Machine learning based concept drift detection for predictive maintenance." *Computers and Industrial Engineering* 137. https://doi.org/10.1016/j.cie.2019.106031.

Zhang, Jie M, Mark Harman, Lei Ma, and Yang Liu. 2020. "Machine Learning Testing: Survey, Landscapes and Horizons." *IEEE Transactions on Software Engineering* 1-37.

Zhang, Weiting, Dong Yang, and Hongchao Wang. 2019. "Data-Driven Methods for Predictive Maintenance of Industrial Equipment: A Survey." *IEEE Systems Journal* 13 (3): 2213-2227. https://doi.org/10.1109/JSYST.2019.2905565.

Zhong, Dong, Zhelei Xia, Yian Zhu, and Junhua Duan. 2023. "Overview of predictive maintenance based on digital twin technology." *Heliyon*.

Zhou, Han, Hongpeng Yin, Dandan Zhao, and Li Cai. 2023. "Incremental Learning and Conditional Drift Adaptation for Nonstationary Industrial." *IEEE Transactions on Industrial Informatics* 19. https://doi.org/10.1109/TII.2022.3179423.

Žliobaitė, Indrė. 2010. "Learning under concept drift: an overview." *arXiv preprint arXiv:1010.4784*.