# Multi-agent deep reinforcement learning-based maintenance optimization for multi-dependent component systems

Phuc Do, Van-Thai Nguyen, Alexandre Voisin, Benoît Iung, Waldomiro Alves

Ferreira Neto

► To cite this version:

Phuc Do, Van-Thai Nguyen, Alexandre Voisin, Benoît Iung, Waldomiro Alves Ferreira Neto. Multi-agent deep reinforcement learning-based maintenance optimization for multi-dependent component systems. Expert Systems with Applications, 2024, 245, pp.123144. 10.1016/j.eswa.2024.123144. hal-04460407

HAL Id: hal-04460407

https://hal.science/hal-04460407v1

Submitted on 19 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# Multi-agent deep reinforcement learning-based maintenance optimization for multi-dependent component systems

Phuc Do<sup>a, *</sup>, Van-Thai Nguyen<sup>a</sup>, Alexandre Voisin<sup>a</sup>, Benoit Iung<sup>a</sup>, Waldomiro Alves Ferreira Neto<sup>b</sup>

## ARTICLE INFO

**Keywords:**

Deep reinforcement learning
Cooperative multi-agent systems
Maintenance decision-making
Multi-component systems

## ABSTRACT

Manufacturing systems consist of a set of interdependent components. However, addressing the dependence between these components remains a challenge in both maintenance modeling and the optimization process. In this paper, we propose a multi-agent deep reinforcement learning-based maintenance approach for a manufacturing system, taking into consideration both stochastic and economic dependencies between components. In this manner, we introduce a novel state interactions model, suggesting that the degradation state of one component may influence the degradation process of others. Subsequently, a maintenance planning approach based on multi-agent deep reinforcement learning is developed to optimize maintenance decisions in both fully and partially observed states. The deployed multi-agent deep reinforcement algorithm, specifically Weighted QMIX, ensures scalability and efficient consideration of state interactions and economic dependencies between components. The feasibility and performance of the proposed maintenance approach are investigated through various numerical studies. When compared to traditional maintenance approaches, such as value iteration method, Dueling Double Deep Q Network, and Multi-Agent Deep Q Network, our proposed approach consistently demonstrates superior results.

## 1. Introduction

Maintenance encompasses technical and managerial tasks aimed at sustaining a machine (component or system) in, or restoring it to, an operational state that aligns with its designed functions (Rau-sand & Hoyland, 2003). Maintenance policies are typically categorized into two main types: corrective maintenance (CM) and preventive maintenance (PM). CM, also known as breakdown or run-to-failure maintenance, involves repairing malfunctioned machines and is often associated with high costs due to unexpected production losses (Ahmad & Kamaruddin, 2012; Gorenstein & Kalech, 2022; Tsang, 1995). In contrast, PM interventions are conducted on functioning machines to forestall potential failures (Huang et al., 2020; Ruan et al., 2021).

PM actions can be scheduled based on either the age or state (degradation level) of machines. The latter, also known as condition-based maintenance (CBM), offers advantages over the former. Its flexibility allows for maintenance decisions based on the current health condition of maintained machines (Guillén et al., 2016). Advances in sensor technology, enabling the collection of rich degradation measurement information, have made CBM a popular approach for maintenance

decision-making and optimization (Ayvaz & Alpay, 2021; Dalzochio et al., 2020; Roda & Macchi, 2021; Zonta et al., 2020).

While CBM scheduling has been extensively studied for single-component systems, many proposed maintenance policies may be applicable for multi-component systems if the dependence between components is neglected. However, such an assumption seems irrelevant for many manufacturing systems, where components are usually interdependent, potentially leading to sub-optimal policies (Alaswad & Xiang, 2017). Thus, it is essential to consider component dependencies in the maintenance modeling and optimization process. These dependencies fall into three broad groups (Dekker et al., 1997; Dong et al., 2019; Nicolai & Dekker, 2008; Shi & Zeng, 2016; Thomas, 1986): (i) economic dependence, (ii) structural dependence, and (iii) stochastic dependence. Economic dependence implies that the joint maintenance cost of several components can be either cheaper (positive dependence) or more expensive (negative dependence) than maintaining them separately (Do et al., 2019). Structural dependence exists when maintenance of one component requires at least the dismantling or maintenance of other components (Dinh et al., 2020).

<footnote><sup>a</sup> Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France</footnote>

<footnote><sup>b</sup> RANDOM, UFPE, Brazil</footnote>

<footnote>* Corresponding author.</footnote>
Stochastic dependence arises when component failures/degradation processes have interactions, meaning the failure/degradation process of a component influences the lifetime distribution of other functioning components. Recently, several CBM approaches have been developed to consider the interactions between components (Keizer et al., 2017). Positive economic dependence is widely considered among component dependencies to take advantage of opportunistic maintenance (Keizer et al., 2017). Efficient opportunistic maintenance is achieved through thresholds-based CBM policies (Castanier et al., 2005; Huynh et al., 2013; Zhang & Zeng, 2015). The idea is to perform preventive maintenance on a component as soon as its degradation approaches a prefixed threshold, namely the preventive threshold. Other maintenance thresholds, called opportunistic thresholds, which are often lower than the preventive threshold, are introduced for opportunistic maintenance of each component. Indeed, a component not selected for preventive maintenance can also be opportunistically maintained together with preventive maintenance of other components if its deterioration level reaches an opportunistic threshold. In this way, the number of maintenance thresholds, which need to be optimized, increases significantly with the number of components, causing heavy computation for the optimization process or potentially leading to sub-optimal solutions. As pointed out in Do et al. (2019), an adaptive opportunistic CBM policy designed for a two-component system considering only replacement actions requires four decision variables: two preventive thresholds and two opportunistic thresholds. Therefore, finding an efficient opportunistic CBM approach for a multi-dependent component system remains an open issue.

To address this challenge, in this work, we develop a multi-agent deep reinforcement learning-based CBM approach for a manufacturing system considering both stochastic and economic dependence between components. In this way, a novel state interactions model is proposed, implying that the degradation state of a component might affect the degradation process of other components. A maintenance planning approach based on multi-agent deep reinforcement learning is then developed to optimize maintenance decisions in both fully and partially observed states. Thanks to the deployed multi-agent deep reinforcement algorithm, namely Weighted QMIX, the proposed maintenance approach is scalable and allows efficient consideration of the state interactions and economic dependence between components. The feasibility and performance of the proposed maintenance approach are investigated through various numerical studies presented in Section 6.

In the next section, related works on the existing stochastic dependence models and deep reinforcement-based maintenance approaches are discussed to identify the related research gaps and highlight the scientific contributions of this study.

## 2. Related works and scientific contributions

This section analyzes recent studies by focusing on two main challenges: (1) stochastic dependence model (2) deep reinforcement-based maintenance optimization in presence of dependence between components.

### 2.1. Stochastic dependence: modeling and integration

According to Keizer et al. (2017), Wang et al. (2022), stochastic dependence can be classified into two main categories: failure-based stochastic dependence and degradation-based stochastic dependence. The majority of studies in the literature have focused on failure-based stochastic dependence, including failure-induced damage, failure-based load sharing, and common-mode failure (Keizer et al., 2017). Another aspect, degradation-based stochastic dependence, suggests that the degradation state of specific components may influence the degradation process of others. This type of dependence is also referred to as degradation interactions (Bian & Gebraeel, 2014; Rasmekomen & Parlikad, 2016).

In particular, stochastic dependence through state interactions, where the increment in the degradation level of a component is a function of its own state and the states of other components, has been studied for an industrial cool box (Rasmekomen & Parlikad, 2014). Subsequently, the research was extended to characterize stochastic dependence through state-rate interaction (Rasmekomen & Parlikad, 2016), indicating that the degradation levels of a component accelerate the degradation process of its dependent components. Similar degradation interactions were also investigated for a networked system using continuous stochastic differential equations and for a gearbox system with component degradation processes modeled by gamma distributions (Do et al., 2019). A recent state interactions model proposed in Wang et al. (2022) is designed for general multi-component systems using Wiener processes.

It is noteworthy that the degradation-based stochastic dependence models developed in the aforementioned works are tailored for components with continuous states, requiring continuous monitoring and incurring high maintenance costs (Keizer et al., 2017; Zhou et al., 2013). Additionally, implementing an online monitoring system for every machinery component in a large, complex engineering system (e.g., a water supply system) would be prohibitively expensive (Zhou et al., 2013). Moreover, practical applications often involve degradation processes modeled by discrete-state models in discrete time, as highlighted in Nguyen et al. (2022). Such models offer a more synthetic view of the system states and simplify the maintenance decision-making process, aiding decision-makers (Nguyen et al., 2019). Thus, there is still an opportunity to model degradation interactions in manufacturing systems consisting of multi-state components that degrade discretely over time.

### 2.2. Deep reinforcement learning for maintenance optimization

A way to avoid the heuristics of opportunistic maintenance is to formulate the maintenance decision-making process as a Markov decision process (MDP) or other variants. However, finding an optimal solution for a large MDP is often challenging due to the curse of dimensionality. In the context of maintenance planning, this implies that the space of degradation states and maintenance actions grows exponentially with the number of components, causing heavy computation for optimization algorithms. Fortunately, recent advancements in the field of deep reinforcement learning (DRL) provide new tools to address the sequential maintenance decision-making problem for large systems (Skordilis & Moghaddass, 2020; Zhang et al., 2021).

In particular, the double deep Q-network (DDQN) algorithm is employed in Zhang and Si (2020) to optimize preventive maintenance for a 12-component system, considering economic dependence between components and competing failure risks. This assumes that several components are physically integrated and must be maintained simultaneously to reduce the complexity of the maintenance decision space. The DDQN algorithm is also used in Huang et al. (2020) to optimize the preventive maintenance policy of a 6-machine-5-buffer serial production line. Although the state space of this maintenance decision-making problem is extremely large, the number of actions is quite small. In Rocchetta et al. (2019), separate neural networks are employed for estimating each action's value following a Q-learning fashion for maintenance operation optimization of power grid systems, showing possibilities of reaching a global solution. However, this method is only feasible for decision-making applications with a relatively small action space size. Despite the effectiveness of single-agent DRL algorithms in optimizing long-term policies for decision-making problems with large state spaces, as shown in the above-mentioned works, it is pointed out in the literature that these algorithms are not suitable for maintenance scheduling applications with large action spaces.

The paradigm of Multi-Agent Deep Reinforcement Learning (MADRL) holds promise for effectively addressing challenges inherent
in the optimization of maintenance planning—a subject that has gar-
nered significant scholarly attention. Within MADRL's framework for
decision-making in multi-agent systems, two discernible approaches,
namely centralized and decentralized learning paradigms, shape the or-
chestration of actions. The decentralized paradigm within MADRL con-
fers autonomy upon individual agents or subsystems, allowing for inde-
pendent decisions based on local information and eliminating the need
for system-level access. Addressing the imperative of decentralized
decision-making, Andriotis and Papakonstantinou (2021) puts forth a
MADRL-based decentralized maintenance policy. This paradigm grants
components or subsystems autonomy, facilitating maintenance deci-
sions based on local information while acknowledging resource limi-
tations and risk-related constraints. A decentralized MADRL approach
was implemented in Su et al. (2022) to investigate individual preven-
tive maintenance policies. Optimized through a value-decomposition
actor-critic algorithm, this exploration underscores the inherent adapt-
ability of the MADRL framework. The approach, which facilitates
independent decision-making at the agent level, accentuates the pivotal
importance of tailoring optimization strategies to suit the distinc-
tive characteristics of individual manufacturing components. While
the decentralized paradigm offers scalability advantages, it requires
skillful management of individual agent behaviors, potentially posing
challenges in the pursuit of finding the optimal global solution.

In contrast, In the centralized approach, decision-making is har-
monized through a unified system, fostering collaborative information
sharing among agents. An illustrative example of this is seen in the
optimization of maintenance actions for expansive structures through
the application of a deep centralized multi-agent actor-critic algorithm
in Andriotis and Papakonstantinou (2019). This centralized strategy
embodies coordinated decision-making among agents, resulting in an
overall enhancement of system performance. Conversely, Nguyen et al.
(2022) introduces a centralized adaptation of the value decomposition
network algorithm, emphasizing optimal maintenance decisions in the
context of imperfect maintenance actions and the absence of individual
cost data—parameters typically required for computing maintenance
costs at the system level. While the centralized paradigm nature guar-
antees a cohesive decision-making process, it is crucial to note that
scalability challenges may surface as the number of agents increases.
Hence, a critical imperative lies in scrutinizing the scalability issues
associated with the centralized approach. Furthermore, a pivotal facet
of MADRL lies in its adaptability to divergent levels of environmental
observability. Specifically, MADRL is adept at optimizing maintenance
policies under two conditions: when information at the system level
is fully available and when it is only partially observable (fully and
partially observability).

The latter scenario is suitable for industrial systems where collecting
system-wide data may be inefficient, allowing the choice of mainte-
nance actions for each machine based on local data. However, crafting
an effective maintenance policy in partially observable environments
is generally more intricate due to component dependencies and agents'
limited observability. This complexity can potentially lead to reduced
performance of optimization algorithms. Therefore, a primary chal-
lenge in MADRL is devising an efficient multiagent training scheme that
addresses nonstationarity and partial observability while managing the
computational complexity inherent in dealing with large multicompo-
nent systems (Wong et al., 2023). However, partial information has not
been widely addressed in the application of MADRL for maintenance
optimization. Recently, Rokhforoz et al. (2023) considered incomplete
information for MADRL in joint bidding and maintenance scheduling
of generation units. However, maintenance is only treated as a critical
constraint without receiving primary focus on its optimization.

Furthermore, a pivotal facet of MADRL lies in its adaptability to
divergent levels of environmental observability. Specifically, MADRL
is adept at optimizing maintenance policies under two conditions:
when information at the system level is fully available and when it is
only partially observable (fully and partially observability). The latter

scenario is suitable for industrial systems where collecting system-wide
data may be inefficient, allowing the choice of maintenance actions
for each machine based on local data. However, crafting an effective
maintenance policy in partially observable environments is generally
more intricate due to component dependencies and agents' limited
observability. This complexity can potentially lead to reduced per-
formance of optimization algorithms. Therefore, a primary challenge
in MADRL is devising an efficient multiagent training scheme that
addresses nonstationarity and partial observability while managing the
computational complexity inherent in dealing with large multicompo-
nent systems (Wong et al., 2023). However, partially information has
not been widely addressed when applying MADRL for maintenance op-
timization. Recently, Rokhforoz et al. (2023) has considered incomplete
information for a MADRL for joint bidding and maintenance scheduling
of generation units, but the maintenance is only considered as critical
constraint and do not given primary focus on it optimization.

Therefore, it requires a comprehensive exploration into the perfor-
mance of optimization algorithms in both fully and partially observable
settings. Additionally, an in-depth analysis of the influence of compo-
nent dependencies on the effectiveness of optimized maintenance poli-
cies is essential, especially in systems characterized by interdependent
components.

2.3. *Scientific contributions*

In order to consider the issues analyzed above, in this paper, a
MADRL-based maintenance approach is proposed for systems composed
of a set of multi-state components that are subjected to both state
interactions and economic dependence. The main contributions in this
study are listed as follows:

* Proposing a novel state interactions model describing the state-rate degradation interaction between components. The proposed model allows considering the state dependence between multi-state components. When compared to conventional stochastic dependence models, the proposed model is unique and existing failure-based stochastic models are a special case of the proposed model;

* Developing maintenance planning problems for the studied systems as cooperative multi-agent decision-making tasks under both fully and partially observable contexts to investigate the impact of partial observability on the quality of optimized maintenance policies. The developed maintenance concept is especially suitable in the context of industry 4.0 for which manufacturing systems composed by a set of cyber physical components (CPSs). Thanks to the cyber capacity with embedded reinforcement learning agent, CPS might be able to learn and provide “optimal” maintenance decision-making by collaborating others CPSs;

* Implementing the WQMIX multi-agent reinforcement learning algorithm (Rashid et al., 2020) to optimize maintenance decisions in a partially observable setting. In the fully observable setting, a customized version of WQMIX is proposed to take advantage of system-level information, allowing for reduced training time and better performance;

* Conducting numerical studies on a small 5-component system and a “large” 11-component system to investigate the impact of component dependencies on optimized maintenance policies, as well as to examine the stability of the WQMIX algorithm.

The remainder of this paper is structured as follows. Section 3 describes the studied system and assumptions. Maintenance operations carried out on the studied system and the economic dependence between components are also presented in this section. Section 4 is devoted to presenting the proposed state interactions model. The MADRL-based maintenance approach is developed in Section 5. In the next section, the numerical results conducted on two different systems are depicted and analyzed. Finally, the last section concludes the paper and gives some perspectives.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Illustration of a series-parallel system consisting of M subsystems, where subsystem j is composed of H^j components of the same type (j = 1, 2, ..., M).</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2. Illustration of state transitions of component i.</figcaption></figure>

## 3. System description and problem formulation

### 3.1. System description and assumptions

We examine a series-parallel system comprising N components grouped into M subsystems, as depicted in Fig. 1. Subsystem j consists of H^j components of the same type j, leading to $N = \sum_{j=1}^{M} H^j$.

At periodic inspection times $t_k$, component i can be observed in any discrete health condition state, $s_k^i \in \{0, ..., m^i\}$, ranging from a good state to a failed state. In the absence of maintenance intervention and degradation interaction between components, the state transition of component i between consecutive inspections follows its inherent Markov law. Fig. 2 illustrates the state evolution of component i.

$\tilde{p}_{uv}^i$ is a non-negative real number representing the intrinsic degradation transition probability from state u to v of component i, satisfying $\sum_{i=u}^{m^i} \tilde{p}_{uv}^i = 1$ for all $u \in \{0, 1, ..., m^i\}$. Thus, the transition matrix for component i is given by:

$$ \tilde{P}^i = \begin{bmatrix} \tilde{p}_{00}^i & \tilde{p}_{01}^i & \tilde{p}_{02}^i & \cdots & \tilde{p}_{0m^i}^i \\ 0 & \tilde{p}_{11}^i & \tilde{p}_{12}^i & \cdots & \tilde{p}_{1m^i}^i \\ 0 & 0 & \tilde{p}_{22}^i & \cdots & \tilde{p}_{2m^i}^i \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \cdots & 1 \end{bmatrix} \quad (1) $$

Due to the system configuration, components in the same subsystem may be stochastically dependent (Keizer et al., 2017; Wang et al., 2022). The interaction between components will be discussed in Section 3.

### 3.2. Maintenance operations and economic dependence analysis

Both corrective and preventive maintenance actions are considered for the studied system. Corrective maintenance involves replacing a

failed component with a new one of the same type, while preventive maintenance is performed on a functioning component to prevent future failures, and it can be either perfect or imperfect. A perfect preventive maintenance action completely restores a surviving component to be “as good as new”, while an imperfect one implies that the component’s state after maintenance is between its state before maintenance and the “as good as new” state. Maintenance interventions can only be discretely carried out after component health conditions are revealed by inspections. If a component fails between two successive inspections, corrective maintenance intervention can only be performed at the next inspection date.

#### 3.2.1. Maintenance actions

A maintenance action implemented at inspection time $t_k$ for component i is denoted as $a_k^i$. There are three possible maintenance actions for each component at any timestamp $t_k$, encoded as follows:

$$ a_k^i = \begin{cases} 0 & \text{leave component } i \text{ as it is} \\ 1 & \text{implement imperfect maintenance on component } i \\ 2 & \text{replace component } i \text{ with a new one of the same type} \end{cases} \quad (2) $$

Imperfect maintenance implies that a component can be repaired to a more healthy state, which is somewhere between the state before maintenance and “as good as new.” Imperfection in maintenance may result from factors such as the lack of skilled repairmen, poor spare parts quality, or limited repair duration. In this work, we use the imperfect maintenance model proposed in Do and Bérenguer (2012). Specifically, the state after maintenance of a component can be obtained by uniformly discretely sampling from the interval from the good state to its state before maintenance. The maintenance cost of a component i is calculated as follows:

$$ c_k^{m,i} = c^{r,i} \left( \frac{s_k^i - \bar{s}_k^i}{s_k^i} \right)^{\beta^i} \quad (3) $$
where $c^{i,j}$ is a constant representing the cost of replacing component $i$ with a new one; $s_k^i > 0$ and $s_k^i \ge \bar{s}_k^i \ge 0$ are respectively the state before and after maintenance of component $i$; $\beta^i$ is a positive real number representing the imperfect maintenance characteristics of component $i$. If component $i$ is in a good state ($s_k^i = 0$), it is unnecessary to maintain the component, i.e., the cost model in Eq. (3) is not needed. Moreover, due to $(s_k^i - \bar{s}_k^i)/s_k^i \le 1$, if $\beta^i < 1$, $c_k^{m,i}$ is a concave function, implying that the maintenance cost increases more than the improvement gain $(s_k^i - \bar{s}_k^i)$ when performing the maintenance. If $\beta^i = 1$, the preventive maintenance cost is proportional to the improvement gain; and if $\beta^i > 1$, $c_k^{m,i}$ is a convex function, indicating that the maintenance cost increases less than the improvement gain.

### 3.2.2. Cost structure and economic dependence analysis

The individual maintenance cost of a component consists of three elements: an inspection cost, a setup cost, and a component-specific maintenance cost. The cost of inspecting component $i$ is denoted as $c^{ins,i}$, which must be paid even for survival components to reveal their current health conditions. This paper considers two levels of setup costs (Wijnmalen & Hontelez, 1997): the system setup cost, $c^0$, caused by administrative handling or transportation of spare parts, and the component-type setup cost, $c^{t,i}$, originating from the requirement of repairman skills or specific tools. The component-specific maintenance cost is denoted as $c^{m,i}$, which depends on maintenance quality and the component's own characteristics. Based on the above analysis, the cost of separately maintaining a component $i$ is calculated as follows:

$$c^i = c^0 + c^{t,i} + c^{ins,i} + c^{m,i} \quad (4)$$

Maintenance cost for a group of components can be reduced via setup cost sharing mechanisms due to positive economic dependence between them (Dekker et al., 1997). For our studied system, if several maintenance actions are grouped, the system setup cost is paid only once. Similarly, the component-type setup costs are charged only once if some components of the same type are maintained together. The maintenance cost at the system level, denoted as $c$, is given by:

$$c = \sum_{i=1}^{N} c^i - \mathbb{I}^0(N^0 - 1)c^0 - \sum_{j=1}^{M} \mathbb{I}^{m,j}(H^{m,j} - 1)c^{t,j} \quad (5)$$

where:

* $N^0$ is the number of maintained components;

* $H^{m,j}$ is the number of maintained components of subsystem $j$;

* $\mathbb{I}^0$ is the system maintenance indicator, with a value equal to one if there is at least one component being maintained or equal to zero otherwise;

* $\mathbb{I}^{m,j}$ is the maintenance indicator of type $j$, with a value equal to one if there is at least one component of type $j$ being maintained or equal to zero otherwise.

In conjunction with the maintenance cost, the system downtime cost (e.g., production losses) caused by the failure of a component or a group of several components leading to a system shutdown should be taken into account in maintenance optimization processes.

## 4. State interactions modeling

In this study, we explore a scenario where the deterioration process of a component can be accelerated by the degraded states of its dependent components. Within the framework of Markov processes, this interaction can be modeled by decreasing probabilities of staying in the current states of the component while increasing the probabilities of transitioning to worse conditions. We propose the following model:

$$p_{uv}^j = \bar{p}_{uv}^j + \Delta_{uv}^i \quad (6)$$

where:

* $\bar{p}_{uv}^i$ is the intrinsic degradation transition probability from state $u$ to $v$ of component $i$;

* $p_{uv}^i$ is the transition probability from state $u$ to $v$ of component $i$ considering the impact of other components' degradation;

* $\Delta_{uv}^i$ indicates the change in the transition probability of component $i$ caused by the degradation interactions between components and verifies the following proprieties:

$$\left\{ \begin{array}{ll} -\bar{p}_{uu}^i \le \Delta_{uu}^i < 0 & \text{if } u = v \text{ and } u \ne m^i \\ 0 < \Delta_{uv}^i \le 1 - \bar{p}_{uv}^i & \text{if } v > u \text{ and } u \ne m^i \\ \Delta_{uv}^i = 0 & \text{otherwise} \end{array} \right. \quad (7)$$

Moreover, as the sum of all transition probabilities from a given state should be equal to 1, from Eq. (6) we get:

$$\sum_{v=u}^{m^i} \Delta_{uv}^i = 0 \text{ or } \Delta_{uu}^i = - \sum_{v=u+1}^{m^i} \Delta_{uv}^i \quad (8)$$

The transition probability variation $\Delta_{uv}^i$ may depend on the current degradation speed of component $i$ which is determined by the index $u$ and $v$ in the transition matrix as well as on other components' current state. In that way, it can be expressed as below:

$$\Delta_{uv}^i = \xi_{uv}^i \sum_{j=1}^{N} \tau^{ij} \quad (9)$$

where:

* $\xi_{uv}^i$ is a constant term representing the maximal marginal change in the original transition probability from state $u$ to $v$ of component $i$. From a practical point of view, the more the state degrades, the higher the change on the transition probability. Moreover, based on Eq. (7) and (8), it is reasonable to suggest the following model for $\xi_{uv}^i$:

$$\xi_{uv}^i = \begin{cases} -\bar{p}_{uu}^i & \text{if } v = u \text{ and } u \ne m^i \\ \frac{\bar{p}_{uv}^i}{1-\bar{p}_{uu}^i} \bar{p}_{uv}^j & \text{if } v > u \text{ and } u \ne m^i \\ 0 & \text{otherwise} \end{cases} \quad (10)$$

* $\tau^{ij}$ represents the degradation interaction impact of component $j$ on component $i$. It may depend on both the current state $s^j$ of component $j$ and the dependence level between component $i$ and $j$. In that way, we suggest a general model as follows:

$$\tau^{ij} = \xi^{ij} \left( \frac{s^j}{m^j} \right)^{\alpha^j} \quad (11)$$

in which:

- $\xi^{ij}$ is a non-negative real number quantifying the degree of stochastic influence of component $j$ on the degradation process of component $i$, $0 \le \xi^{ij} \le 1$. $\xi^{ij} = 0$ when $i = j$ or component $i$ and $j$ are stochastically independent, otherwise $0 < \xi^{ij} \le 1$. For example, the degradation interaction may only exits within a subsystem of series-parallel systems due to the sharing of a common load (Oakley et al., 2022), i.e., $\xi^{ij} > 0$ if components $i$ and $j$ belong to the same subsystem and $\xi^{ij} = 0$ if components $i$ and $j$ in two different subsystems. We denote $Z = \{\xi^{ij}\}_{N \times N}$ the degradation interaction matrix.

- $\alpha^j$ is a non-negative constant describing the inherent state-rate interaction of component $j$. When component $i$ and $j$ are stochastically dependent ($\xi^{ij} > 0$), $\alpha^j = 0$ means that the degradation interaction between component $i$ and $j$ does not depend on the current state of component $j$. When $\xi^{ij} > 0$ and $\alpha^j > 0$, the inherent state-rate interaction impact is proportional with the degradation level (state) of component $j$. For instance, consider an industrial cool
<figure><img src="image_5.png" /><figcaption>Fig. 3. Reliability block diagram of a parallel structure of two components.</figcaption></figure>

box system where the degradation interaction between two components manifests only when one gas tube is already in a highly fouled state (high degradation state). This occurs because the excess-heated gas is forced to pass through the other tube, leading to accelerated fouling (Rasmekomen & Parlikad, 2016). Furthermore, if component *j* is in a given surviving state, i.e., $0 < s^j/m^j < 1$, then, the higher $\alpha^j$ is, the lower the inherent state-rate interaction impact. The later also implies that when $\alpha^j$ is very high, the degradation interactions between component *i* and *j* will be triggered only by the failure of component *j*. It should be noticed that the proposed state interactions model in this case becomes an existing stochastic dependence model: the failure of a component affects the degradation process of other components (Keizer et al., 2017; Wang et al., 2022).

The components' inherent transition probability matrix can be estimated from data collected when they operate individually or might be provided by their ordinary equipment manufacturer. The parameters representing degradation interactions between components, which depend on the system's characteristics such as design structure, operating conditions, component types should be estimated from testing or historical data. If historical degradation data is available, one can use particle filters to estimate those parameters which allow for an online numerical estimation of the parameter values by means of a recursive Bayesian inference approach (Do et al., 2019). Due to the lack of real data, we assume in this study that the degradation-interaction-related parameters and the inherent state transition matrices are known.

The components' inherent transition probability matrix can be estimated from data collected when they operate individually or provided by their ordinary equipment manufacturer. The parameters representing degradation interactions between components should be estimated from testing or historical data. In this study, we assume that the degradation-interaction-related parameters and the inherent state transition matrices are known. As an example, we consider a parallel structure of two identical pumps (Fig. 3).

The health state of each pump can be specified as “good state” (state 0), “minor degradation” (state 1), “severe degradation” (state 2) and “failure” (state 3). It is assumed that the inherent transition matrix of each pump is given as follows:

$$ \bar{P}^1 = \begin{bmatrix} 0.80 & 0.10 & 0.05 & 0.05 \\ 0.00 & 0.80 & 0.10 & 0.10 \\ 0.00 & 0.00 & 0.80 & 0.20 \\ 0.00 & 0.00 & 0.00 & 1.00 \end{bmatrix} \quad (12) $$

It is assumed that the two pumps share a total load. If both pumps are in a good state (state 0), there are no interactions between them, and their transition probabilities remain unchanged as they operate with their nominal load. However, when one pump is in a degraded state (state 1 or 2), its load needs to be reduced, and the remaining component shares a part of the reduced load. Consequently, the degradation process of the remaining pump accelerates due to the overload induced by the degraded pump.

<table><caption>**Table 1**

Illustration of the change in the transition probabilities of pump 1 vs. the state of pump 2.</caption><thead><tr><th rowspan="2">State of pump 2</th><th colspan="4">Transition probabilities of pump 1</th></tr><tr><th>p<sup>1</sup><sub>00</sub></th><th>p<sup>1</sup><sub>01</sub></th><th>p<sup>1</sup><sub>02</sub></th><th>p<sup>1</sup><sub>03</sub></th></tr></thead><tbody><tr><td>Good</td><td><strong>0.8</strong></td><td><strong>0.1</strong></td><td><strong>0.05</strong></td><td><strong>0.05</strong></td></tr><tr><td>Minor degradation</td><td>0.69</td><td>0.15</td><td>0.08</td><td>0.08</td></tr><tr><td>Major degradation</td><td>0.59</td><td>0.21</td><td>0.1</td><td>0.1</td></tr><tr><td>Failure</td><td>0.48</td><td>0.26</td><td>0.13</td><td>0.13</td></tr></tbody></table>

To illustrate this interaction using our proposed model, we set $\alpha^j = 1.0$ with $i = 1, 2$ and assume that the degradation interaction matrix is described by the following matrix:

$$ Z = \begin{bmatrix} 0.0 & 0.4 \\ 0.4 & 0.0 \end{bmatrix} \quad (13) $$

Now, assume pump 1 is in a good state, i.e., its inherent transition probabilities are $p_{00}^1 = 0.80$, $p_{01}^1 = 0.10$, $p_{02}^1 = 0.05$, and $p_{03}^1 = 0.05$. According to the state interaction between the two pumps, the transition probabilities of pump 1 change depending on the current state of pump 2. Table 1 reports the changes in the transition probabilities ($p_{00}^1, p_{01}^1, p_{02}^1, p_{03}^1$) of pump 1 based on the state of pump 2. It is evident that the state of pump 2 significantly affects the degradation process of pump 1. The extent of this change depends on the interaction degree $\zeta^{ij}$ and the inherent state-rate interaction impact $\alpha^j$.

The impacts of state interactions between components on maintenance planning and optimization will be discussed in the next section.

# 5. MADRL-based maintenance decision optimization

In this section, we develop a MADRL-based approach for maintenance optimization in multi-component systems, considering both state interactions and economic dependencies to derive optimal maintenance policies. We present the maintenance optimization process by formulating the maintenance decision-making problem as a cooperative multi-agent task (see Section 5.1). Subsequently, we describe the decision optimization process in both partially and fully observable scenarios (see Section 5.2).

## 5.1. Cooperative maintenance decision-making formulation

The maintenance optimization problem for the studied system is modeled as a cooperative multi-agent decision-making task with a group of $N$ agents, denoted as $\mathcal{A}G$. Each agent, $\mathcal{A}G \in \mathcal{A}G$, controls maintenance decisions for one component. At each decision time $t_k$, each agent $\mathcal{A}G^i$ chooses an action $a_k^i$ from its own action space $\mathcal{A}^i$ based on its local policy $\pi^i$. The individual actions form a joint action $a_k \in \mathcal{A}^{\text{joint}} \equiv \mathcal{A}^1 \times \mathcal{A}^2 \times \dots \times \mathcal{A}^N$. Once the chosen joint maintenance action is carried out, the system transitions from its state before maintenance $s_k$ to a state after maintenance $\bar{s}_k$ and releases a scalar reward $r_k$ shared by all agents. Subsequently, the maintained system naturally degrades to a next state before maintenance $s_{k+1}$ at the next inspection time $t_{k+1}$. The objective of cooperative agents is to maximize the expectation of discounted return $g_k = \sum_{l=0}^{\infty} \gamma^l r_{k+l}$, where $\gamma \in [0, 1)$ is the discount factor. The joint policy $\pi$ (global policy) has a joint action-value function $Q(s_k, a_k) = \mathbb{E}[g_k|s_k, a_k]$ and a global value function $V(s_k) = \mathbb{E}[g_k|s_k]$.

### 5.1.1. Local policy

**Partial observation.** Due to partial observation, an agent *i* at timestamp $t_k$ receives only an observation $o_k^i = [s_{k-1}^i, a_{k-1}^i]$ from the component it controls, instead of the true system state $\bar{s}_k$. It is assumed that each agent *i* has the ability to memorize the history of state-action transitions $h_k^i = [o_0^i, o_1^i, \dots, o_k^i]$, conditioning a stochastic policy $\pi^i(a_k^i|h_k^i)$. In this paper, we adopt the centralized training and decentralized execution paradigm (CTDE), meaning that learning agents can access the global state during training but make maintenance decisions based on their own local histories during the execution phase.
**Full observation.** In this setting, local agents can access the system state during both training and execution phases. As a result, they make maintenance decisions conditioned on observed states at the system level. Therefore, we denote the maintenance policy of an agent i as $\pi^i(a_k^i|s_k)$.

### 5.1.2. Reward definition

Maintenance decision optimization is usually associated with the problem of balancing the trade-off between maintenance frequency and unplanned downtime cost. If maintenance actions are carried out more often, maintenance costs might be high, or if maintenance operations are conducted less frequently, system breakdowns occur more recur- rently, leading to negative consequences. To address this issue, the reward generator is defined as a function of the maintenance cost at the system level and the downtime cost, given by:

$$r_k = -c_k - \mathbb{I}_k c^{dt} \quad (14)$$

where $\mathbb{I}_k$ is the binary system failure status indicator at time $t_k$: $\mathbb{I}_k = 1$ if the system is still working, $\mathbb{I}_k = 0$ otherwise. $c^{dt}$ is a real positive constant quantifying the downtime cost.

## 5.2. WQMIX algorithm

In this section, the WQMIX algorithm is applied to optimize main- tenance decisions of the studied system in both partially and fully observable contexts. We first present the main idea and core compo- nents of this MADRL algorithm for the former scenario in Section 5.2.1 and a customized version for the latter scenario in Section 5.2.2.

### 5.2.1. Partially observable setting

Two main components of the WQMIX algorithm are the mono- tonic factorization scheme of the joint action-value function and the weighting scheme in the loss function used to update learning agents' Q-network. The factorization scheme aims to address the decision selection consistency at the component and system levels. Cooperative learning agents try to choose a common maintenance decision $a_k$ from their joint action space according to the action-value function at the system level that should be consistent with the combination of individual actions chosen by each agent separately. Mathematically, this is expressed as:

$$\begin{aligned} & \underset{a_k \in A}{\operatorname{argmax}} Q^{\text{tot}}(h_k, a_k) \\ & = \left[ \underset{a_k^1 \in A^1}{\operatorname{argmax}} Q^1(h_k^1, a_k^1) \quad \underset{a_k^2 \in A^2}{\operatorname{argmax}} Q^2(h_k^2, a_k^2) \quad \dots \quad \underset{a_k^N \in A^N}{\operatorname{argmax}} Q^N(h_k^N, a_k^N) \right] \end{aligned} \quad (15)$$

where $Q^{\text{tot}}(h_k, a_k)$ is an approximation of $Q(s_k, a_k)$. This factoriza- tion method assumes that the system action-value function can be decomposed into the per-component ones and is a continuous mono- tonic function of them. This assumption can be realized in practice by employing a mixing network, which is an MLP taking individual agents' own Q-values as input and outputting values of the approximate system action-value function $Q^{\text{tot}}(h_k, a_k)$, whose weights are generated by a hyper network to guarantee their values, ensuring they are not negative.

However, the monotonic factorization scheme restricts an agent from choosing its own action independent of the actions chosen by other agents, which may lead to finding suboptimal solutions for decision-making applications requiring strong cooperation efforts be- tween learning agents. The authors of WQMIX in Rashid et al. (2020) showed that this limit emanates from the equal weighting placed on each joint action in the loss function used to update the joint Q- function, and then proposed a special weighting method to address this issue. In particular, a feedforward network without any restriction on its weights is used to estimate $Q(s_k, a_k)$, which is then employed as baselines to put more attention on potential optimal joint actions in

<figure><img src="image_6.png" /><figcaption>Fig. 4. (a) WQMIX architecture. (b) Mixing network structures.</figcaption></figure>

the loss function. The losses of one transition sample, $(s_k, a_k, r_k, s_{k+1})$ used to update the networks representing $Q^{\text{total}}(h_k, a_k)$ and $Q(s_k, a_k)$ are computed as follows:

$$\begin{aligned} L^{\text{tot}} &= \omega(s_k, a_k)(Q^{\text{tot}}(h_k, a_k) - y)^2 \\ L^{\text{Q}} &= (Q(s_k, a_k) - y)^2 \end{aligned} \quad (16)$$

where:

* $y = r + \gamma Q_{\text{target}}(s_k, \underset{a_{k+1}}{\operatorname{argmax}} Q_{\text{target}}^{\text{tot}}(h_{k+1}, a_{k+1}))$ is the fixed target, with $Q_{\text{target}}$ and $Q_{\text{target}}^{\text{tot}}$ computed from the target networks of $Q$ and $Q^{\text{tot}}$ respectively.

* $\omega(s_k, a_k)$ is the weight of the joint action $a_k$, whose value is 1 if $Q^{\text{tot}}(h_k, a_k) < y$ or equal to $\alpha \in (0, 1]$. This weighting mechanism is called “optimistic weighting” because it assigns higher weighting to an action, $a_k$, which might be optimal if its corresponding $Q^{\text{tot}}(s_k, a_k)$ is underestimated relative to the true value that is replaced by $y$ (Rashid et al., 2020).

The general architecture of WQMIX is illustrated in Fig. 4, and the local agent network structure is depicted in Fig. 5a, which is incorporated with a recurrent neural network (e.g., Gated Recurrent Unit) aiming to memorize the history of state-action transitions of corresponding components.

### 5.2.2. Fully observable setting

In this scenario, local agents can access the global state during both training and execution phases. Therefore, we propose replacing sepa- rate agent networks used in the partially observable setting with a sin- gle branching dueling network (branching network for short) (Tavakoli et al., 2018) to take advantage of the common system state. The special structure of the branching network allows us to achieve a linear increase in the size of deep Q-networks' output layer to avoid the curse of dimensionality. Furthermore, it also allows creating virtual communication channels between cooperative learning agents to facil- itate decision-making processes as well as to avoid the use of recurrent neural networks to memorize local transition histories in the partially observable setting that may slow down the learning process.

The structure of the branching network is depicted in Fig. 5b. It can be noticed that the system state is considered as an input for all agents; however, outputs of each branch of the branching network are still individual action values of corresponding agents. Specifically, the system state, $s_k$, is first passed through the shared MLP to encode
<figure><img src="image_7.png" /><figcaption>Fig. 5. (a) Agent network structure in partially observable setting (b) Agent network structure in fully observable setting.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 6. Reliability block diagram of the studied five-component system.</figcaption></figure>

latent representations between learning agents, which is then used to calculate the common system state value, $V(s_k)$, as well as the individual advantage values, $\Omega^i(s_k, a_k^i)$ ($i \in \{1, ..., N\}$). After that, the agents' local action values are computed by aggregating the system state and the advantage values in each branch using the following equation (Tavakoli et al., 2018):

$$Q^i(s_k, a_k^i) = V(s_k) + \Omega^i(s_k, a_k^i) - \frac{1}{|\mathcal{A}^i|} \sum_{a_k \in \mathcal{A}^i} \Omega^i(s_k, a_k^i) \quad (17)$$

It is noteworthy that since the gradient of all branches is backpropagated through the shared MLP, the combined gradient prior to entering the deepest layer of this network is rescaled practically by a factor of $1/\sqrt{N} + 1$ to improve the stability of training processes. Moreover, it should be noted that the general architecture of WQMIX and the computational process of $Q^{tot}$ and $Q$ are the same for both partially and fully observable contexts.

# 6. Numerical experiments

We conduct simulation experiments on two series-parallel systems of different sizes, namely, a 5-component system (Section 6.1) and an 11-component system (Section 6.2). The goal is to benchmark the performance of the WQMIX algorithm with several baselines and investigate the impact of component dependencies on optimized maintenance policies. The experiments are implemented on a desktop computer with a 3.91 GHz CPU, 64 GB RAM, and 16 GB dedicated GPU memory using the Python programming language and PyTorch framework.

## 6.1. Five-component system

### 6.1.1. System description

This small system consists of 5 components divided into two subsystems of identical components, as illustrated in Fig. 6. Specifically, components 1 and 2 belong to the first subsystem, and the remaining components belong to the second one.

<table><caption>Table 2
Parameters of the studied 5-component system.</caption><thead><tr><th></th><th>c<sup>i</sup></th><th>c<sup>ins</sup></th><th>c<sup>r</sup></th><th>β</th></tr></thead><tbody><tr><td>Components in H<sup>1</sup></td><td>10</td><td>2</td><td>60</td><td>5</td></tr><tr><td>Components in H<sup>2</sup></td><td>5</td><td>3</td><td>40</td><td>3</td></tr></tbody></table>

Each component has 4 different states: “good state” (state 0), “minor degradation” (state 1), “severe degradation” (state 2), and “complete failure” (state 3). The representative inherent transition matrices of the components of the two subsystems are given as follows:

$$\tilde{P}^1 = \begin{bmatrix} 0.80 & 0.10 & 0.05 & 0.05 \\ 0.00 & 0.80 & 0.10 & 0.10 \\ 0.00 & 0.00 & 0.80 & 0.20 \\ 0.00 & 0.00 & 0.00 & 1.00 \end{bmatrix}, \quad \tilde{P}^3 = \begin{bmatrix} 0.70 & 0.20 & 0.05 & 0.05 \\ 0.00 & 0.70 & 0.20 & 0.10 \\ 0.00 & 0.00 & 0.70 & 0.30 \\ 0.00 & 0.00 & 0.00 & 1.00 \end{bmatrix} \quad (18)$$

We set $\alpha^i = 1.0 \forall i \in \{1, 2, ..., 5\}$ and assume that degradation interactions only exist between components of the same subsystem, which is described through the following matrix:

$$Z = \begin{bmatrix} 0.0 & 0.4 & 0.0 & 0.0 & 0.0 \\ 0.4 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.2 & 0.2 \\ 0.0 & 0.0 & 0.2 & 0.0 & 0.2 \\ 0.0 & 0.0 & 0.2 & 0.2 & 0.0 \end{bmatrix} \quad (19)$$

All cost parameters are given in arbitrary units. The system setup cost and the downtime cost are $c^0 = 20$ and $c^{dt} = 200$, respectively. The representative parameters of the components of the two subsystems of the studied 5-component system are depicted in Table 2.

### 6.1.2. Baselines and training descriptions

In this comparison study, the WQMIX algorithm is employed to optimize maintenance policy for the five-component system, and the results are compared with Dueling DDQN (Wang et al., 2016) and dynamic programming or value iteration (VI) (Sutton & Barto, 2018). To facilitate the distinction between WQMIX agents in fully and partially observable settings, we denote the simulation results of these two settings as “F-WQMIX” and “P-WQMIX” respectively.

**F-WQMIX.** This MADRL algorithm is trained through $0.8 \times 10^6$ steps. The learning rate is scheduled to decrease from 0.004 to 0.00025 during the first $300 \times 10^3$ training steps. The constant used for the exploration of learning agents is annealed linearly from 1.0 to 0.05 over $200 \times 10^3$ training steps and kept constant for the rest of the learning. A uniform replay buffer that can store up to $300 \times 10^3$ system transitions is employed as a dynamic dataset, with a mini-batch of size 128 being uniformly sampled to update agents' networks. We set the discount factor $\gamma = 0.99$. The target update frequency is $20 \times 10^3$ steps. To monitor
<table><caption>**Table 3**
Cost rate and optimization time summary for the studied 5-component system.</caption><thead><tr><th></th><th>F-WQMIX</th><th>Dueling DDQN</th><th>P-WQMIX</th><th>VI</th></tr></thead><tbody><tr><td>Cost rate</td><td>67.15</td><td>68.75</td><td>71.25</td><td>67.15</td></tr><tr><td>Optimization time (h)</td><td>2.12</td><td>1.58</td><td>4.42</td><td>9.55</td></tr></tbody></table>

the learning progress, the latest policy networks after every $10^3$ steps are employed to generate $5 \times 10^3$ consecutive maintenance decisions on a validation environment to calculate corresponding cost rates.

The input layer size of F-WQMIX's branching network is equal to the number of components of the studied system, which is 5. The shared MLP is composed of two layers of 128 hidden units, and the advantage MLP for each branch and the MLP used for calculating the system value function contains a single layer of 64 and 128 hidden units, respectively. Each branch of the branching network outputs 3 action-values corresponding to 3 maintenance decisions. The mixing network for $Q^{tot}$ is comprised of two hidden layers of 64 units, whose weights are generated by two separate hyper-networks of 64 units. The mixing network of $Q$ is an MLP of two hidden layers of 64 units.

**P-WQMIX.** The local agent networks of P-WQMIX contain a fully-connected layer of 128 hidden units followed by a GRU of 64-dimensional hidden states and a fully-connected layer of 64 hidden units. The configuration of the mixing networks, training step, discount factor, learning rate scheduling, and policy evaluation are the same as the ones of F-WQMIX.

While the branching network of F-WQMIX is updated via batches of single system transition points, the local agent networks of P-WQMIX are updated through the gradient of sequences of system transitions due to the incorporation of a recurrent neural network. As a result, a replay buffer that can hold up to 2000 sequences of 100 system transitions is employed. A mini-batch size of 64 sequences is sampled uniformly from this buffer to update agent networks.

**Dueling DDQN.** The size of the dueling network's input layer is 5, and its shared MLP is composed of two hidden layers of 256 units. A single fully-connected layer of 128 hidden units is used to compute the system advantage and value function. The training setup for Dueling DDQN is similar to F-WQMIX.

VI. This dynamic programming algorithm can be used here to optimize maintenance decisions thanks to the small state and action space of the studied system, which are 1024 and 243, respectively. The VI algorithm is trained until the expected estimation accuracy of the value function reaches 0.001. The obtained policy is evaluated via simulation results through 5 runs of 5000 maintenance interventions.

### 6.1.3. Simulation results

The evolution of cost rates during training of the three reinforcement learning algorithms is depicted in Fig. 7, and their best found cost rates are summarized in the first row of Table 3. It can be noticed that P-WQMIX and Dueling DDQN are incapable of approaching the global optimal policy obtained by VI and F-WQMIX with the corresponding cost rate of 67.15. The lower performance of P-WQMIX in comparison to F-WQMIX can be explained by the partial observability that causes difficulties for local agents to communicate with each other. For the reason why Dueling DDQN cannot approach the global policy, we think that it is due to the relatively large size of the action space which is 243 that can harm the performance of the single-agent DRL algorithm.

Based on the second row of Table 3, we can see that the computing time of VI is the largest, confirming the well-known optimization time problem of dynamic programming algorithms. The training time of F-WQMIX and Dueling DDQN is relatively reasonable for this small system. In comparison to two other reinforcement learning algorithms, P-WQMIX has the largest training time due to the update scheme used for recurrent neural networks.

<table><caption>**Table 4**
Illustration of the optimized maintenance policies for systems with and without component degradation interactions.</caption><thead><tr><th>s<sub>k</sub></th><th>a<sup>*</sup><sub>k</sub></th><th>a<sub>k</sub></th></tr></thead><tbody><tr><td>[2, 3, 3, 2, 1]</td><td>[1, 2, 2, 1, 0]</td><td>[0, 2, 2, 0, 0]</td></tr><tr><td>[2, 3, 3, 2, 2]</td><td>[1, 2, 2, 1, 1]</td><td>[0, 2, 2, 0, 0]</td></tr><tr><td>[2, 3, 3, 2, 3]</td><td>[1, 2, 2, 1, 2]</td><td>[0, 2, 2, 1, 0]</td></tr><tr><td>[2, 3, 3, 3, 0]</td><td>[1, 2, 2, 2, 0]</td><td>[0, 2, 2, 0, 0]</td></tr><tr><td>[3, 2, 1, 2, 2]</td><td>[2, 1, 0, 1, 1]</td><td>[2, 0, 0, 0, 0]</td></tr><tr><td>[3, 2, 1, 2, 3]</td><td>[2, 1, 0, 1, 2]</td><td>[2, 0, 0, 0, 2]</td></tr><tr><td>[3, 2, 1, 3, 0]</td><td>[2, 1, 0, 2, 0]</td><td>[2, 0, 0, 0, 0]</td></tr><tr><td>[3, 2, 1, 3, 1]</td><td>[2, 1, 0, 2, 0]</td><td>[2, 0, 0, 0, 0]</td></tr><tr><td>[3, 2, 1, 3, 2]</td><td>[2, 1, 0, 2, 1]</td><td>[2, 0, 0, 2, 0]</td></tr><tr><td>[3, 3, 3, 1, 0]</td><td>[2, 2, 2, 0, 0]</td><td>[2, 2, 0, 0, 0]</td></tr><tr><td>[3, 3, 3, 1, 1]</td><td>[2, 2, 2, 0, 0]</td><td>[2, 2, 0, 0, 0]</td></tr><tr><td>[3, 3, 3, 1, 2]</td><td>[2, 2, 2, 0, 1]</td><td>[2, 2, 2, 0, 0]</td></tr><tr><td>[3, 3, 3, 1, 3]</td><td>[2, 2, 2, 0, 2]</td><td>[2, 2, 2, 0, 0]</td></tr><tr><td>[3, 3, 3, 2, 0]</td><td>[2, 2, 2, 1, 0]</td><td>[2, 2, 0, 0, 0]</td></tr><tr><td>[3, 3, 3, 3, 2]</td><td>[2, 2, 2, 2, 1]</td><td>[2, 2, 2, 0, 1]</td></tr><tr><td>[3, 3, 3, 3, 3]</td><td>[2, 2, 2, 2, 2]</td><td>[2, 2, 2, 2, 0]</td></tr></tbody></table>

### 6.1.4. Sensitivity analysis

In order to examine the performance of maintenance policies that do not take degradation interactions between components into consideration, a similar experimental study is conducted on the same studied five-component system except that the components degrade independently of each other. Maintenance actions optimized by F-WQMIX are then employed to make maintenance decisions on the ground-truth system in which there exists the degradation interaction between components to determine its corresponding average cost rate. The obtained cost rate is 76.07, which is larger than the optimal one (67.15).

An illustration of two optimized policies is depicted in Table 4 showing that the optimal policy has an incentive to perform maintenance actions more frequently than the non-optimal one due to the fact that the degradation interactions between components make the system more prone to be shut down if one component degrades to a severe condition state. Moreover, it can be noticed that imperfect maintenance actions are implemented for the system with degradation interactions as a way to improve the system reliability with lower cost than conducting replacement actions.

## 6.2. Eleven-component system

### 6.2.1. System description

The examined series-parallel system comprises 11 components grouped into five subsystems of identical components, as depicted in Fig. 8. Specifically, Component 1 forms the first subsystem, followed by the second subsystem composed of Components 2 and 3. Components 4, 5, and 6 together constitute the third subsystem. The fourth subsystem consists of Components 7 and 8, while the last subsystem comprises the remaining components.

Each component has four condition states, defined identically to those used for the five-component system. Subsystems 1 and 2 degrade according to two representative transition matrices $\bar{P}^1$ and $\bar{P}^3$ in the absence of degradation interaction between components. The inherent transition matrices for the three remaining subsystems are given below:

$$ \bar{P}^6 = \begin{bmatrix} 0.65 & 0.25 & 0.05 & 0.05 \\ 0.00 & 0.65 & 0.25 & 0.10 \\ 0.00 & 0.00 & 0.65 & 0.35 \\ 0.00 & 0.00 & 0.00 & 1.00 \end{bmatrix}, \quad \bar{P}^8 = \begin{bmatrix} 0.60 & 0.30 & 0.05 & 0.05 \\ 0.00 & 0.60 & 0.30 & 0.10 \\ 0.00 & 0.00 & 0.60 & 0.40 \\ 0.00 & 0.00 & 0.00 & 1.00 \end{bmatrix}, \qquad (20) $$
<figure><img src="image_9.png" /><figcaption>(a) The evolution of cost rates during training</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>(b) A closer look at the evolution of cost rates</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 8. Reliability block diagram of the studied 11-component system.</figcaption></figure>

<table><caption>**Table 5**
Parameters of the studied 11-component system.</caption><thead><tr><th></th><th>c<sup>r</sup></th><th>c<sup>ins</sup></th><th>c<sup>r</sup></th><th>β</th></tr></thead><tbody><tr><td>Components in H<sup>1</sup></td><td>10</td><td>2</td><td>60</td><td>5</td></tr><tr><td>Components in H<sup>2</sup></td><td>5</td><td>3</td><td>40</td><td>3</td></tr><tr><td>Components in H<sup>3</sup></td><td>15</td><td>4</td><td>35</td><td>4</td></tr><tr><td>Components in H<sup>4</sup></td><td>10</td><td>5</td><td>30</td><td>3</td></tr><tr><td>Components in H<sup>5</sup></td><td>5</td><td>4</td><td>35</td><td>3</td></tr></tbody></table>

In this experiment, we assume degradation interaction can occur between components of different subsystems, including subsystems 1 and 2, and subsystems 4 and 5. Additionally, degradation interaction exists within the same subsystem. Thus, we use the following degradation interaction matrix to describe component interactions:

$$Z = \begin{bmatrix} 0.0 & 0.1 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.1 & 0.0 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.1 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.1 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.1 & 0.0 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.1 & 0.0 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.1 & 0.0 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.1 & 0.1 & 0.1 & 0.1 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.1 & 0.1 & 0.1 & 0.1 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.1 & 0.1 & 0.1 & 0.1 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.1 & 0.1 & 0.1 & 0.1 \end{bmatrix} \quad (21)$$

The system setup cost and downtime cost for the considered system are increased to $c^0 = 150$ and $c^{dt} = 600$ respectively due to the system's size. Representative cost parameters for the five subsystems are provided in Table 5.

### 6.2.2. Baselines and training descriptions

VI and Dueling DDQN are not applicable here due to the high dimensionality of the state and action space of the studied system. The total number of system states exceeds $4 \times 10^6$, and the number of all possible system actions surpasses $177 \times 10^3$. These numbers highlight the curse of dimensionality that decision optimization algorithms must confront when applied to large-scale systems.

In evaluating the efficacy of F-WQMIX and P-WQMIX, an exhaustive comparative analysis is undertaken, encompassing a maintenance strategy grounded in the state-of-the-art Multi-Agent Deep Q Network (MADQN). In contrast to QMIX, MADQN represents a decentralized paradigm tailored for scenarios where agents autonomously operate within a common environment. Within the MADQN framework, each agent employs an individual deep Q network to learn and optimize decision-making processes based on fully observed environmental states. This decentralized structure not only promotes scalability and simplicity but also caters to scenarios requiring autonomous decision-making among multiple agents. Beyond its conceptual framework, MADQN has demonstrated applicability in diverse domains, including maintenance and process optimization, as evidenced by recent studies (He et al., 2022; Lee & Song, 2023; Wang et al., 2019; Zhou et al., 2022). The subsequent paragraphs elucidate the training configurations of the MADRL algorithms.

**F-WQMIX.** The F-WQMIX algorithm is trained over $3 \times 10^6$ steps with a linearly decreasing learning rate from 0.001 to 0.00025 during the first $500 \times 10^3$ training steps. The mini-batch size, discount factor, and the size of the replay buffer are set to 128, 0.99, and $500 \times 10^3$ respectively. The target update frequency is $20 \times 10^3$ steps. The input size of the branching network of F-WQMIX is 11, corresponding to the number of components, and the number of hidden units of the shared MLP is increased from 128 to 256 due to the larger system size.
<figure><img src="image_12.png" /><figcaption>Fig. 9. Agent training monitoring for the studied 11-component system.</figcaption></figure>

<table><caption>**Table 6**
Cost rate and optimization time summary for the studied 11-component system.</caption><thead><tr><th></th><th>F-WQMIX</th><th>P-WQMIX</th><th>MADQN</th></tr></thead><tbody><tr><td>Cost rate</td><td>279.85</td><td>328.56</td><td>309.58</td></tr><tr><td>Optimization time (h)</td><td>26.33</td><td>107.25</td><td>10.10</td></tr></tbody></table>

**P-WQMIX.** The configuration for training the local agent networks of P-WQMIX is the same as in the previous section, except that the replay buffer can now hold up to 3000 sequences of 300 system transitions.

**MADQN.** The MADQN algorithm undergoes training over $3 \times 10^6$ steps with a scheduled learning rate of 0.004. The exploration constant for learning agents linearly decreases from 1.0 to 0.05 over $200 \times 10^3$ training steps and remains constant thereafter. Utilizing a uniform replay buffer capable of storing up to $10 \times 10^3$ system transitions, a dynamic dataset is used, and a mini-batch of size 128 is uniformly sampled to update agent networks. A discount factor of 0.99 is set, and the target network is updated every $20 \times 10^3$ steps. The input layer size for each Deep Q Network (DQN) is determined by the number of components in the system, which is 11. Comprising two layers with 256 hidden units each, the DQN network outputs three action-values corresponding to three distinct maintenance decisions.

### 6.2.3. Simulation results

The simulation results, presented in Fig. 9 and Table 6, reveal that the cost rate obtained by the P-WQMIX-based approach is the highest at 328.56. F-WQMIX agents achieve the best cost rate of 279.85. The challenges associated with partial observation in WQMIX become increasingly evident with a higher number of components. Particularly, the performance of P-WQMIX may deteriorate as the system size expands. The curse of dimensionality intensifies with the system's growing complexity, restricting access to the full state and hindering effective policy exploration. In larger systems, intricate component interactions can substantially influence optimal policies, but partial observation limits the agent's access to crucial information. Specifically, the optimized cost rates of F-WQMIX are 6.11% and 17.41% higher than those achieved by P-WQMIX for the 5-component and 11-component systems studied, respectively. The absence of complete information complicates the learning of the system's dynamics, leading to slower and less effective decision-making. Notably, the training time of P-WQMIX significantly increases due to the larger number of training steps and the extended sequence length of system transitions used to train agent networks.

Regarding the MADQN approach, the achieved cost rate is 309.58. A comparative analysis between MADQN and the F-QMIX approach

reveals distinctive performance characteristics. Specifically, MADQN demonstrates faster convergence, while WQMIX consistently produces superior results. The accelerated convergence of MADQN is attributed to its decentralized learning paradigm, enabling individual agents to swiftly adapt their policies autonomously. In contrast, WQMIX's efficacy in achieving superior outcomes stems from its centralized training, fostering global coordination among agents. This centralized approach empowers WQMIX to capture and leverage collective intelligence, enhancing decision-making in maintenance optimization scenarios. However, the trade-off lies in the computational complexity associated with WQMIX, given that its centralized learning process demands increased computational resources.

## 7. Conclusions and future works

In this study, we proposed a multi-agent deep reinforcement learning-based maintenance approach for systems consisting of multi-state components subjected to both state interactions and economic dependence. A novel state interactions model was introduced to describe the stochastic dependence between components. We developed a maintenance planning approach based on the multi-agent deep reinforcement learning algorithm, WQMIX, for decision-making and optimization in both fully and partially observed states. Thanks to the WQMIX algorithm, the proposed maintenance approach is scalable and efficiently considers the state interactions and economic dependence between components.

Comprehensive studies were conducted to investigate the impact of component dependencies and partial observability on finding optimal maintenance policies. Numerical results demonstrated that optimal maintenance policies are incentivized to perform maintenance actions in groups due to economic dependence and more frequently because of state interactions, making the system more prone to shutdown if one component degrades to a severe health condition state.

Additionally, the performance of the WQMIX algorithm is compromised in the partially observable setting, especially when the size of the maintained system is large. When compared to traditional maintenance methods such as value iteration method, Dueling Double Deep Q Network, and Multi-Agent Deep Q Network, the proposed maintenance approach is more suitable. However, the computing time is considerably high, particularly for partial settings and/or large-scale systems.

Our future research will focus on improving the learning speed of MADRL algorithms based on reward shaping techniques to help cooperative agents distinguish similar actions and/or incorporating
expert knowledge into the optimization process to reduce the size of the policy search space. Additionally, the application of the proposed maintenance approach to real industrial case studies remains a challenging research topic for our future work.

### CRediT authorship contribution statement

**Phuc Do:** Investigation, Writing – original draft, Conceptualization, Methodology, Writing – review & editing. **Van-Thai Nguyen:** Investigation, Conceptualization, Methodology, Writing – original draft. **Alexandre Voisin:** Supervision, Methodology, Reviewing. **Benoit Iung:** Supervision, Reviewing. **Waldomiro Alves Ferreira Neto:** Investigation, Reviewing.

### Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

### Acknowledgments

MODAPTO has received funding from the European Union's Horizon Europe research and innovation program under grant agreement No 101091996.

## Appendix. Acronyms & notations

### Acronyms

<table><tr><td>CBM</td><td>Condition-Based Maintenance</td></tr><tr><td>CTDE</td><td>Centralized Training and Decentralized Execution</td></tr><tr><td>DCMAC</td><td>Deep Centralized Multi-Agent Actor-Critic</td></tr><tr><td>DDQN</td><td>Double Deep Q-Network</td></tr><tr><td>Dec-POMDP</td><td>Decentralized Partially Observable Markov Decision Process</td></tr><tr><td>CM</td><td>Corrective Maintenance</td></tr><tr><td>GA</td><td>Genetic Algorithm</td></tr><tr><td>DRL</td><td>Deep Reinforcement Learning</td></tr><tr><td>MADRL</td><td>Multi-Agent Deep Reinforcement Learning</td></tr><tr><td>MDP</td><td>Markov Decision Process</td></tr><tr><td>MPL</td><td>Multi-Layer Perceptron</td></tr><tr><td>PM</td><td>Preventive Maintenance</td></tr><tr><td>VDN</td><td>Value Decomposition Networks</td></tr><tr><td>WQMIX</td><td>Weighted QMIX Algorithm</td></tr></table>

### Notations

<table><tr><td>N</td><td>Number of components</td></tr><tr><td>H<sup>j</sup></td><td>Number of components of type j</td></tr><tr><td>k</td><td>Index for timestamps</td></tr><tr><td>m<sup>i</sup></td><td>m<sup>i</sup> + 1 is the number of health states of component i</td></tr><tr><td>s<sup>i</sup><sub>k</sub></td><td>State of component i before maintenance at kth inspection</td></tr><tr><td>s̄<sup>i</sup><sub>k</sub></td><td>State of component i after maintenance at kth inspection</td></tr><tr><td>s<sub>k</sub></td><td>System state before maintenance at kth inspection</td></tr><tr><td>s̄<sub>k</sub></td><td>System state after maintenance at kth inspection</td></tr><tr><td>a<sup>i</sup><sub>k</sub></td><td>Maintenance action of component i at kth inspection</td></tr></table>

<table><tr><td>a<sub>k</sub></td><td>System action at kth inspection</td></tr><tr><td>P̄<sup>i</sup></td><td>Inherent transition matrix of component i</td></tr><tr><td>P<sup>i</sup></td><td>Transition matrix of component i affected by state interactions</td></tr><tr><td>τ<sup>i</sup></td><td>Degradation interaction factor of component i</td></tr><tr><td>ζ<sup>i,j</sup></td><td>Constant quantifying structure influence of component j on component i's degradation nature</td></tr><tr><td>α<sup>j</sup></td><td>Constant describing inherent degradation interaction characteristics of component j</td></tr><tr><td>ζ<sup>i,j</sup><sub>uv</sub></td><td>Maximal change in the original transition probability from state u to v of component i</td></tr><tr><td>Z</td><td>Degradation interaction matrix</td></tr><tr><td>c<sup>ins,i</sup></td><td>Inspection cost of component i</td></tr><tr><td>c<sup>0</sup></td><td>System setup cost</td></tr><tr><td>c<sup>i,j</sup></td><td>Component-type setup cost</td></tr><tr><td>c<sup>m,j</sup></td><td>Component-specific maintenance cost</td></tr><tr><td>c<sup>i</sup></td><td>Cost of maintaining component i separately</td></tr><tr><td>c<sup>dt</sup></td><td>Downtime cost</td></tr><tr><td>c</td><td>Total maintenance cost at the system level</td></tr><tr><td>Q<sup>i</sup></td><td>Action-value function of component i</td></tr><tr><td>Ω<sup>i</sup></td><td>Advantage function of component i</td></tr><tr><td>Q</td><td>Joint action-value function</td></tr><tr><td>Q<sup>tot</sup></td><td>Factorized joint action-value function</td></tr><tr><td>V</td><td>Joint state-value function</td></tr></table>

### References

Ahmad, R., & Kamaruddin, S. (2012). An overview of time-based and condition-based maintenance in industrial application. *Computers & Industrial Engineering*, 63(1), 135-149.

Alaswad, S., & Xiang, Y. (2017). A review on condition-based maintenance optimization models for stochastically deteriorating system. *Reliability Engineering & System Safety*, 157, 54-63.

Andriotis, C., & Papakonstantinou, K. (2019). Managing engineering systems with large state and action spaces through deep reinforcement learning. *Reliability Engineering & System Safety*, 191, Article 106483.

Andriotis, C., & Papakonstantinou, K. (2021). Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints. *Reliability Engineering & System Safety*, 212, Article 107551.

Ayvaz, S., & Alpay, K. (2021). Predictive maintenance system for production lines in manufacturing: A machine learning approach using IoT data in real-time. *Expert Systems with Applications*, 173, Article 114598.

Bian, L., & Gebraeel, N. (2014). Stochastic framework for partially degradation systems with continuous component degradation-rate-interactions. *Naval Research Logistics*, 61(4), 286-303.

Castanier, B., Grall, A., & Bérenguer, C. (2005). A condition-based maintenance policy with non-periodic inspections for a two-unit series system. *Reliability Engineering & System Safety*, 87(1), 109-120.

Dalzochio, J., Kunst, R., Pignatton, E., Binotto, A., Sanyal, S., Favilla, J., & Barbosa, J. (2020). Machine learning and reasoning for predictive maintenance in Industry 4.0: Current status and challenges. *Computers in Industry*, 123, Article 103298.

Dekker, R., Wildeman, R. E., & Van der Duyn Schouten, F. A. (1997). A review of multi-component maintenance models with economic dependence. *Mathematical Methods of Operations Research*, 45(3), 411-435.

Dinh, D.-H., Do, P., & Iung, B. (2020). Degradation modeling and reliability assessment for a multi-component system with structural dependence. *Computers & Industrial Engineering*, 144, Article 106443.

Do, P., Assaf, R., Scarf, P., & Iung, B. (2019). Modelling and application of condition-based maintenance for a two-component system with stochastic and economic dependencies. *Reliability Engineering & System Safety*, 182, 86-97.

Do, V. P., & Bérenguer, C. (2012). Condition-based maintenance with imperfect preventive repairs for a deteriorating production system. *Quality and Reliability Engineering International*, 28(6), 624-633.

Dong, W., Liu, S., & Du, Y. (2019). Optimal periodic maintenance policies for a parallel redundant system with component dependencies. *Computers & Industrial Engineering*, 138, Article 106133.

Gorenstein, A., & Kalech, M. (2022). Predictive maintenance for critical infrastructure. *Expert Systems with Applications*, 210, Article 118413.

Guillén, A. J., Crespo, A., Gómez, J. F., & Sanz, M. D. (2016). A framework for effective management of condition based maintenance programs in the context of industrial development of E-maintenance strategies. *Computers in Industry*, 82, 170-185.

He, Z., Tran, K. P., Thomassey, S., Zeng, X., Xu, J., & Yi, C. (2022). Multi-objective optimization of the textile manufacturing process using deep-q-network based multi-agent reinforcement learning. *Journal of Manufacturing Systems*, 62, 939-949.
Huang, J., Chang, Q., & Arinez, J. (2020). Deep reinforcement learning based preventive maintenance policy for serial production lines. *Expert Systems with Applications, 160*, Article 113701.

Huynh, K. T., Barros, A., & Bérenguer, C. (2013). A reliability-based opportunistic predictive maintenance model for k-out-of-n deteriorating systems. *Chemical Engineering Transactions, 33*, 493-498.

Keizer, M. C. O., Flapper, S. D. P., & Teunter, R. H. (2017). Condition-based maintenance policies for systems with multiple dependent components: A review. *European Journal of Operational Research, 261*(2), 405-420.

Lee, D., & Song, J. (2023). Risk-informed operation and maintenance of complex lifeline systems using parallelized multi-agent deep Q-network. *Reliability Engineering & System Safety, 239*, Article 109512.

Nguyen, K. T., Do, P., Huynh, K. T., Bérenguer, C., & Grall, A. (2019). Joint optimization of monitoring quality and replacement decisions in condition-based maintenance. *Reliability Engineering & System Safety, 189*, 177-195.

Nguyen, V.-T., Do, P., Vosin, A., & Iung, B. (2022). Artificial-intelligence-based maintenance decision-making and optimization for multi-state component systems. *Reliability Engineering & System Safety, 228*, Article 108757.

Nicolai, R. P., & Dekker, R. (2008). Optimal maintenance of multi-component systems: a review. *Complex System Maintenance Handbook*, 263-286.

Oakley, J. L., Wilson, K. J., & Philipson, P. (2022). A condition-based maintenance policy for continuously monitored multi-component systems with economic and stochastic dependence. *Reliability Engineering & System Safety, Article 108321*.

Rashid, T., Farquhar, G., Peng, B., & Whiteson, S. (2020). Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. *Advances in Neural Information Processing Systems*.

Rasmekomen, N., & Parlikad, A. K. (2014). Optimising maintenance of multi-component systems with degradation interactions. *IFAC Proceedings Volumes, 47*(3), 7098-7103.

Rasmekomen, N., & Parlikad, A. K. (2016). Condition-based maintenance of multi-component systems with degradation state-rate interactions. *Reliability Engineering & System Safety, 148*, 1-10.

Rausand, M., & Hoyland, A. (2003). *System reliability theory: Models, statistical methods, and applications*, vol. 396. John Wiley & Sons.

Rocchetta, R., Bellani, L., Compare, M., Zio, E., & Patelli, E. (2019). A reinforcement learning framework for optimal operation and maintenance of power grids. *Applied Energy, 241*, 291-301.

Roda, I., & Macchi, M. (2021). Maintenance concepts evolution: a comparative review towards advanced maintenance conceptualization. *Computers in Industry, 133*, Article 103531.

Rokhforoz, P., Montazeri, M., & Fink, O. (2023). Safe multi-agent deep reinforcement learning for joint bidding and maintenance scheduling of generation units. *Reliability Engineering & System Safety, 232*, Article 109081.

Ruan, J., Wang, Z., Chan, F. T., Patnaik, S., & Tiwari, M. (2021). A reinforcement learning-based algorithm for the aircraft maintenance routing problem. *Expert Systems with Applications, 169*, Article 114399.

Shi, H., & Zeng, J. (2016). Real-time prediction of remaining useful life and preventive opportunistic maintenance strategy for multi-component systems considering stochastic dependence. *Computers & Industrial Engineering, 93*, 192-204.

Skordilis, E., & Moghaddass, R. (2020). A deep reinforcement learning approach for real-time sensor-driven decision making and predictive analytics. *Computers & Industrial Engineering, 147*, Article 106600.

Su, J., Huang, J., Adams, S., Chang, Q., & Beling, P. A. (2022). Deep multi-agent reinforcement learning for multi-level preventive maintenance in manufacturing systems. *Expert Systems with Applications, 192*.

Sutton, R. S., & Barto, A. G. (2018). *Reinforcement learning: An introduction*. MIT Press.

Tavakoli, A., Pardo, F., & Kormushev, P. (2018). Action branching architectures for deep reinforcement learning. In *Proceedings of the AAAI conference on artificial intelligence*, vol. 32, no. 1.

Thomas, L. (1986). A survey of maintenance and replacement models for maintainability and reliability of multi-item systems. *Reliability Engineering, 16*(4), 297-309.

Tsang, A. H. (1995). Condition-based maintenance: tools and decision making. *Journal of Quality in Maintenance Engineering*.

Wang, Y., Li, X., Chen, J., & Liu, Y. (2022). A condition-based maintenance policy for multi-component systems subject to stochastic and economic dependencies. *Reliability Engineering & System Safety, 219*, Article 108174.

Wang, Y., Liu, H., Zheng, W., Xia, Y., Li, Y., Chen, P., Guo, K., & Xie, H. (2019). Multi-objective workflow scheduling with deep-q-network-based multi-agent reinforcement learning. *IEEE Access, 7*, 39974-39982.

Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., & Freitas, N. (2016). Dueling network architectures for deep reinforcement learning. In *International conference on machine learning* (pp. 1995-2003). PMLR.

Wijnmalen, D. J., & Hontelez, J. A. (1997). Coordinated condition-based repair strategies for components of a multi-component maintenance system with discounts. *European Journal of Operational Research, 98*(1), 52-63.

Wong, A., Bäck, T., Kononova, A. V., & Plaat, A. (2023). Deep multiagent reinforcement learning: Challenges and directions. *Artificial Intelligence Review, 56*(6), 5023-5056.

Zhang, N., & Si, W. (2020). Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks. *Reliability Engineering & System Safety, 203*, Article 107094.

Zhang, X., & Zeng, J. (2015). Deterioration state space partitioning method for opportunistic maintenance modelling of identical multi-unit systems. *International Journal of Production Research, 53*(7), 2100-2118.

Zhang, P., Zhu, X., & Xie, M. (2021). A model-based reinforcement learning approach for maintenance optimization of degrading systems in a large state space. *Computers & Industrial Engineering, 161*, Article 107622.

Zhou, Y., Li, B., & Lin, T. R. (2022). Maintenance optimisation of multicomponent systems using hierarchical coordinated reinforcement learning. *Reliability Engineering & System Safety, 217*, Article 108078.

Zhou, Y., Zhang, Z., Lin, T. R., & Ma, L. (2013). Maintenance optimisation of a multi-state series-parallel system considering economic dependence and state-dependent inspection intervals. *Reliability Engineering & System Safety, 111*, 248-259.

Zonta, T., Da Costa, C. A., da Rosa Righi, R., de Lima, M. J., da Trindade, E. S., & Li, G. P. (2020). Predictive maintenance in the Industry 4.0: A systematic literature review. *Computers & Industrial Engineering, 150*, Article 106889.