SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The transition model can be defined based on the deterioration and inspection models previously described. If no inspection and maintenance are taken (i.e. do-nothing action), the damage condition progresses each time step according to the fatigue deterioration model formulated in Eq. 1. Note that in our three sets of environments, a time step represents a year. Considering that the damage follows a non-stationary deterioration process, the crack size distribution  $d_{t+1}$  can be efficiently encoded as a function of the annual deterioration rate,  $\tau_{t+1}$ , and the crack size at the previous time step  $d_t$  as  $p(d_{t+1}|d_t,\tau_{t+1})$ . Starting from  $\tau_0 = 0$ , the deterioration rate increases by one unit every year, unless a component is repaired, in which case the deterioration rate returns to the initial value. The deterioration evolution over a time step can be computed as:
$$
p \left(d _ {t + 1}\right) = \sum_ {\tau_ {t + 1}} \sum_ {d _ {t}} p \left(d _ {t + 1} \mid d _ {t}, \tau_ {t + 1}\right) p \left(d _ {t}\right) p \left(\tau_ {t + 1}\right). \tag {4}
$$
If an inspection action is planned, a damage indication  $i_{d_{t+1}}$  is collected, and the crack size distribution can be updated via Bayes' rule:
$$
p \left(d _ {t + 1} \mid i _ {d _ {t + 1}}\right) \propto p \left(i _ {d _ {t + 1}} \mid d _ {t + 1}\right) p \left(d _ {t + 1}\right), \tag {5}
$$
where the likelihood corresponds to the specific inspection model, described by a probability of detection curve, as mentioned before. Since the damage probabilities are discrete, the normalisation constant can be straightforwardly computed by simply summing the unnormalised bins [13].
To enable efficient computation of the deterioration evolution under correlation, a Gaussian hierarchical structure is adopted [36], in which the crack size probability is defined conditional on a common factor,  $\alpha$  as  $p(d_t|\alpha)$ . In this work, we consider that the initial damage probabilities are equally correlated among components with a Pearson coefficient equal to 0.8.
The damage transitions, in this case, are formulated as:
$$
p \left(d _ {t + 1} \mid \alpha\right) = \sum_ {\tau_ {t + 1}} \sum_ {d _ {t}} p \left(d _ {t + 1} \mid d _ {t}, \tau_ {t + 1}\right) p \left(d _ {t} \mid \alpha\right) p \left(\tau_ {t + 1}\right). \tag {6}
$$
Once an inspection outcome is available, the common correlation factor is also updated based on the new information, thus influencing all components. The likelihood of collecting one inspection indication given  $\alpha$  can be computed as:
$$
p \left(i _ {d _ {t + 1}} \mid \alpha\right) = \sum_ {d _ {t + 1}} \left[ p \left(d _ {t + 1} \mid \alpha\right) p \left(i _ {d _ {t + 1}} \mid d _ {t + 1}\right) \right], \tag {7}
$$
and the correlation factor can then be updated:
$$
p (\alpha | i _ {d _ {t + 1}}) \propto p (\alpha) p (i _ {d _ {t + 1}} | \alpha). \tag {8}
$$
Finally, the marginal damage probabilities are computed as:
$$
p \left(d _ {t + 1}\right) = \sum_ {\alpha} \left[ p \left(d _ {t + 1} \mid \alpha\right) p (\alpha) \right]. \tag {9}
$$
Table 4: Main options available in IMP-MARL.
Option name	Environment	Dict key	Type
Number of components	struct	n_comp	int
k components	struct	k_comp	int
Correlation	struct	env_corrlation	bool
Campaign cost	struct	campaign_cost	bool
Number of wind turbines	owf	n_comp	int
Number of components per wind turbine	owf	lev	int
Campaign cost	owf	campaign_cost	bool
Table 5: Observations options available in IMP-MARL.
Option name	Env.	Dict key	Type	Dimensionality
Component damage probability	struct	*by default	float	30
Component deterioration rate	struct	obs_d_rate	float	1
All components damage probability	struct	obs_multiple	float	n_comp · 30
All components deterioration rate	struct	obs_all_d_rate	float	n_comp
Correlation condition	struct	obs_alphas	float	80
Component damage condition	owf	*by default	float	60
Component deterioration rate	owf	obs_d_rate	float	1
All components damage condition	owf	obs_multiple	float	n_comp · 60
All components deterioration rate	owf	obs_all_d_rate	float	n_comp
C Options available in IMP-MARL and reward model
In IMP-MARL, the environments can be easily set up with specific options, from the definition of the number of agents to the observation information perceived by the agents and the state information received by mixers/critics. This can be straightforwardly specified through the configuration files provided on IMP-MARL's GitHub repository. These options are in fact parameters included in IMP-MARL's classes. In particular, Table 4 lists the main options available.
In addition to the main options previously mentioned, it is also possible to tailor the information encoded in the observations and states. One can choose which local information,  $o_{t}^{a}$ , the agents will receive, as well as the global information,  $s_{t}$ , available during training. Tables 5 and 6 list all possible options. Since these options are coded as booleans, the selected configuration can be easily defined by assigning a True value. Additional details can be found in the code.
In the experiments conducted in this work, the selected parameters are listed in Table 7. Through the code provided in IMP-MARL, future works may investigate alternative observation and state information options.
C.1 Reward model
The goal of the agents is to maximise the expected sum of discounted rewards,  $\mathbb{E}[R_0] = \mathbb{E}\left[\sum_{t=0}^{T-1} \gamma^t \left[R_{t,f} + \sum_{a=1}^{n} \left(R_{t,ins}^a + R_{t,rep}^a\right) + R_{t,camp}\right]\right]$ , as stated in Section 3.2. The rewards
Table 6: States options available in IMP-MARL.
Option name	Environment	Dict key	Type	Dimensionality
All component damage condition	struct	state_obs	float	n_comp · 30
All components deterioration rate	struct	state_d_rate	float	n_comp
Correlation condition	struct	state_alphas	float	80
All component damage condition	owf	state_obs	float	n_comp · 60
All components deterioration rate	owf	state_d_rate	float	n_comp
Table 7: Options set up in our experiments.
Option name	struct uc	struct_c	owf
state_obs	True	True	True
state_d_rate	True	True	False
state_alphas	False	True	False
obs_d_rate	False	False	False
obs_multiple	False	False	False
obs_all_d_rate	False	False	False
obs_alphas	False	True	False
env(correlation	False	True	False
campaign_cost	True & False	True & False	True & False
Table 8: Rewards specified in our experiments.
Component	Campaign cost	Rins	Rrep	cf	Rcamp
struct	False	-1	-20	-10,000	0
True	-0.2	-20	-10,000	-5
owf upper level	False	-1	-10	-1,000	0
True	-0.2	-10	-1,000	-5
owf middle level	False	-4	-30	-1,000	0
True	-1	-30	-1,000	-5
collected at each time step may include inspection  $R_{ins}$  and repair  $R_{rep}$  costs for all considered components, along with the system failure risk, which is defined as the system failure probability  $p_{f_{sys}}$  multiplied by the associated consequences of a failure event  $c_f$ , formulated as  $R_f = p_{f_{sys}} \cdot c_f$ . Additionally, a campaign cost  $R_{camp}$  may also be included if that option is active. The discount factor is defined as  $\gamma = 0.95$  in our experiments and the specific rewards are listed in Table 8.
D Cooperative MARL methods
We considered methods from two families of algorithms in MARL: value-based and policy-based. Here, we propose a brief description of these methods, starting from single-agent RL (SARL) definitions to MARL. However, for more detailed information, we refer the reader to the original papers. Furthermore, in addition to the definition of Dec-POMDP in Section 3.1, we define the value function, also  $V$  function, which evaluates the current joint policy  $V^{\pi}(s) = \mathbb{E}[R_t|s_t = s,\pi ]$ . Another function of interest is the state-joint-action value function  $Q_{tot} = Q^{\pi}(s,u) = \mathbb{E}[R_t|s_t = s,u_t = u]$ , also called  $Q$  function. The individual  $Q$  function is defined by  $Q^{\pi ,a}(s,u) = \mathbb{E}[R_t|s_t = s,u_t^a = u,\pi ]$  or  $Q_{a}$  for short. The reward is common to all agents and implies that  $Q_{tot} = Q_{a}\forall a$ . The advantage function is defined as  $A(s,u) = Q(s,u) - V(s)$ .
Value-based methods aim to learn the optimal  $Q$  function defined as  $Q^{\pi^{*}}(s,u) = \max_{\pi}Q^{\pi}(s,u)$ . This enables the agent to greedily select the action  $\pi^{*}(s) = \operatorname{argmax}_{u}Q^{\pi^{*}}(s,u)$ . In SARL, this is accomplished through Q-learning [54], originally designed for tabular settings. However, as the size of the state-action space increases, it becomes impractical to compute  $Q$  for each state-action pair. A solution, named DQN [1], approximates  $Q$  with a neural network  $\theta$  and learn  $Q(s,u;\theta)$  by minimising the loss  $\mathcal{L}(\theta) = \mathbb{E}_{\langle .\rangle \sim B}\left[\left(r_t + \gamma \max_{u\in \mathcal{U}}Q(s_{t + 1},u;\theta ') - Q(s_t,u_t;\theta)\right)^2\right]$  where  $B$  is a replay buffer composed of transitions  $\langle s_t,u_t,r_t,s_{t + 1}\rangle$  and  $\theta^\prime$  is the target network, a copy of  $\theta$  updated periodically. This approach can train a centralised learner in a Dec-POMDP if all agents can access the state  $s$  during execution, resulting in the learning of  $Q_{tot} = Q^{\pi}(s,\pmb {u})$ . Issues are that the joint action space scales exponentially with  $n$ , and in practice, agents select their action based only on their history of observation  $(o,\tau)$  and not the state  $s$ . There is a decentralised solution, named IQL [20], which consists in learning independently  $Q_{a}$ . However, there are also CTDE methods that take the state  $s$  into account during training. A solution proposed in QMIX [16] is to approximate  $Q_{tot}$  as a function of all  $Q_{a}$  and  $s$  during training. Agents select actions based on their  $Q_{a}$ , which are now utility functions that factorise  $Q_{tot}$  and not  $Q$  function. One condition is that individual
$Q_{a}$  satisfy the individual global max (IGM):  $\arg \max_{\mathbf{u}_t} Q(s_t, \mathbf{u}_t) = \bigcup_a \arg \max_{u_t^a} Q_a(\tau_t^a, u_t^a)$  [55]. In QMIX, the factorisation is achieved by a constrained hypernetwork [56] which links  $s$  and  $Q_{a}$  to  $Q_{tot}$ . QVMix [17] extends QMIX with the Deep Quality-Value method [57, 58], learning both  $V$  and  $Q$ , using the former as the target of the latter. QPLEX [18] extends QMIX with the dueling structure  $Q = V + A$  [59], learning a factorisation of  $V$  and  $A$  with transformers [60]. We selected these methods based on their IGM consistency, code availability, and results in the literature.
Policy-based methods learn directly the optimal policy through a neural network  $\pi(s, u; \theta)$  that maximises  $J(\theta) = \mathbb{E}_{\pi_{\theta}}[R_0]$ . The well-known REINFORCE method [61] ascends the gradient  $\nabla_{\theta} J = \mathbb{E}[\sum_t R_t \nabla_{\theta} \log \pi(u_t | s_t; \theta)]$  to find  $\pi^*$ . Actor-critic methods [62, 63] expand upon this method by incorporating a parameterised critic that estimates  $Q(s_t, u_t; \phi)$ , replacing  $R_t$ , with the actor serving as the parameterised policy. To reduce variance, a baseline  $b(s)$  is injected into the gradient, usually  $b(s) = V(s)$ , and  $Q(s, u; \phi)$  is replaced by  $A(s, u; \phi)$  [64], leading to the new gradient expression  $\nabla_{\theta} J = \mathbb{E}[\sum_t A(s_t, u_t; \phi) \nabla_{\theta} \log \pi(s_t, u_t; \theta)]$ . Advantage estimation is accomplished either by  $A(s_t, u_t; \phi) = Q(s_t, u_t; \phi) - \sum_u \pi(u | s_t; \theta) Q(s_t, u; \phi)$  or by  $A(s_t, u_t; \phi) = r_t + \gamma V(s_{t+1}; \phi) - V(s_t; \phi)$ . Extending these methods to MARL is a straightforward process and the decentralised solution is named IAC [19]. This approach involves each agent learning independently an actor and a critic, based only on the tuple  $(\tau, o)$ . However, this solution does not exploit the additional information provided by the state  $s$ . During training, the critic may exploit the full state  $s$ , which would result in a centralised critic. However, this approach provides the same feedback to all agents, missing out on the crucial aspect of credit assignment [65]. To address this, MADDPG [8] allows each agent to learn its own critic  $Q_a(s, u; \phi)$  that is considered centralised since its use of  $s$  and  $u$ . On the other hand, COMA [19] and FACMAC [11] propose solutions with a single centralised critic. FACMAC suggests using a central but factored critic by employing the value function factorization of QMIX. The joint-action  $Q(s, u; \phi)$  is built as a function of  $Q_a(\tau, u^a)$ , without the need to satisfy IGM and without the constraints on the hypernetwork. COMA, inspired by difference reward [66], proposes having the centralised critic compute a counterfactual baseline for each agent. For an agent  $a$ , the difference reward is  $R(s_{t+1}, s_t, u_t) - R(s_{t+1}, s_t, (u_t^{-a}, c_t^a))$  where  $c$  is a default action. Computing this requires simulating the environment steps several times. But in COMA, the centralised critic computes the advantage  $A_a(s_t, u_t; \phi) = Q(s_t, u; \phi) - \sum_{u'^a} \pi(u'^a | \tau_t^a; \theta) Q(s_t, (u_t^{-a}, u'^a); \phi)$  for each agent, allowing it to approximate  $A$  without more environment steps. This has the cost of requiring to increase the input space of  $A$  as it additionally takes  $u^{-a}$  actions as input.
Other methods In addition to QMIX [16], QVMix [17] and QPLEX [18], QTRAN [55] and Weighted-QMIX [67] factorise  $Q_{tot}$  differently from QMIX, but do not always satisfy IGM [55]. Other methods, such as MAVEN [68] and LAN [69], also extend over QMIX. The first improves exploration capabilities while the second learns to cooperate without factorising  $Q_{tot}$ . There are also policy-based methods that rely on the actor-critic paradigm, such as COMA [19] and FACMAC [11] with a single centralised critic. MADDPG [8] is another well-established method, which does not learn a single centralised critic, but one per agent and is designed for continuous action spaces. Another method, LIIR [70], aims to provide credit assessment with individual intrinsic rewards, while HATPRO and HAPPO [71] demonstrate that popular actor-critic methods like TRPO [72] and PPO [73] can be extended to cooperative MARL tasks. HATRPO and HAPPO could have been a great addition to our study but are unfortunately not implemented within the PyMarl library.
Another approach for dealing with cooperative multi-agent settings is to link the recent success of sequence models and reinforcement learning by using a multi-agent transformer (MAT) that learns to transform a sequence of observations into a sequence of actions, one per agent [74].
E.1 Description of the parameters set up in the experiments
In this section, we provide the information required to reproduce the results reported in this paper. Since the neural networks are trained via MARL using PyMarl's [9] library, the parameters are here described following PyMarl's convention. However, their purpose can be easily deduced from the names themselves. The experimental parameters set up equal across all experiments are presented in Table 9, while the parameters specific to each method are hereafter detailed. Note that in Table 9, some parameters are not used by all methods, e.g., RMS parameters. Besides, target update intervals,
Table 9: Parameters set in our experiments.
Parameter name	Parameters value	Parameter name	Parameters value
γ	0.95	Time max	2,050,000
Target update	200	RMS epsilon	10-5
RMS alpha	0.99	Grad norm clip	10
Learning rate	0.0005	Obs last action	True
Agent network	[] - 64 GRU - []	Save model interval	20,000
Table 10: Exploration parameters.
Epsilon start	Epsilon finish	Epsilon anneal time
Value-base methods	0.5	.01	5000
FACMAC	0.3	0.005	50000
buffer size, and batch size are specified based on the number of episodes. When the number of agents increases, we only augment the number of trainable parameters of the mixer/critic networks, while the actor networks and other parameters are not modified. All the parameters can be found in the configuration files available on the GitHub repository and can be used to launch any of the experiments conducted in this paper.
Regarding the agent network representation for CTDE methods and IQL, it consists of one GRU layer with a hidden state that includes 64 features. This means that the input is fully connected to the 64 hidden states, which are then fully connected to the outputs, one per action. We represent it as "[ ] - 64 GRU - ["]. Since the number of actions, and hence the number of outputs, of the DQN network is  $3^n$ , a network with more representation capacity is needed. In that case, linear layers, whose number of output features are specified between brackets, are also included in the agent network surrounding the GRU layer. With  $n = 2$  or  $n = 3$  agents, the network is set as "[128] - 128 GRU - [128,64]" while for  $n = 4$  or  $n = 5$  agents, it is set as "[256] - 256 GRU - [256,256]". Note that the linear layers before the GRUs include a Relu activation function and the last taken action is also added to the observation of the agents as an additional input.
As mentioned previously, some parameters are common to almost all methods. For instance, the optimiser selected is RMSProp for all methods, except for FACMAC, which is trained with ADAM. In nearly all methods, the buffer stores the latest 2,000 episodes, and at each episode, 64 episodes are sampled to update the network. However, since COMA is an on-policy method, the networks are updated with the episodes just played. Therefore, in our experiments, COMA's networks are updated every four episodes based on these last experienced ones. This update is performed four times to ensure a fair amount of network updates with respect to the other methods, which are, in turn, updated every episode.
During training, the value-based methods rely on an epsilon-greedy policy, whose parameters are specified in Table 10, while they act following a greedy policy at testing. Note, however, that COMA and FACMAC utilise different training policies. FACMAC samples discrete actions through a Gumbel softmax for its actor, whereas exploration is performed via an epsilon, whose values are specified in Table 10. On the other hand, COMA follows a classic stochastic policy during training. At the testing stage, FACMAC and COMA select actions adhering to a greedy approach, selecting the action associated with the maximum probability.
In the conducted experiments, the parameters set up in the environments differ with respect to the number of agents and we distinguish three cases (i)  $n < = 10$ , (ii)  $n = 50$ , and (iii)  $n = 100$ . Starting with QMIX, the parameters are all the same when increasing  $n$ , except for the architecture of the mixing network, whose embedding size in the middle of the mixer is: (i) 32, (ii) 64, and (iii) 128. QMIX relies on a double-Q feature, i.e., the loss computed to update  $\theta$  differs from the original. While in the original loss function, the target Q value used for the update is selected with the action that maximises the target Q value parameterised by  $\theta'$ , in double-Q, the action is the one that maximises the Q value parameterised by  $\theta$ . Therefore, we have:  $\mathcal{L}(\theta) = \mathbb{E}_{\langle .\rangle \sim B}\bigl [(r_t + \gamma Q(s_{t + 1},u*;\theta') - Q(s_t,u_t;\theta))^2\bigr]$  where  $u* = \operatorname{argmax}_u Q(s_{t + 1},u;\theta)$ . This target network is updated every 200
Table 11: Number of trainable parameters in uncorrelated k-out-of-n systems.
Method	Network	n = 3	n = 5	n = 10	n = 50	n = 100
QMIX	Agent	27,587	27,715	28,035	30,595	33,795
Mixer	18,657	41,249	133,569	5,430,657	42,202,113
QVMix	Agent	27,587	27,715	28,035	30,595	33,795
Mixer	64,771	110,083	295,043	10,891,779	84,437,891
QPLEX	Agent	27,587	27,715	28,035	30,595	33,795
Mixer	58,249	83,289	155,269	1,830,933	4,901,797
COMA	Agent	27,587	27,715	28,035	30,595	33,795
Critic	35,971	45,955	70,915	1,195,907	10,840,323
FACMAC	Agent	27,587	27,715	28,035	30,595	33,795
Critic	48,002	72,706	134,466	1,042,370	3,313,218
IQL	Agent	27,587	27,715	28,035	30,595	33,795
/	0	0	0	0	0
DQN	Agent	157,979	758,003	/	/	/
/	0	0	/	/	/
episodes. QVMix is a variant of QMIX and the parameter values are similarly specified. In particular, QVMix contains two networks: (i) a Q network with the same architecture as QMIX, and (ii) a V network that is a copy of the Q network, but with only one output. As for QPLEX, we try to be close to the parameters selected for the SMAC experiments they conduct in their paper. When  $n$  increases, we change only the attention layer size composed of  $a$  layers and  $b$  heads. In particular, we set up (i)  $1L4H$ , (ii)  $2L4H$ , and (iii)  $2L10H$ . The mixer embedding is 64 for all experiments.
With respect to COMA, which is an actor-critic method, a few specific parameters should be additionally set up. The agent network, here denoted as the actor, is the same as for the other previously described. However, the critic varies with  $n$  because the input of the critic becomes rather large, and its architecture is fully linear: (i) [128, 128], (ii) [512, 256, 128, 128], and (iii) [2048, 1024, 512, 256, 128]. A TD-λ used to update the critic is set to 0.8 and the learning rate to train the critic is the same as the one of the actor, specified in Table 9. In terms of FACMAC, the parameters of interest are those at the critic, as its size increases with the number of agents  $n$ . FACMAC features a 2-layer-mixing network and, therefore, we specify the size for both: (i) 64-64, (ii) 128-64, and (iii) 128-128. As in COMA, the TD-λ parameter is set to 0.8. The critic has a target network, similar to those included in value-based methods, that is also updated every 200 episodes with a soft target update with  $\tau = 0.001$ . Moreover, we follow the optimiser selection made in FACMAC's original paper and used ADAM, with an epsilon equal to  $10^{-8}$ .
Regarding IQL, which is a decentralised method, the parameters are identical in all tested environments. IQL also has the double-Q feature activated and learns independently individual Q values via DQN's algorithm. For DQN, we only run experiments with less than five agents, i.e.,  $n <= 5$ . The main difference between tested environments is the size of their networks. Since DQN features  $3^n$  outputs, only 64 GRU cells are not enough in terms of network capacity. In our experiments, we increase the network size and confirm that DQN manages to achieve similar results as the other MARL methods. In DQN, the network architecture is the only parameter that is adjusted with the number of agents  $n$ .
Finally, we list in Table 11 the number of trainable parameters of the networks. The input is slightly different between the tested sets of IMP-MARL environments and, therefore, we show in the table only the parameters for one of them. Note that there is an agent network taking actions for all agents and we purposely duplicate the agent network row to emphasise that all agent's networks are identical across experiments.
E.2 Hardware and experiments duration
Our experiments are all run on different clusters managed by SLURM [75]. They are executed with specific hardware requirements based on the number of agents: experiments with up to 10 agents are run on only CPUs, while we execute experiments on GPUs with 50 and 100 agents. The efficiency does not substantially improve when running experiments with less than 10 agents on
Table 12: Hardware configurations for training and testing experiments.
Parameters	Train only on CPUs	Train on GPUs	Test only on CPUs	Test on GPUs
Number of CPU	4	2	8	5
RAM	5 Gb	6 Gb	5Gb	10 Gb
Figure 4: Training duration for experiments with  $n < = 10$  performed on only CPUs.