SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In an extensive benchmark campaign, we test seven RL methods: one fully centralised, one fully decentralised, and five CTDE approaches. The centralised controller, which has an action space that scales exponentially with the number of agents, is trained with the fully centralised method DQN [1] and is the only method taking  $s_t$  as input. Furthermore, the fully decentralised method we test is IQL [20], in which all agents are independently trained. Regarding the five CTDE methods, we investigate three value-based methods: QMIX [16], QVMix [17], and QPLEX [18]. They factorise the value function during training, allowing agents to independently select actions during execution after a centralised value function is jointly learnt during training. The last two are the CTDE actor-critic methods COMA [19] and FACMAC [11]. They train independent policy networks and rely on a single centralised critic during training. Appendix D provides a detailed description of each method as well as a discussion of other methods of interest that are not included in the benchmark. We selected these methods for our benchmark study because they are well established and their implementations are open-sourced and available within the PyMarl framework [9].
All investigated MARL methods are compared against a representative baseline in the reliability engineering community [21, 36]. This baseline, referred to as expert-based heuristic policy, consists of a set of heuristic decision rules that are defined based on expert knowledge. The heuristic policy includes both parametric and non-parametric rules. Parametric decision rules depend on two parameters: (i) the inspection interval and (ii) the number of inspected components. On the other hand, non-parametric rules involve taking a repair action after detecting a crack and prioritising component inspections with higher failure probability. To determine the best heuristic policy, and for each environment, we evaluate all parametric rule combinations over 500 policy realisations, thereby identifying the heuristic policy that maximises the expected sum of discounted rewards among all policies evaluated.
Table 1: Number of agents specified in all investigated IMP environments.
IMP environments	Number of agents
k-out-of-n system	3	5	10	50	100
Correlated k-out-of-n system	3	5	10	50	100
Offshore wind farm	2	4	10	50	100
4.2 Experimental setup
The above-mentioned seven MARL methods are tested in the three sets of IMP environments defined in Section 3.3. The environments differ by the number of agents and by whether or not they include a campaign cost model. The numbers of agents tested in the six types of environments are presented in Table 1. To objectively interpret the variance associated with the examined MARL methods, 10 training realisations with different seeds are executed in each environment. As explained in Section 3, an agent makes decisions based on its local damage probability, the current normalised time step, and sometimes correlation information is additionally provided; while the state, used by DQN and CTDE methods, encompasses all of the information combined. In all cases, the action space features three possible discrete actions per agent, except for DQN, where the centralised controller selects an action among the  $3^n$  possible combinations. For complexity reasons, we only test DQN in k-out-of-n environments featuring 3 and 5 components, as well as in environments with 1 and 2 wind turbines. Detailed information on rewards, observations, and states can be found in Appendix C.
Given the importance of hyperparameters on the performance of RL methods [49], we initially selected their values reported by the original authors. In an attempt to objectively compare the examined methods, parameters that play the same role across methods are equal. Notably, the learning rate and gamma, among others, are identical in all experiments. The controller agent network features the same architecture in all methods, consisting of a single GRU layer with a hidden state composed of 64 features encapsulated between fully-connected layers and three outputs, one per action, except for DQN, where the network output includes  $3^{n}$  actions. In our case, DQN's architecture includes additional fully-connected layers and a larger size of hidden GRU states. Moreover, following common practice, agent networks are shared among agents, and thus a single agent network is trained. Specifically, we train only one network that is used for all agents, instead of training  $n$  distinct agent networks. The training process with a single agent network improves data efficiency because the same episode can be used to perform  $n$  backpropagations through the same agent network, using  $n$  different observations. In contrast, only one backpropagation per agent network would be possible with a single episode if training is performed with  $n$  different agent networks. To allow diversity in agents' behaviour, a one-hot encoded vector is also added to the input of this common network to indicate which one of the  $n$  agents is making the decision. In CTDE methods, critics or mixers are also incorporated at the training stage with specific architectures according to each method and environment configuration. In most cases, the neural networks are updated after each played episode based on 64 episodes sampled from the replay buffer, which contains the latest 2,000 episodes. The only exception is COMA, which follows an on-policy approach, where the network parameters are updated every four episodes. For value-based methods, the training episodes are played following an epsilon greedy policy, whereas test episodes are executed with a greedy policy. The epsilon value is initially specified as 1 and linearly decreases to 0.05 after 5,000 time steps. This is different for COMA and FACMAC. Appendix E and the source code list more details and all parameters.
The number of time steps allocated for one training realisation is 2 million time steps for all methods. These 2 million training time steps are executed with training policies, e.g. epsilon greedy policy, saving the networks every 20,000 training time steps. To evaluate them, we execute 10,000 test episodes and obtain the average sum of discounted rewards per episode per saved network. These test episodes are executed with testing policies, e.g. greedy policy. We show in Appendix E.3 that 10,000 test episodes are needed due to the variance induced in the implemented environments. We emphasise that 10 training realisations are executed with different seeds for the same parameter values. All parameters are listed in Appendix E and in the source code.
5 Benchmark results and discussion
The results from the benchmark campaign are presented in Figure 3, showcasing the relative performance of MARL methods with respect to expert-based heuristic policies in terms of their expected sum of discounted rewards. In each boxplot of Figure 3, each of the 10 seeds is represented by its best policy, which achieved the highest average sum of discounted rewards during evaluation. We further explain the connection between learning curves and boxplots in Appendix F, Figure 10. Our analysis relies on relative performance metrics because the optimal policies are not available in the environments investigated. Additionally, the corresponding learning curves and the best-performing policy realisations can be found in Appendix F.
MARL-based strategies outperform expert-based heuristic policies. While heuristic policies provide reasonable infrastructure management planning policies, the majority of the tested MARL methods yield substantially higher expected sum of discounted rewards, yet the variance over identical MARL experiments is still sometimes significant. In environments with no campaign cost, the performance achieved by MARL methods with respect to the baseline differs in configurations with a high number of agents, as shown at the top of Figure 3. In contrast, MARL methods reach better relative results in environments with a high number of agents when the campaign cost model is adopted, as illustrated at the bottom of Figure 3. In general, the superiority of MARL methods with respect to expert-based heuristic policies is justified by the complexity of defining decision rules in high-dimensional multi-component engineering systems, where the sequence of optimal actions is very hard to predict based on engineering judgment [36].
IMP challenges. In correlated k-out-of-n IMP environments, the variance over identical MARL experiments is higher than in the uncorrelated ones, emphasising a specific IMP challenge. Under correlation, inspecting one component also provides information to uninspected components, impacting their damage probability and thus hindering cooperation between MARL agents. Another challenge is imposed in offshore wind farm environments, where the benefits achieved by MARL methods with respect to the baseline are also reduced in environments with a high number of agents. This can be explained by the fact that each wind turbine is controlled by two agents, being independent of other turbines in terms of rewards. Each agent must then cooperate closely with only one of all agents, hence complicating global cooperation in environments featuring an increasing number of agents.
Campaign cost environments. Yet another challenge can be observed in campaign cost environments under 50 agents, where MARL methods' superior performance with respect to heuristic policies is more limited. The aforementioned environments are challenging for MARL methods because agents should cooperate in order to group component inspection/repair actions together, saving global campaign costs. In addition, the heuristic policies are designed to automatically schedule group inspections, being favourable in this case. This is confirmed by the learning curves presented in Figures 11 and 12. On the other hand, in environments with more than 50 agents, MARL methods substantially outperform heuristic policies. At least one component is inspected or repaired at each time step and the results reflect that avoiding global annual campaign costs becomes less crucial.
Centralised RL methods do not scale with the number of agents. DQN reaches better results than heuristic policies, though achieving lower rewards than CTDE methods in most environments, despite benefiting from larger networks during execution. This highlights the scalability limitations of such centralised methods, mainly due to the fact that they select one action out of each possible combination of component actions.
IMP demands cooperation among agents. The results reveal that CTDE methods clearly outperform IQL in all tested environments, especially those with a high number of agents. This confirms that realistic infrastructure management planning problems demand coordination among component agents. Providing only independent local feedback to each IQL agent during training leads to a lack of coordination in cooperative environments, also shown by Rashid et al. [16]. However, the performance may be improved by enhancing networks' representation capabilities by including more neurons, yet this is true for all investigated methods.
Infrastructure management planning via CTDE methods. Overall, CTDE methods generate more effective IMP policies than the other investigated methods, demonstrating their capabilities for supporting decisions in real-world engineering scenarios. While Figure 3 presents the variance of the best results across runs, the learning curves further confirm this finding in Appendix F. In particular, QMIX and QVMIX generally learn effective policies with low variability over runs. Slightly more
Figure 3: Performance reached by MARL methods in terms of normalised discounted rewards with respect to expert-based heuristic policies in all IMP environments, H referring to the heuristics result. Every boxplot gathers the best policies from each of 10 executed training realisations, indicating the 25th-75th percentile range, median, minimum, and maximum obtained results. The coloured boxplots are grouped per method, vertically arranging environments with an increasing number of  $n$  agents, as indicated in the top-left legend boxes. Note that the results are clipped at -100%.
unstable, QPLEX also yields similar results to QMIX and QVMIX in terms of achieved results. While being able to outperform heuristic policies in almost every environment, FACMAC exhibits a high variance among runs. However, FACMAC effectively scales up with the number of agents and environment complexity (as reported in [11]), achieving some of the best results in IMP environments with over 50 agents as well as in correlated IMP environments. The results also suggest that COMA is the least scalable MARL method in our benchmark. This can be attributed to the fact that the computation of the critic's counterfactual becomes challenging with an increasing number of agents.
This work offers an open-source suite of environments for testing scalable cooperative multi-agent reinforcement learning methods toward the efficient generation of infrastructure management planning (IMP) policies. Through our publicly available code repository, we also encourage the implementation of additional IMP environments, e.g., bridges, transportation networks, pipelines, and other relevant engineering systems, whereby specific disciplinary challenges can be identified in a common simulation framework. Based on the reported benchmark results, we can conclude that centralised training with decentralised execution methods are able to generate very effective infrastructure management policies in real-world engineering scenarios. While the results reveal that MARL methods outperform expert-based heuristic policies, additional research efforts should still be devoted to the development of scalable cooperative MARL methods. While we model the IMP decision-making problem as a Dec-POMDP, modelling IMP problems as mean-field games [50] is a promising direction to be considered in environments with an increasing number of agents. Moreover, specific improvements are still required in environments where a global cost is triggered from the actions taken by any local agent, e.g., global campaign cost. Besides, more stable training is still needed in environments where local information perceived by one agent can influence the damage condition probabilities of others, as in the correlated IMP environments. In the future, more realistic and challenging environments for cooperative MARL methods could be investigated. One example would be assigning campaign costs to specific groups of components, instead of specifying only one global campaign cost.
The authors acknowledge the computational resources provided by the Consortium des Equipements de Calcul Intensif (CECI), funded by the Fonds de la Recherche Scientifique de Belgique (F.R.S.-FNRS) under Grant No. 2.5020.11 and by the Walloon Region. We further acknowledge the insightful comments provided by our colleagues Adrien Bolland, Victor Dachet, Nandar Hlaing, Gaspard Lambrechts, Gilles Louppe, Matthias Pirlet, and Maurizio Vassallo.
A IMP-MARL public repository, license, data, and documentation
The new assets we provide in this paper are listed hereafter. IMP-MARL's suite of environments is publicly available on the GitHub repository https://github.com/moratodpg/imp_marl, featuring an open-source Apache v2 license. Moreover, IMP-MARL contains wrappers to facilitate its implementation on typical MARL ecosystems, i.e., Gym [42], RLLib [45], and PyMarl [9], as detailed in Section 4.
To reproduce the work reported in this paper, the following process can be executed: (i) cloning the repository, (ii) installing a virtual environment with the package requirements, and (iii) executing the script instructions of the corresponding method and IMP-MARL environments. In addition to the instructions for reproducing our results, we also provide tutorials to add new environments or wrappers.
We also provide the data files resulting from our experiments, enabling the reproduction of any reported result without re-running the experiments, as well as the corresponding implementation code, hence facilitating future cross-comparisons. The readily available results include Figures 3, 11, and 12, along with Tables 13, 14, 15.
Configuration, execution, and results files are permanently stored at Zenodo, accessible via https:// zenodo.org/record/8032339. Additionally, the controller networks' weights of the best policies presented in Figure 3 are also stored there, thus fostering further interpretability studies of MARL-based strategies. The dataset is open-access and registered with the Digital Object Identifier (DOI) 10.5281/zenodo.8032339. More information and dedicated tutorials can be found on the repository.
B Modelling infrastructure management in IMP-MARL
In this appendix, we thoroughly describe the deterioration, inspection, and transition models implemented in this work. These models drive the dynamics of the IMP-MARL environments provided in this paper. Based on this information, one can easily learn how to create new environments.
B.1 Deterioration models
The deterioration processes introduced here specifically correspond to fatigue deterioration mechanisms, yet corrosion, erosion, and many other practical infrastructure management problems can be similarly modelled.
Correlated and uncorrelated k-out-of-n systems Throughout the text and the code, the set of environments related to correlated and uncorrelated k-out-of-n systems are abbreviated as struct_c and struct uc, respectively, or denoted as struct when referring to both of them. In the k-out-of-n environments currently included in IMP-MARL, the structural components are exposed to fatigue deterioration, and unless a repair is undertaken, the crack size  $d_{t}$  (i.e., damage condition) evolves over time  $t$  as [51]:
$$
d _ {t + 1} = \left[ \left(1 - \frac {m}{2}\right) C _ {F M} S _ {R} ^ {m} \pi^ {m / 2} n _ {S} + d _ {t} ^ {1 - m / 2} \right] ^ {2 / (2 - m)}, \tag {1}
$$
where  $\ln (C_{FM})\sim \mathcal{N}(\mu = -35.2,\sigma = 0.5)$  and  $m = 3.5$  stand for material variables, which directly influence the crack growth. Due to environmental and operational conditions, the components are subject to a dynamic load characterised by the stress range,  $S_{R}\sim \mathcal{N}(\mu = 70,\sigma = 10\mathrm{N / mm}^{2})$  over  $n_S = 10^6$  annual stress cycles, i.e., the number of load cycles experienced by the structural component in one year. At the initial step or after a component is repaired, the initial crack size is at its intact condition, defined by its initial distribution  $d_0\sim \mathrm{Exp}(\mu = 1\mathrm{mm})$ , and a component level failure occurs when the crack size exceeds a critical size of  $d_c = 20\mathrm{mm}$ . The component failure probability  $p_F$ , defined as  $p_F = p[g\leq 0]$ , can be computed following a through-thickness failure criterion [52], where the failure limit at time step  $t$  is formulated as  $g_{t} = d_{c} - d_{t}$ . At the system level, a failure event occurs if  $k$  (out of  $n$ ) components fail, and its corresponding system failure probability,  $p_{F_{sys}}$ , can be efficiently computed as a function of all components failure probabilities, as proposed in [40].
The continuous crack size is discretised into a certain number of discrete bins in order to enable efficient Bayesian inference once inspection indications are available. Further details can be found in
Table 2: Description of the discretisation scheme implemented.
Environment	Variable	Interval boundaries	Bins
struct	dt	[0, exp{ln(10-4): (ln(dc) - ln(10-4))/28: ln(dc)},∞]	30
owf	dt	[0, d0: (dc - d0)/(60 - 2): dc,∞]	60
Table 3: Variables specified in the offshore wind farm deterioration models.
Upper component	Middle component	Mudline component
ln(CFM)	μ = -26.45	μ = -26.04	μ = -26.12
σ = 0.12	σ = 0.4	σ = 0.39
m	3	3	3
q	μ = 10.21	μ = 7.40	μ = 6.74
CoV = 25%	CoV = 25%	CoV = 25%
dc	20	60	60
nS	5,049,216	5,049,216	5,049,216
[13]. If the initial crack size among components is correlated (i.e., we are dealing with a correlated k-out-of-n system), the damage condition of each component is defined conditional on a common correlation factor,  $\alpha$ , via a Gaussian hierarchical structure [36]. In that case, the discretised damage bins should be defined conditional on the correlation factor. The specific discretisation implemented in our environments is defined in Table 2.
Offshore wind farm In this set of environments, a group of n_comp offshore wind substructures is considered, in which three representative structural components are modelled at different locations of the wind turbine: (i) at the atmospheric zone - upper level, (ii) at the splash zone - middle level, (iii) below the seabed - mudline. The deterioration, inspection, and cost models hence differ for each of the three considered components. While the fatigue deterioration is calculated according to Eq. 1, the expected dynamic load,  $S_{r}$ , is in this case defined based on industrial standards [53], as:
$$
S _ {r} = q \Gamma (1 + 1 / \lambda) Y, \tag {2}
$$
corresponding to the expected value of a Weibull distribution defined by the scale parameters listed in Table 3,  $q \sim \mathcal{N}$ , and shape factor,  $\lambda = 0.8$ , weighted by a geometric parameter,  $Y \sim \mathcal{LN}(\mu = 0.1, \sigma = 0.1)$ . The initial crack size distribution is specified for all wind turbine components as  $d_0 \sim \mathrm{Exp}(\mu = 0.11)$  and the remaining specific fatigue variables associated with each wind turbine component are listed in Table 3. At the wind turbine level, the failure event occurs if one component of the wind turbine fails. The wind turbine failure risk is then defined as the wind turbine failure probability multiplied by the consequences associated with a wind turbine failure event. At the wind farm level, the damage condition of a wind turbine does not influence the condition of the other wind turbines, and the wind farm system failure risk is defined as the sum of all turbines' failure risk.
B.2 Inspection models
The inspection models implemented in IMP-MARL are hereafter described, defining the likelihood of retrieving a certain inspection outcome as a function of the damage size.
Correlated and uncorrelated k-out-of-n systems The inspection model is normally characterised depending on the accuracy of the measurement instrument, formally specified through probability of detection (PoD) curves, in which the probability of observing a crack is defined as a function of the crack size [13]. In this case, the inspection model is described by an exponential distribution  $p(i_{d_t}|d_t)\sim \mathrm{Exp}(\mu = 8)$ , defining the probability of observing a crack during an inspection.
Offshore wind farm In this more practical set of environments, an eddy current inspection technique is here considered, whose PoD can be modelled according to industrial standards [53], as:
$$
p \left(i _ {d _ {t}} \mid d _ {t}\right) = 1 - \frac {1}{1 + \left(d _ {t} / \chi\right) ^ {b}}, \tag {3}
$$
where the factors  $\chi$  and  $b$  are specified as 0.4 and 1.43, respectively, for the upper component, but considered as 1.16 and 0.90, for the middle component. Naturally, less accurate inspection outcomes
can be expected for the middle component, as it is located in a region below the water level, where the visibility is reduced.
B.3 Transition models
An overview of the transition model is explained hereafter. For a more detailed description, we refer the reader to [36]. Since the crack size is discretised in this work, the transition and inspection models can be stored in tables. In our code, they are encoded in Numpy files, which are stored in the repository folder pomdp_models. In particular, the files are named Dr3031C10 .npz, Dr3031_H08 .npz, and owf6021 .npz, for k-out-of-n system, correlated k-out-of-n system, and offshore wind farm, respectively. By relying on already stored transition and inspection models, the environments can be simulated efficiently. Alternatively, the crack size evolution could also be directly computed at execution time, yet an additional computational expense would be then incurred.