IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARL
Authors: Pascal Leroy
Pablo G. Morato
Jonathan Pisane
Athanasios Kolios
Damien Ernst
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: cooperative multi-agent reinforcement learning
infrastructure management planning
centralised training decentralised execution
benchmark

Abstract: We introduce IMP-MARL, an open-source suite of multi-agent reinforcement learning (MARL) environments for large-scale Infrastructure Management Planning (IMP), offering a platform for benchmarking the scalability of cooperative MARL methods in real-world engineering applications.
In IMP, a multi-component engineering system is subject to a risk of failure due to its components' damage condition.
Specifically, each agent plans inspections and repairs for a specific system component, aiming to minimise maintenance costs while cooperating to minimise system failure risk.
With IMP-MARL, we release several environments including one related to offshore wind structural systems, in an effort to meet today's needs to improve management strategies to support sustainable and reliable energy systems.
Supported by IMP practical engineering environments featuring up to 100 agents, we conduct a benchmark campaign, where the scalability and performance of state-of-the-art cooperative MARL methods are compared against expert-based heuristic policies. 
The results reveal that centralised training with decentralised execution methods scale better with the number of agents than fully centralised or decentralised RL approaches, while also outperforming expert-based heuristic policies in most IMP environments.
Based on our findings, we additionally outline remaining cooperation and scalability challenges that future MARL methods should still address.
Through IMP-MARL, we encourage the implementation of new environments and the further development of MARL methods.

IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARL
*Pascal Leroy'f, *Pablo Morato Jonathan Pisane', Athanasios Kolios?, Damien Ernst
Montefiore Institute, University of Liège, 'Technical University of Denmark, Thales Belgium *LTCI, Telecom Paris, Institut Polytechnique de Paris fpleroy@uliege be, "pgmdo8dtu.c dk Authors contributed equally
Abstract
We introduce IMP-MARL, open-source suite of multi-agent reinforcement learning (MARL) environments for large-scale Infrastructure Management Plan- ning (IMP), offering platform for benchmarking the scalability of cooperative MARL methods real-world engineering applications. In IMP, a multi-component engineering system is subject risk of failure due its components' damage con- dition. Specifically, each agent plans inspections and repairs for a specific system component, aiming to minimise maintenance costs while cooperating to minimise system failure risk. With IMP-MARS we release several environments including one related to offshore wind structural systems, in an effort to meet today's 's needs improve management strategies to support sustainable and reliable energy systems. Supported by IMP practical engineering environments featuring up 100 agents, we conduct a benchmark campaign, where the scalability and performance of state-of-the-an cooperative MARL methods are compared against expert-base based heuristic policies. The results reveal that centralised training with decentralised execution methods scale better with the number agents than fully centralised decentralised RL approaches, while also outperforming expert-based heuristic policies in most IMP environments. Based on our findings, we additionally out- line remaining cooperation and scalability challenges that future MARL methods should still address. Through IMP-MARL, we encourage the implementation new environments and the further development of MARL methods.
Introduction
Intelligent agents trained with reinforcement learning (RL) have proven successful in solving com- plex decision-making tasks, e.g., games [133, autonomous driving 4 (). human healthcare [6]. nuclear fusion [7], among others. RL training approaches where multiple agents interact together are commonly denoted as multi-agent reinforcement learning (MARL) methods. In certain applications, these agents must cooperate to accomplish a common goal, leading the special case of cooperative MARL. To support the advancement of cooperative MARL methods, multiple environments based on games and simulators have served as benchmark testbeds, e.g., the particle environment [8) StarCraft Multi-Agent Challenge (SMAC) 9 10]. and MaMuloCo [II.. Benchmarking on environments based on games and simulators is useful for the development of MARL methods specific collabo- rative/competitive tasks, but additional challenges may still be encountered when deploying MARL
methods real-world applications [12].
Infrastructure Management Planning (IMP) is a contemporary application that responds to current societal and environmental concerns. In IMP, inspections, repairs, and/or retrofits should be timely planned in order to control the risk of potential system failures, e.g., bridge and wind turbine failures,
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

System failure
+= Rf
R+ Rf
Probability
Actions
Q
X
Component
Damage size
+ Reep
R+-0
X
Q X
Component
R + Rins
Q X
Component
- Riss
Figure 1: Overarching representation of an infrastructure management planning (IMP) problem. The system failure risk is defined function of the probability distribution over the components' damage condition. To control the system failure risk, components can be inspected or repaired at each time step and, typically, an agent controls one component. The objective of IMP's problem is the maximisation of the expected sum of discounted rewards, by balancing the system failure risk R against inspections Rins and repairs Rrep, all three being negative rewards. Here, we show three components with the same damage probability at time step l. When a component is not inspected nor repaired, its damage probability evolves according to deterioration process. If component inspected, information from the inspection is also considered when updating the damage probability, and if component is repaired, the damage probability resets to its initial damage distribution.
among many others [13]. Formally, the system failure risk is defined as the system failure probability multiplied by the consequences associated with failure event, typically defined in monetary units. Due to model and measurement uncertainties, the components' damage is not perfectly known, and decisions are made based on a probability distribution over the damage condition, henceforth denoted as damage probability. The system failure probability is defined as function of components' damage probabilities. Starting from its initial damage distribution, each component's damage probability transitions according to deterioration stochastic process, but also according to the decisions made [3]. Naturally, the damage probability transitions based on its deterioration model when the component is neither inspected nor repaired, i.e., do-nothing action. If a component inspected, its damage probability is updated with respect to the inspection outcome. When component is repaired, its damage condition is directly improved and the damage probability resets to its initial damage distribution. schematic typical IMP problem shown in Figure
In an effort to generate more efficient strategies for managing engineering systems through the application of cooperative MARL methods, we introduce IMP-MARL, a novel open-source suite of multi-agent environments. In IMP-MARL, each agent is responsible for managing one constituent component a system, making decisions based on the damage probability of the component. Besides seeking to reduce component inspection and maintenance costs, agents should effectively cooperate to minimise the system failure risk. With IMP-MARL, our goal implementation of new customisable environments. By jointly minimising system failure risks and inspection/maintenacce costs, more effective IMP policies contribute to a better allocation of resources from socictal perspective. Furthermore, additional societal impact is also made by controlling the risk of system failure events. For example, the failure of wind turbine may affect the available electricity production. Beyond economic considerations, our proposed IMP-MARL framework can also be used to include sustainability and societal metrics within the objective function by accounting for those directly in the reward model.
facilitate the definition and
To assess the capability of cooperative MARL methods for generating effective policies for IMP problems involving many components, we additionally benchmark here state-of-the-art cooperative MARL methods terms of scalability and optimality. Most of the benchmarked methods are centralised training with decentralised execution (CTDE) methods [4 [5), in which each agent

---

acts based on only local information, while global information can be utilised during training. Specifically, we benchmark five CTDE methods: QMIX [T6] QVMix [77. QPLEX [8] COMA
[19] and FACMAC [II], along with a decentralised method, i.e., IQI. (2), and centralised one, i.e., DQN M. All tested MARL methods are compared against expert-based heuristic policies, which can be categorised as state-of-the-an method to deal with IMP problems in the reliability engineering community [3 21]. our study, three sets of IMP environments are investigated. including one related to offishoee wind structural systems, where MARL methods are tested with up to 100 agents. Additionally, these environments can be set up with two distinct reward models, and one of them incorporates explicit cooperative objectives. For the sake of enabling the reproduction of any published result, we have made our best effort to ensure that the necessary code is publicly available.
Our contributions can be outlined as follows:
We introduce IMP-MARI,, novel open-source suite of environments, motivating the development of scalable MARL methods as well as the creation of new IMP environments, enabling the effective management of multi-component engineering systems and, as such, leading a positive societal impact.
In an extensive benchmark campaign, we test state-of-the-art cooperative MARL methods in very high-dimensional IMP environments featuring up to 100 agents. The resulting management strategies are evaluated against aapert-based heuristic policies. We publicly provide the source code for reproducing our reported results and for easing direct comparisons with future developments. Based on our results, we draw relevant insights for both machine learning and reliability engi- neering communities, further highlighting important challenges that must still be resolved. While cooperative MARL methods can learn superior strategies compared to expert- heuristic poli- cies, the relative performance benefit decreases in environments with over 50 agents. In certain environments, cooperative MARL policies are characterised by a high variance and sometimes underperform expert-based heuristic policies, suggesting the need for further research efforts.
2 Related work
MARL environments Cooperative MARL has a long-standing history, and decentralised approaches such as IQL were already originally proposed in 1993 (20]. There has been recent interest in the development of CTDE methods [5] (see Section 4.1. inducing the creation of new environments with cooperative tasks. With continuous action spaces, popular environments include the particle environment S a suite of communication oriented environments with cooperative scenarios, and MaMuJoCo which aims at factorising the decision of MuJoCo [22] a physics-based simulator.
In contrast, the StarCraft multi-agent challenge (SMAC) 9 and its upgraded version SMACv2 10 are probably the most studied environments with discrete action spaces. SMAC is based on the StarCraft II Learning Environment (23) with a suite of micro-management challenges where each game unit is an independent agent. Other cooperative environments based on game simulators include the Hanabi Challenge [24]. a "cooperative solitaire" between two and five players, and Google Research Football (25). football game simulator. Cooperative MARL methods are mostly benchmarked on these games and simulators, but also on real-world applications: target coverage control (MATE) 26 train scheduling (Flatland-Rl) -RL.) (27. traffic control (CityFlow) 28 multi-robot warehouse (RWARE) [29 [30]. Oroojlooy and Hajinezhad [12] provide a review of cooperative
MARL, including more detailed list of applications. IMP-MARL introduces two key advancements:
support for environments with up to 100 agents and seamless creation of diverse IMP environments. This enables the utilisation of RL in real-world scenarios, ranging from complex factories with heterogeneous components to offshore wind farms with multiple homogeneous components.
Infrastructure management planning methods Recent heuristic-based inspection and maintenance (I&M) planning methods generate IMP policies based on an optimised set of predefined decision rules 21 BT] By evaluating only a set of decision rules of the entire policy space, the previously mentioned approaches might yield suboptimal policies 13]. In the literature, one can also find POMDP-based methods applied to the I&M planning of engineering components, in most cases, relying on efficient point-based solvers [3 32 33). When dealing with multi- culli-commpoent engineering systems, solving point- POMDPs becomes computationally complex. that case, the policy and value function can be approximated by neural networks, enabling the treatment of high-dimensional engineering systems. Specifically, actor-critic, and value function-based methods have been proposed in the literature for the management of engineering systems (34 with some of them relying
3

---

Note that no soen-source methods nor publicly available environments are provided in the above-mentioned references. This emphasises the importance of our efforts to enhance comparison and reproducibility within the reliability engineering community.
CTDE methods [37 38
3 IMP-MARL: A suite of Infrastructure Management Planning environments
In IMP, the damage condition of multiple components deteriorates stochastically over time, inducing a system failure risk that is penalised at each time step. To control the system failure risk, components can be inspected or repaired, yet, incurring additional costs. The objective is the minimisation of the expected sum of discounted costs, including inspections, repairs, and system failure risk. This can be achieved through the agents' cooperative behaviour, assigning component inspections and repairs while jointly controlling the system failure risk. The introduced IMP decision-making problem can be modelled decentralised partially observable Markov decision process
3.1 Preliminaries
Dec-POMDP 39] can be defined by a tuple [S. 2,4, n. P, where n agents simultaneously
choose an action at every time step t. The state of the environment is st E S where S is the set of states. The observation function O: : S {l, n + Z maps the state to an observation of e Z perceived by agent at time i, where 2 is the observation space. Each agent
{1..., n) selects an action executed,
ug E U2 and the joint action space is L 41 x Un. After the joint action ut the transition function determines the new state with probability P(s++1|$ [s,, u4) : S x u + RR+, and r, = R($ St, Ut) : S" x l + R is the team reward obtained by all agents. An agent's policy is function x" (uG) [[ of) (Z U. - R+, which maps its history Tp E (Z Uo Uaat-1 and its observation of to the probability of taking action uf. The joint policy is denoted by ा = (T (T,,,,, T*). The cumulative discounted reward obtained from time step t over the next T time steps is defined by R2 Er- 7/re+* and y [0, 1) is the discount factor. The goal of agents is find the optimal joint policy that maximises the expected RG during the entire episode: π argmax, E6 TT..
3.2 Environments formulation
States and observations As introduced, each agent in IMP perceives of an observation corresponding to its respective component damage probability and the current time step. Each component damage probability transitions based deterioration model, defined according to physics- based engineering models, e.g., numerical simulators and/or analytical laws (32). The damage probability is also updated based on maintenance decisions, as explained in Section Since the components' damage is not perfectly known, the state of the Dec-POMDP is defined as the collection of all components' damage probabilities along with the current time step: 8 = (ot,,., ().
Actions and rewards Each agent controls a component and collaborates with other agents in order to minimise the system failure risk while minimising local costs associated with individual repair and/or inspection actions. At each time step f, an agent decides un between (i) do-nothing, (ii) inspect, or (iii) repair actions, described in Section T. Both inspection and repair actions incur significant costs, formally included in the Dec-POMDP framework as negative rewards, Rins and Ryep. respectively. Moreover, the system failure risk is defined as R CF PF... where PF.,, is the system failure probability and Op is the associated consequences of a failure event, encompassing economic, environmental, and societal losses. In IMP, we include two reward models. The first is a campaign cost model where a global cost, Reamp. is incurred if at least one component is inspected or repaired, plus surplus, Rins Reep- per inspected/repaired component. This campaign cost explicitly incentivises agents to cooperate. The second is campaign cost model, where the campaign cost is set equal to (i.c., Reamp = 0), and only component inspections and repairs costs are considered. Acting on finite-horizon episodes that span over T' time steps, all agents aim at maximising the expected sum of
discounted rewards [[Ro] = E ETo 7¹ [R. + Σm=1 (Ri + R + Rt.camp]
Real-world data While IMP policies are trained based on simulated data, they policies can then be deployed to applications where real-world streams of data are available. In that case, the damage condition of the components is updated based on collected real-world data, e.g., inspections.

---

system fails
repair inspection
compaign vost
componest
componeent
(a) k-out-of-n system environment.
(b) An offshore wind farm environment.
correlation
comelation
(c) Uncorrelated and correlated initial damage distribution.
(d) A campaign cost environment.
Figure 2: Visual representation of available IMP-MARL environment sets and options. In 2a 4-out-of-5 system fails if 2 component fails. In 2c when the environment is under deterioration correlation, the information collected by inspecting one component also influences uninspected components. In campaign cost environments, a global cost is incurred if any component is inspected and/or repaired plus surplus per inspected/repaireed component.
components fail. In 26 wind turbine fails if any constituent
3.3 IMP-MARL environments
With IMP-MARL, we provide three sets of environments to benchmark cooperative MARL methods. For all three, components are exposed to fatigue deterioration during a unite-horizon episode, inducing the growth of crack over 7' time steps. The first set of environments is named k-out-of-n system and refers to systems for which a system failure occurs if (n-k+1) components fail. Those systems have been widely studied in the reliability engineering community [40]. The second type of generic environment is named correlated k-out-of-n system and is a variation of the first one for which initial components' damage distributions are correlated. The last one is named offsshre wind farm and allows the definition of environments for which group of offshore wind turbines must be maintained. The proposed IMP-MARL environment sets and options are graphically illustrated in Figure and hereafter provide details about these sets of environments. Additionally. the deterioration processes and implementation details are formally described Appendices and C
k-out-of-n system In this set of environments, the components' damage probability distribution, p(d)), defined as vector of 30 bins, with each bin representing a crack size interval. Here, the failure probability of one component is defined as the probability indicated in the last bin. The specificity of k-out-of-n system is that it fails if (n-k+1) components fail, establishing a direct link between the system failure probability and the component failure probabilities. For this first system, the initial damage distribution is statistically independent among components and the time horizon is T = 30 time steps. Since it is finite, we normalise each time step input and we define 8, = (p(d)) ) ..., p(d,), ) t/T) and of = (p(d)) t/T). The interest this system is that, in many practical scenarios, the reliability of an engineering system can be modelled as a k-out-of-n system.
Correlated k-out-of-n system The second of environments is the same the first one previously defined, with the difference that the initial damage distribution is correlated among all components. Therefore, inspecting one component also provides information about other uninspected components, depending on the specified degree of correlation. This setting is particularly challenging when approached from decentralised scheme without providing components' correlation information to individual agents. To further address this issue, and in addition to their 30-bin local damage probability, the agents perceive correlation information aa common to all, which updated based on inspection outcomes collected from all components. We thus have: s; (p(d)).....pp) az t/T)
5

---

and of = (p(d)), at t/T).. This damage correlation structure is inspired by practical engineering applications where initial defects among components are statistically correlated due to the fact that components undergo similar manufacturing processes [3]
Offshore wind farm The third set of environments is different from the previous ones it considers a system with a certain number of wind turbines. Specifically, each wind turbine contains three representative components: (i) the top component located in the atmospheric zone, (ii) the middle component in the underwater zone, and (iii) the mudline component submerged under the seabed. In this case, we consider that the mudline component cannot be inspected nor repaired, as it is installed under the seabed in an inaccessible region and, since only the top and middle components can be inspected or repaired, two agents are assigned for each wind turbine. Furthermore, the damage probability, p(d"), ), is a vector with 60 bins and transitions differently depending on the component location in the wind turbine, as corrosion- indundd effects accelerate deterioration in certain areas. Besides individual component damage models, inspection techniques and their associated costs also depend on the component location: it is cheaper to inspect or repair the top components than the middle one [4]] Moreover, while the mudline component cannot be directly maintained, its damage probability also impacts the failure risk of a wind turbine. In offshore wind farm environments, a wind turbine fails if any of its three constituent components fails, and the overall system failure risk is defined as the sum of all individual wind turbine failure risks. In this case, p(df) is modelled as a 60-bin vector and the time horizon is T 20. In this set of environments St (p(d)) ), p(d))) t/T) and of = (p(df). t/T).
Implementation All defined IMP environments are integrated with well-known MARL ecosystems, i.e. Gym [42] Gymnasium 4% PettingZoo [44] and PyMarl [9. through wrappers. The tested MARL methods are adopted from PyMarl's library, but other libraries are also compatible with our wrappers, e.g., RLlib [45), CleanRL [46]. MARLlib [47), or TorchRL [48) All developments are available on public GitHub repository, wttp::ttt//////tt thub. com/moratodpg haapl//g//mmmmmaaa_mmm marl featuring an open-source Apache v2 license.
Benchmark campaign of MARL methods
4.1 Tested methods
In an extensive benchmark campaign, we test seven RL methods: one fully centralised, one fully decentralised, and five CTDE approaches. The centralised controller, which has an action space that scales exponentially with the number of agents, is trained with the fully centralised method DQN M and is the only method taking s; as input. Furthermore, the fully decentralised method we test IQL (20). in which all agents are independently trained. Regarding the five CTDE methods, we investigate three value-based methods: QMIX (6). QVMix [17], and QPLEX 18]. They factorise the value function during training, allowing agents to independently select actions during execution after a centralised value function is jointly learnt during training. The last two are the CTDE actor-critic methods COMA [19] and FACMAC [1I] They train independent policy networks and rely on popendix(pprovidees a detailed description of each method as well as discussion of other methods of interest that are not included in the benchmark. We selected these methods for our benchmark study because they are well established and their implementations are open- sperceourced and available within the PyMarl framework 9
single centralised critic during training.
All investigated MARL methods are compared against a representative baseline in the reliability engineering community (3). This baseline, referred to as expert-based heuristic policy, consists of a set of heuristic decision rules that are defined based on expert knowledge. The heuristic policy includes both parametric and non -on-maraettiic rules. Parametric decision rules depend on two parameters: (i) the inspection interval and (ii) the number of inspected components. On the other hand, -orpaaraettiic rules involve taking a repair action after detecting a crack and prioritising component inspections with higher failure probability. To determine the best heuristic policy, and for each environment, we evaluate all parametric rule combinations over 500 policy realisations, thereby identifying the heuristic policy that maximises the expected sum of discounted rewards among all policies evaluated.

---

Table 1: Number of agents specified in all investigated IMP environments.
IMP environments
Number of agents
k-out-of-n system
3
5
10
50
100
Correlated k-out-of-n system
3
5
10
50
100
Offshore wind farm
2
10
50
100
4.2 Experimental setup
The bbove-mentioned seven MARL methods are tested in the three sets of IMP environments defined in Section 3.3 The environments differ by the number of agents and by whether or not they include a
campaign cost model. The numbers of agents tested in the six types of environments are presented in Table To objectively interpret the variance associated with the examined MARL methods, 10 training realisations with different seeds are executed each environment. an agent makes decisions based on its local damage probability, the current normalised time step. and sometimes correlation information is additionally provided; while the state, used by DQN and CTDE methods, encompasses all of the information combined. In all cases, the action space features three possible discrete actions per agent, except for DQN, where the centralised controller selects an action among the 3 possible combinations. For complexity reasons, we only test DQN in k-out-of-n environments featuring 3 and 5 components, as well as in environments with I and 2 wind turbines. Detailed information rewards, observations, and states can be found in Appendix C
explained in Section 3
Given the importance of hyperparameters on the performance of RI. methods [49], we initially selected their values reported by the original authors. In an attempt to objectively compare the examined methods, parameters that play the same role across methods are equal. Notably, the learning rate and gamma, among others, identical in all experiments. The controller agent network features the same architecture in all methods, consisting of a single GRU layer with a hidden state composed of 64 features encapsulated between fully-connected layers and three outputs, one per action, except for DQN, where the network output includes 3" actions. In our case, DQN's architecture includes additional fully-connected layers and a larger size of hidden GRU states. Moreover, following common practice. agent networks are shared among agents, and thus a single agent network is trained. Specifically, we train only one network that is used for all agents, instead of training distinct agent networks. The training process with a single agent network improves data efficiency because the same episode can be used to perform n. backpropagatiins through the same agent network, using different observations. In contrast, only one backpropagation per agent network would be possible with a single episode if training is performed with different agent networks. To allow diversity in agents' behaviour, one-hot encoded vector is also added to the input of this common network to indicate which one of the n agents is making the decision. In CTDE methods, critics or mixers are also incorporated at the training stage with specific architectures according to each method and environment configuration. In most cases, the neural networks are updated after each played episode based on 64 episodes sampled from the replay buffer, which contains the latest 2,000 episodes. The only exception COMA, which follows an on-policy approach, where the network parameters are updated every four episodes. For value-based methods, the training episodes are played following an epsilon greedy policy, whereas test episodes are executed with a greedy policy. The epsilon value is initially specified as and linearly decreases to 0.05 after 5,000 time steps. This is different for COMA and FACMAC. Appendix and the source code list more details and all parameters.
The number of time steps allocated for one training realisation is 2 million time steps for all methods. These 2 million training time steps are executed with training policies, e.g. epsilon greedy policy. saving the networks every 20,000 training time steps. To evaluate them, we execute 10,000 test episodes and obtain the average sum of discounted rewards per episode per saved network. These test episodes are executed with testing policies, e.g. greedy policy. We show in Appendix E.3 that 10,000 test episodes are needed due to the variance induced in the implemented environments. We emphasise that 10 training realisations are executed with different seeds for the same parameter values. All parameters are listed in Appendix [ and in the source code.
7

---

5 Benchmark results and discussion
The results from the benchmark campaign are presented in Figure 3 showcasing the relative perfor- mance of MARL methods with respect to expert-based heuristic policies in terms of their expected sum of discounted rewards. In each boxplot of Figureß] each of the 10 seeds is represented by its best policy, which achieved the highest average sum of discounted rewards during evaluation. We further explain the connection between learning curves and boxplots in Appendix [F] Figure [0] Our analysis relies on relative performance metries because the optimal policies are not available in the environments investigated. Additionally, the corresponding learning curves and the best-perforning
policy realisations can be found in Appendix FF
MARL-based strategies outperform expert-based heuristic policies. While heuristic policies provide reasonable infrastructure management planning policies, the majority of the tested MARL methods yield substantially higher expected sum of discounted rewards, yet the variance over identical MARL experiments is still sometimes significant. In environments with no campaign cost, the performance achieved by MARL methods with respect to the baseline differs in configurations with high number of agents, as shown at the top of Figure In contrast, MARL methods reach better relative results in environments with a high number of agents when the campaign cost model is adopted, illustrated the bottom of Figure 3 In general, the superiority of MARL methods with respect to sspert-based heuristic policies is justified by the complexity of defining decision rules in high-dimensional multi-component engineering systems, where the sequence of optimal actions is very hard to predict based on engincering judgment (36).
IMP challenges. In correlated k-out-of-n IMP environments, the variance over identical MARL experiments is higher than in the uncorrelated ones, emphasising a specific IMP challenge. Under cor- relation, inspecting one component also provides information to uninspected components, impacting their damage probability and thus hindering cooperation between MARL agents. Another challenge imposed in offshore wind farm environments, where the benefits achieved by MARL methods with respect to the baseline are also reduced in environments with a high number of agents. This can be explained by the fact that each wind turbine is controlled by two agents, being independent of other turbines in terms of rewards. Each agent must then cooperate closely with only one of all agents, hence complicating global cooperation in environments featuring an increasing number of agents.
Campaign cost environments. Yet another challenge can be observed in campaign cost environments under 50 agents, where MARL methods' superior performance with respect to heuristic policies is more limited. The aforementioned environments are challenging for MARL methods because agents should cooperate in order group component inspection/repair actions together, saving global campaign costs. In addition, the heuristic policies are designed to automatically schedule group inspections, being favourable in this case. This is confirmed by the learning curves presented in Figures [T and T2 On the other hand, in environments with more than 50 agents, MARL methods substantially outperform heuristic policies. At least one component inspected or repaired each time step and the results reflect that avoiding global annual campaign costs becomes less crucial.
Centralised RL methods do not scale with the number of agents. DQN reaches better results than heuristic policies, though achieving lower rewards than CTDE methods in most environments, despite benefiting from larger networks during execution. This highlights the scalability limitations of such centralised methods, mainly due to the fact that they select one action out of each possible combination of component actions.
IMP demands cooperation among agents. The results reveal that CTDE methods clearly outperform IQL. in all tested environments, especially those with high number of agents. This confirms that realistic infrastructure management planning problems demand coordination among component agents. Providing only independent local feedback to each IQL agent during training leads to lack of coordination in cooperative environments, also shown by Rashid et al. [16]. However, the performance may be improved by enhancing networks" representation capabilities by including more neurons, yet this is true for all investigated methods.
Infrastructure management planning via CTDE methods. Overall, CTDE methods generate more effective IMP policies than the other investigated methods, demonstrating their capabilities for supporting decisions in real-world engineering scenarios. While Figure F the variance of the best results across runs, the learning curves further confirm this finding in Appendix [F] In particular, QMIX and QVII generally learn effective policies with low variability over runs. Slightly more
8

---

uncorrolated k-uut-a-nn
correlated k-out-df-n
offishre wind form
-12
67.7
18
292.
268
58
240
01
56
1463-6
38
-262
106
-2:8.
160
-2925.0
168
100%%
-0%
s0%
-100%
-50%
50%
-100%
-50%
50%
Normal sed relative reward (IHHH
incorreiated k-out-of-n; campaign cost
comellaee k-out-of-n, campaign cost
offehhree farm; campaign
-62-2
-28
-30
-115
.64.5
18
-68.5
18
-267
2
18
-232
-211.01
56
-1248.
56
-100%
-50%
50%
-100%
Normalised relative roward (XHWH
OMIX
QVMx
QRLEX
COMA
FACMAC
IQL
Heuristic
Figure 3: Performance reached by MARL methods in terms of normalised discounted rewards with respect to expert-based heuristic policies in all IMP environments, referring the heuristics result. Every boxplot gathers the best policies from each of 10 executed training realisations, indicating 25th-75th percentile range, median, minimum, and maximum obtained results. The coloured boxplots are grouped per method, vertically arranging environments with increasing number of agents, as indicated in the tep-left legend boxes. Note that the results are clipped at -100%.

---

unstable, QPLEX also yields similar results to QMIX and QVMIX in terms of achieved results. While being able to outperform heuristic policies almost every environment, FACMAC exhibits high variance among runs. However, FACMAC effectively scales up with the number of agents and environment complexity (as reported in [II), , achieving some of the best results in IMP environments with over 50 agents well as in correlated IMP environments. The results also suggest that COMA the least scalable MARL method our benchmark. This can be attributed to the fact that the computation of the critic's ss counterfactual becomes challenging with an increasing number of agents.
6 Conclusions
This work offers an open-source suite of environments for testing scalable cooperative multi-agent re- inforcement learning methods toward the efficient generation of infrastructure management planning (IMP) policies. Through our publicly available code repository. we also encourage the implementation of additional IMP environments, e-g., bridges, transportation networks, pipelines, and other relevant engineering systems, whereby specific disciplinary challenges can be identified in a common simula- tion framework. Based on the reported benchmark results, we can conclude that centralised training with decentralised execution methods are able to generate very effective infrastructure management policies in real-world engineering scenarios. While the results reveal that MARL methods outperform expert-based heuristic policies, additional research efforts should still be devoted to the development of scalable cooperative MARL methods. While we model the IMP secision-making problem as Rec-POMDP, modelling IMP problems as mean-fiel field games [50] is promising direction to be considered environments with an increasing number of agents. Moreover, specific improvements are still required in environments where global cost is triggered from the actions taken by any local agent, e.g.. global campaign cost. Besides, more stable training is still needed in environments where local information perceived by one agent can influence the damage condition probabilities of others, as in the correlated IMP environments. In the future, more realistic and challenging environments for cooperative MARL methods could be investigated. One example would be assigning campaign costs to specific groups of components, instead of specifying only one global campaign cost.
10

---

Acknowledgments and Disclosure of Funding
The authors acknowledge the computational resources provided by the Consortium des Equipements de Calcul Intensif (CÉCI), funded by the Fonds de la Recherche Scientifique de Belgique (F.R.S.- FNRS) under Grant No. 2.5020.1 11 and by the Walloon Region. We further acknowledge the insightful comments provided by our colleagues Adrien Bolland, Victor Dachet, Nandar Hlaing, Gaspard Lambrechts, Gilles Louppe, Matthias Pirlet, and Maurizio Vassallo.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-5555
2015.
[2] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George den Driess- che. Julian Schrittwieser, Ioannis Antonoglou, Veda Pannecrshelvam,, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
[3] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 11226419):1440-11444 140--1444
2018.
[4] Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):490-4496, 2021.
[5] Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 2174-2182, 2017.
[6] Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 1236:1123 2018.
[7] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli. Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis. and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, (02(7897):414441,, 2022.
Jackie Kay, Antoine Merle,
[8] Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperatieecoompetitiee environments. Advances Neural Information Processing Systems (NIPS), 30, 2017.
[9] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.000433 2019.
[10] Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. SMACv2: An improved benchmark for cooperative multi-agent reinforcement learning, 22Xi::2212.007899 2022.
[11] Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. FACMAC: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
11

---

[12] Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforce- ment learning. Applied Intelligence, 53:13677-13772,, 2023.
[13] Pablo G. Morato, Konstantinos G. Papakonstantinou, Charalampos Andriotis, Jannie Søn- derkær Nielsen, and Philippe Rigo. Optimal inspection and maintenance planning for deteriorat- ing structural components through dynamic Bayesian networks and Markov decision processes. Structural Safery, 94: 14:102240, 2022.
[14] Frans A. Oliehoek, Matthijs T. Spaan, and Nikos Vlassis. Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32:289-353, may 2008.
[15] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
[16] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. Proceedings of the 35th International Conference on Machine Learning, 80:4295-4304, 2018.
[17] Pascal Leroy, Damien Erstt, Pierre Geurts, Gilles Louppe, Jonathan Pisane, and Matthia Sabatelli. QVMix and QVMix-Max: extending the deep quality-value family of algorithms to cooperative multi-agent reinforcement learning, AAAI-21 Workshop Reinforcement Learning in Games, 2020.
[18] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: multi-agent Q-learning. In International Conference Learning Representations, 2021.
Duplex dueling
[19] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[20] Ming Tan. Multi-agent reinforcement learning: Independent versus cooperative agents. In Proceedings of the Tenth International Conference on International Conference Machine Learning, (CML'93, page 330-337, 1993.
[21] Jesus Luque and Daniel Straub. Risk-based optimal inspection strategies for structural systems using dynamic Bayesian networks. Structural Safety, 76:68-80 80, 2019.
[22] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In JEEE/RSJ international conference on intelligent robots and systems, pages 5026- 5033, 2012.
[23] Oriol Vinyals. Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schritwieser,, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy Lillicrap. Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. StarCraft II: A new challenge for reinforcement learning. arXiv: 110.008788...882... 2017.
[24] Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Marc Lanetot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowling. The Hanabi challenge: A new frontier for AI research. Artificial Intelligence, 203216 2020.
[25) Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34-04, pages 4501-4510, 2020.
[26] Xuehai Pan, Mickel Liu, Fangwei Zhong, Yaodong Yang, Song-Chun Zhu, and Yizhou Wang. Mate: Benchmarking aulti-agent reinforcement learning distributed target coverage control. In Advances in Neural Information Processing Systems, volume 35, pages 27862-27879 2022.
12

---

[27] Sharada Mohanty, Erik Nygren, Florian Laurent, Manuel Schneider, Christian Scheller, Nilabha Bhattacharya, Jeremy Watson, Adrian Egli, Christian Eichenberger, Christian Baumberger, et al. Flatland-RI. Multi-agent reinforcement learning on trains. arXiv:20/2.05882 2020.
[28] Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. CityFlow: multi--gent reinforcement learning environment for large scale city traffic scenario. In The world wide web conference, pages 3620-3624, 2019.
[29] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021.
[30] Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, 2020.
[31] Elizabeth Bismut and Daniel Straub. Optimal adaptive inspection and maintenance planning for deteriorating structural systems. Reliability Engineering & System Safety, 215:107891, 2021.
[32] Konstantinos G. Papakonstantinou and Masanobu Shinozuka. Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part I: Theory. Reliability Engineering & System Safery, 130:202-213, 2014.
[33] Konstantinos G. Papakonstantinou and Masanobu Shinozuka. Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part II: POMDP implementation. Reliability Engineering & System Safety, 130:214-224, 2014.
[34] Charalampos P. Andriotis and Konstantinos G. Papakonstantinou. Managing engineering systems with large state and action spaces through deep reinforcement learning. Reliability Engineering & System Safety, 191:106483, 2019.
[35] Charalampos P. Andriotis and Konstantinos G. Papakonstantinou. Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints. Reliability Engineering & System Safety, 212:107551, 2021.
[36] Pablo G. Morato, Charalampos P. Andriotis, Konstantinos G. Papakonstantinou, and Philippe Rigo. Inference and dynamic decision-making for deteriorating systems with probabilistic de- pendencies through bayesian networks and deep reinforcement learning. Reliability Engineering & System Safety, 235:109144 144, 2023.
[37] Van-Thai Nguyen, Phuc Do, Alexandre Voisin, and Benoit Iung. Weighted-QMM QMIX-based optimization for maintenance decision-making of multi-componett systems. PHM Society European Conference, 7(1):360-367, 2022.
[38] Mohammad Saifullah, Charalampos P. Andriotis, Konstantinos G. Papakonstantinou, and Shelley M. Stoffels. Deep reinforcement learning-based life-cycle management of deteriorating transportation systems. Bridge Safery, Maintenance, Management, Life-Cycle, Resilience and Sustainability, pages 293-301. CRC Press, 2022.
[39] Frans A, Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016
[40] Richard E Barlow and Klaus Heidtmann. Computing k-out-of-n system reliability. IEEE Transactions on Reliability, 33()):322--33)3 1984.
[41] Felipe Giro, Jose Mishael, Pablo G. Morato, and Philippe Rigo. Inspection and maintenance planning for offshore wind support structures: Modelling reliability and inspection costs at the system level. In International Conference on Offshore Mechanics and Arctic Engineering. volume 85864, 2022.
[42] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv:1606.0155 2016.
13

---

[43] Mark Towers, Jordan K. Terry. Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, 2023.
[44] J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente. et al. Pettingzoo: Gym for multi-agent reinforeement learning. Advances Neural Information Processing Systems, 14315503-15004, 2021.
[45] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. Proceedings of the 35th International Conference on Machine Learning, 80:3053- 3062, 2018.
[46] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274)11-18, 2022.
[47] Siyi Hu, Yifan Zhong, Minquan Gao, Weixun Wang, Hao Dong, Zhihui Li, Xiaodan Liang, Xiaojun Chang, and Yaodong Yang. Marllib: A scalable multi-agent reinforcement learning library. arXiv preprint arXiv:2210.13740 2022.
[48] Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. TorchRL: data-driven decision-making library for Py PyTorch. arXiv:2306.00577 00577, 2023.
[49] Rihab Gorsane, Omayma Mahjoub, Ruan John de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. Towards a standardised performance evaluation protocol for cooperative marl. Advances in Neural Information Processing Systems, 35:5510-5522 2022.
[50] Mathieu Laurière, Sarah Perrin, Matthieu Geist, and Olivier Pietquin. Learning mean field games: A survey. arXiv:2205. /2944, 2022.
[51] Ove Ditlevsen and Henrik O. Madsen. Structural reliability methods, volume 178. Wiley New York, 1996.
[52] Nandar Hlaing, Pablo Morato, Jannie S Nielsen, Peyman Amirafshari, Athanasios Kolios, and Philippe Rigo. Inspection and maintenance planning for offshore wind structural components: integrating fatigue failure criteria with Bayesian networks and Markov decision processes. Structure and Infrastructure Engineering. 18(7):983-1001 2022.
[53] Inge Lotsberg, Gudfinnur Sigurdsson, Amee Fjeldstad, and Torgeir Moan. Probabilistic methods for planning of inspection for fatigue cracks in offshore structures. Marine Structures, 46: 167-192, 2016.
[54] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):779-292, 1992.
[55] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. Proceedings of the 36th International Conference on Machine Learning, 2019.
[56] David Ha, Andrew Dai, and Quoe V. Le. HyperNetworks. 5th International Conference on Learning Representations, 2016.
[57] Mathii Sabatelli, Gilles Louppe, Pierre Geurts, and Marco A Wiering. Deep quality-value
(DQV) learning. Advances Neural Information Processing Systems, Deep Reinforcement Learning Workshop, 2018.
[58] Mathia Sabatelli, Gilles Louppe, Pierre Geurts, and Marco Wiering. The deep quality-value family of deep reinforcement learning algorithms. International Joint Conference on Neural Networks (IJCNN), 2020.
14

---

[59] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforeement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[61] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Reinforcement learning, pages 5-32, 1992.
[62] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[63] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[64] Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-bare based reinforcement learning. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, page 538-545, 2001.
[65] Yu-Han Chang, Tracey Ho, and Leslie Kaelbling. All learning is local: Multi-agent learning in global reward games. Advances in neural information processing systems, 16, 2003.
[66] David H. Wolpert and Kagan Tumer. Optimal payoff functions for members of collectives. Advances in Complex Systems, ((23)):665-277) 2001.
[67] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: Ex- panding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 13:10199-10210, 2020.
[68] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: Multi- agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
[69] Raphaël Avalos, Mathieu Reymond, Ann Nowé, and Diederik M. Roijers. Local advantage networks for cooperative multi-agent reinforcement learning. In Proceedings of the s International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, page 1524-1526, 2022.
[70] Yali Du, Lei Han, Meng Fang. Ji Liu, Tianhong Dai, and Dacheng Tao. LIIR: Learning indi- vidual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[71] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. Interna- tional Conference on Learning Representations, 2021.
[72] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[73] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707 707446477 2017.
[74] Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is sequence modeling problem. In Advances in Neural Information Processing Systems, 2022.
[75] Andy B. Yoo, Morris A. Jette, and Mark Grondona. Slurm:
Simple linux utility for resource management. In Job Scheduling Strategies for Parallel Processing: 9th International Workshop. pages 44-60. Springer, 2003.
15

---

A IMP-MARL public repository, license, data, and documentation
The new assets we provide in this paper are listed hereafter. IMP-MARL's suite of environments is publicly available on the GitHub repository battps://gite http://ot/ooooo////ggg...m ___p_rrrr fea- turing an open-source Apache v2 license. Moreover, IMP-MARL contains wrappers to facilitate its implementation on typical MARL ecosystems, i.c., Gym 42 RLLib 45 and PyMarl [ as detailed in Section 4
To reproduce the work reported in this paper, the following process can be executed: (i) cloning the repository, (ii) installing a virtual environment with the package requirements, and (iii) executing the script instructions of the corresponding method and IMP-MARL environments. In addition to the instructions for reproducing our results, also provide tutorials to add environments or wrappers.
We also provide the data files resulting from our experiments, enabling the reproduction of any reported result without re-running the experiments, as well as the corresponding implementation code, hence facilitating future cross-comparison. The readily available results include Figures 3
and 12 along with Tables 13
15
Configuration, execution, and results files are permanently stored at Zenodo, accessible via htthttps:// zenodo. wtt//rooorrr//////// /80323/9 Additionally, the controller networks' weights of the best policies presented in Figure 3 are also stored there, thus fostering further interpretability studies of MARL- based strategies. The dataset open-access and registered with the Digital Object Identifier (DOI) More information and dedicated tutorials can be found on the repository.
10.5281/zenodo.ce
B Modelling infrastructure management in IMP-MARL
In this appendix, we thoroughly deseribe the deterioration, inspection, and transition models imple- mented in this work. These models drive the dynamics of the IMP-MARL environments provided in this paper. Based on this information, one can easily learn how create new environments.
B.1 Deterioration models
The deterioration processes introduced here specifically correspond to fatigue deterioration mecha- nisms, yet corrosion, erosion, and many other practical infrastructure management problems can be similarly modelled.
Correlated and uncorrelated k-out-of-n systems Throughout the text and the code, the set of environments related to correlated and uncorrelated k-out-of-n systems are abbreviated as struct and struct uc, respectively, or denoted as struct when referring to both of them. In the k-out-of-n environments currently included in IMP-MARL, the structural components are exposed to fatigue deterioration, and unless repair is undertaken, the crack size de (i.e., damage condition) evolves over time / as (51:
C CFMST Sm 1 ns d. m/2 22(2mm)
de+1 = (1
where In(CFM) Nip 35.2 0.5) and m 3.5 stand for material variables, which directly influence the crack growth. Due to environmental and operational conditions, the components are subject to dynamic load characterised by the stress range, SR N(u 70, a 10 N/mm²), over ns = 10 annual stress cycles, i.e., the number of load cycles experienced by the structural component in one year. At the initial step or after a component is repaired, the initial crack size is at its intact condition, defined by its initial distribution do ~ Exp(u 1 mm), and component level failure occurs when the crack size exceeds a critical size of d, = 20 mm. The component failure probability pr. defined as Py plg < 0), can be computed following through-thickness thickness failure criterion [52] where the failure limit at time step is formulated as g. = d. de- At the system level, failure event occurs if & (out of n) components fail, and its corresponding system failure probability, P can be efficiently computed as a function of all components failure probabilities, as proposed in [40]
The continuous crack size is discretised into a certain number of discrete bins in order to enable efficient Bayesian inference once inspection indications available. Further details can be found in
16

---

Table 2: Description of the discretisation scheme implemented.
Environment
Variable Interval boundaries
Bins
struet owf
de
0, exppll(110 (In(de) - In((100))/28) In(de)), oo]
30
de
(0, do (d.- do)/(60 - 2) de, oc]
60
Table 3: Variables specified in the offshore wind farm deterioration models.
Upper component
Middle component
Mudline component
# 26.45
# -2.004
# 226.12
In(CFm)
σ = 0.12
σ = 0.4
o = 0.39
m
3
3
1 = 10.21
1l = 7.40
/ = 6.74
q
CoV = 25%
CoV 25%
CoV 25%
de
20
60
60
ns
5,049,216
5,049,216
5,049.216
13 If the initial crack size among components is correlated (i.e., we dealing with a correlated k-out-of-n system), the damage condition of each component is defined conditional on a common correlation factor, a, via Gaussian hierarchical structure (36). In that case, the discretised damage bins should be defined conditional on the correlation factor. The specific discretisation implemented in our environments is defined in Table 2
Offshore wind farm In this set of environments, a group of n_ c_mopp offshore wind substructures is considered, in which three representative structural components the wind turbine: (i) at the atmospheric zone upper level, (ii) at the splash zone middle level, (iii) below the seabed mudline. The deterioration, inspection, and cost models hence differ for each of the three considered components. While the fatigue deterioration is calculated according to Eq. [ the expected dynamic load, S,, is in this case defined based on industrial standards (53) as:
modelled at different locations of
S. = q1 + 1/A)Y,
(2)
corresponding to the expected value of a Weibull distribution defined by the scale parameters listed in Table 3 q ~ NN, and shape factor, À 0.8, weighted by a geometric parameter, Y ~ LN( 0.1, 0.1). The initial crack size distribution is specified for all wind turbine components as do ~ Exp(u 0.11) and the remaining specific fatigue variables associated with each wind turbine component are listed in Table 3 At the wind turbine level, the failure event occurs if one component of the wind turbine fails. The wind turbine failure risk is then defined as the wind turbine failure probability multiplied by the consequences associated with a wind turbine failure event. At the wind farm level, the damage condition of a wind turbine does not influence the condition of the other wind turbines, and the wind farm system failure risk is defined as the sum of all turbines' failure risk.
B.2 Inspection models
The inspection models implemented in IMP-MARI are hereafter described, defining the likelihood of retrieving a certain inspection outcome as a function of the damage size.
Correlated and uncorrelated k-out-of-n systems The inspection model is normally characterised depending the accuracy of the measurement instrument, formally specified through probability of detection (PoD) curves, in which the probability of observing a crack is defined as a function of
the crack size [3]. In this case, the inspection model is described by exponential distribution p(ia, (de) ~ Exp(u = 8), defining the probability of observing a crack during an inspection.
Offshore wind farm In this more practical set of environments, eddy current inspection technique is here considered, whose PoD can be modelled according to industrial standards [53] as:
p(is, (d))
+ (d./x)²
where the factors x and b are specified as 0.4 and 1.43, respectively, for the upper component, but considered as 1.16 and 0.90, for the middle component. Naturally, less accurate inspection outcomes
17

---

can be expected for the middle component, as it is located in region below the water level, where the visibility reduced.
B.3 Transition models
An overview of the transition model is explained hereafter. For more detailed description, refer the reader to [36]. Since the crack size discretised in this work, the transition and inspection models be stored in tables. In code, they are encoded in Numpy files, which stored in the reposi- tory folder pomdp_ models.. In particular, the files are named Dr3031C10 mpz, Dr3031 H08 mpz, and owf6021. .np.. npp, for k-out-of-n system, correlated k-out-of-n system, and offshore wind farm, respectively. By relying on already stored transition and inspection models, the environments can be simulated efficiently. Alternatively, the crack size evolution could also be directly computed at execution time, yet an additional computational expense would be then incurred.
comdp_models.
The transition model can be defined based on the deterioration and inspection models previously described. no inspection and maintenance are taken (i.e. do-nothing action), the damage condition progresses each time step according to the fatigue deterioration model formulated in Eq. Note that in our three sets of environments, time step represents a year. Considering that the damage follows non-stationary deterioration process, the crack size distribution d++1 can be efficiently encoded as a function of the annual deterioration rate, Tt+11 and the crack size at the previous time step de as p(d++1 (de, re+1). Starting from To O, the deterioration rate increases by one unit every year, unless a component is repaired, in which case the deterioration rate returns to the initial value. The deterioration evolution over time step can be computed as:
p(d++1)=
Tea Epddde T( (p(e)((()1))) TE-1
an inspection action is planned, a damage indication id... is collected, and the crack size distribution can be updated via Bayes' rule:
p(det1lid p(ider (d+1 (dde++1)
where the likelihood corresponds to the specifie inspection model, described by probability of detection curve, as mentioned before. Since the damage probabilities are discrete, the normalisation constant can be straightforwardly computed by simply summing the unnormalised bins [
To enable efficient computation of the deterioration evolution under correlation, a Gaussian hier- archical structure is adopted [36], in which the crack size probability is defined conditional common factor, a as p(d/(a) (x). In this work, we consider that the initial damage probabilities are equally correlated among components with Pearson coefficient equal to 0.8.
The damage transitions, in this case, are formulated as:
p(d(41/a)==
Expldenale Ten plddppd))))) TE de
(6)
Once an inspection outcome is available, the common correlation factor is also updated based on the new information, thus influencing all components. The likelihood of collecting one inspection indication given a can be computed as:
plide4s (a:)
Σ dea1
(rdetzla)p(ia.. (dot2)))ddtt) (dot1)
and the correlation factor can then be updated:
p(alia...) x pla)p(ia
(8)
Finally, the marginal damage probabilities are computed as:
p(d(+1) = Σ p(d1+1la)p(a)
(9)
18

---

Table Main options available in IMP-MARL.
Option name
Environment
Dict key
Type
Number of components
struct
n_mp
int int bool
k components
struct
k_ k_comp
Correlation
struct
env_correlation
Campaign cost
struct
campaign_cost bool
Number of wind turbines
owf n_comp
int
Number of components per wind turbine Campaign cost
owf lev
int bool
owf
campaign_cost
Table Observations options available in IMP-MARL.
Option name
Env. Dict key
Type Dimensionality
Component damage probability
struct
*by default
float 30
Component deterioration rate
struct
obs_d_ras d_rate
float 1
All components damage probability
struct
obs_multiple
float n_comp 30
All components deterioration rate
struct
obs_all_d_rate
float n_comp
Correlation condition
struct
obs_alphas
float 80
Component damage condition
owf
"b default
float 60
Component deterioration rate
owf
obs_d_rate float
All components damage condition
owf
obs_multiple float n_momp 60
All components deterioration rate
owf
obs_all_d_rate float n_momp
C Options available in IMP-MARL and reward model
In IMP-MARL, the environments can be easily set up with specific options, from the definition of the number agents to the observation information perceived by the agents and the state information received by mixers/critics. This can be straightforwardly specified through the configuration files provided on IMP-MARL's GitHub repository. These options are in fact parameters included in IMP-MARL's classes. In particular, liste the main options available.
In addition to the main options previously mentioned, encoded in the observations and states. One can choose which local information, of the agents will receive, as well as the global information, 8t, available during training. and all possible options. Since these options are coded as booleans, the selected configuration can be easily defined by assigning a True value. Additional details can be found in the code.
also possible to tailor the information
In the experiments conducted in this work, the selected parameters are listed in Table 7 Through the code provided in IMP-MARL, future works may investigate alternative observation and state information options.
C.1 Reward model
The goal of the agents is to maximise the expected sum of discounted rewards, E[Ro] TT-1 7 [Re.] + Zami (R,, + Rt,rep) Re,camp]] as stated in Section 3.2 The rewards
Table 6: States options available in IMP-MARL.
Option name
Environment
Dict key
Type Dimensionality
All component damage condition
struct
state_obs state_d_rate
float
n_ n_comp 30
All components deterioration rate
struct
float n_comp
Correlation condition
struct owf
atpha_
float 80
All component damage condition
state_obs float n_comp 60
All components deterioration rate
owf
btate_d_rate float n_comp
19

---

Table Options set up our experiments.
Option name
struct uc
struct_c
owf
state_obs state_d_ree
True
True
True
True
True
False
state_alphas
False
True
False
obs_d_rate
False
False
False
obs obs_multiple
False
False
False
obs obs_all__ rratte
False
False
False
obs_alphas
False
True
False
env_correlation
False
True
False
campaign_cost
True & False
True False
True False
Table 8: Rewards specified in our experiments.
Component
Campaign cost
Rins
Rrep
cf
R Reamp
False
-20
-10,000
struct
True
-0.2
-20
-10,000
-5
False
-1
-10
-1,000
0
owf upper level
True
-0.2
-10
-1,000
-5
False
-30
-1,000
0
owf middle level
True
-1
-30
-1,000
-5
collected at each time step may include inspection Rins and repair Rrep costs for all considered components, along with the system failure risk, which is defined the system failure probability Pr.., multiplied by the associated consequences of a failure event Cy. formulated as R = Pfyy Cf- Additionally, campaign cost Reamp may also be included if that option is active. The discount factor is defined as = 0.95 in our experiments and the specific rewards are listed in Table g
D Cooperative MARL methods
We considered methods from two families of algorithms in MARL: value-based and policy-based. Here, we propose a brief description of these methods, starting from single-agent RL (SARL) definitions to MARL. However, for more detailed information, we refer the reader to the original papers. Furthermore, in addition the definition of Dec- Dec-POMDP in Section[..1 we define the value function, also V function, which evaluates the current joint policy (TT(s) E RR function of interest is the state-joint-action value function Q..t = Q u) E 8t = 8,
TT.. Another u).
also called Q function. The individual Q function is defined by QT. u) EE| (e |s: u u, лt or Qa for short. The reward is common to all agents and implies that Quo = Q.Va. The advantage function is defined as A(s, u)
Q(s, u) V(s).
Value-based methods aim to learn the optimal Q function defined as Q (s, u) = max, (s, u). This enables the agent to greedily select the action x² (s) argmax, Q" (s, u). In SARL. this is accomplished through Q-learning (54), originally designed for tabular settings. However, as the size of the state-action space increases, it becomes impractical to compute Q for each state-action pair. solution, named DQN [1, approximates Q with neural network 0 and learn (s, u; 0) by
minimising the loss (0) = Exy~B (r: + maxueu Q( 0') ((s, ue; 0)) where B is a replay buffer composed of transitions (St, Tt, St41) and 0' is the target network, a copy of updated periodically. This approach can train centralised learner in a Dec-POMDP all agents can access the state during execution, resulting in the learning of Qtat Q" (s, u). Issues are that the joint action space scales exponentially with n, and in practice, agents select their action based only on their history of observation (o. r) and not the state s. There is decentralised solution, named IQL 20). which consists in learning independently Q.. However, there are also CTDE methods that take the state into account during training. A solution proposed in QMIX [6 is to approximate Qint as function of all Qa and during training. Agents select actions based on their Qa. which now utility functions that factorise Qiot and not Q function. One condition is that individual
20

---

Qa satisfy the individual global max (IGM): argmax Q(s,, un) = U, ,r Qa(Ti' u4) [55] In QMIX, the factorisation is achieved by a constrained hypernetwork [56] which links s and Qo
to Qtot. QVMiX 17 extends QMIX with the Deep Quality- Value method [57 588, learning both
V and Q, using the former as the target of the latter. QPLEX [18] extends QMIX with the dueling
structure Q = V + A [59], learning a factorisation of V and A with transformers [60]. We selected these methods based on their IGM consistency, code availability, and results in the literature.
Policy-based methods learn directly the optimal policy through a neural network 7(8, u; 0) that maximises J( = Exz [Ro]. The well-known REINFORCE method [6] ascends the gradient
VaJ= E R. Ve log (Upg 0)] to find π-. Actor-critic methods [62 63 expand upon this method by incorporating a parameterised critic that estimates Q(s,, ue; Φ). replacing Rs, with the actor serving as the parameterised policy. To reduce variance, a baseline b(s) is injected into the gradient, usually b(s) = V(s), and Q(s, u; Ø) is replaced by A(s, u; Ø) [64], leading to the new gradient expression VoJ = E,E, A(s,, uz; Φ) Vo
o log x (St, uy; 0)]. Advantage estimation is accomplished either by A(s,, uy; Φ) = (st, uy; Ø) E. (u]s; ((s σ) or by A(st, uy; Φ)
r; + x) V(s,; Ø). Extending these methods to MARL is straightforward process and the decentralised solution is named IAC [1]] This approach involves each agent learning independently an actor and a critic, based only on the tuple (T, However, this solution does not exploit the additional information provided by the state 8. During training, the critic may exploit the full state s. which would result in a centralised critic. However, this approach provides the same feedback to all agents, missing out on the crucial aspect of credit assignment [65]. To address this, MADDPG g allows each agent to learn its own critic (s, u; Φ) that is considered centralised since its use of s and u. On the other hand, COMA [19] and FACMAC [I]] propose solutions with single centralised critic. FACMAC suggests using a central but factored critic by employing the value function factorization of QMIX. The joint-action (s, u: Φ) is built as function of Qu u²), without the need to satisfy IGM and without the constraints on the hypernetwork. COMA, inspired by difference reward (66), proposes having the centralised critic compute a counterfactual baseline for each agent. For an agent o, the difference reward is R(*++1, 8t, u+) - R(* 1, 8t, (uz .)) where is default action. Computing this requires simulating the environment steps several times. But in COMA, the centralised critic computes the advantage Au (st. uz: Ø) = Q(st. u; Φ) - Eye (u' rf: 0) (ss,, (u," u²"): ): Φ) for each agent, allowing it to approximate A without more environment steps. This has the cost of requiring to increase the input space of A as it additionally takes u actions as input.
Other methods In addition to QMIX [6.. QVMix 17 and QPLEX 18 QTRAN 555 and
(55) Weighted-QMII [67] factorise Quar differently from QMIX, but do not always satisfy IGM
Other methods, such as MAVEN (68] and LAN [69], also extend over QMIX. The first improves exploration capabilities while the second Iearns to cooperate without factorising Qtor- There are also policy-bascd methods that rely on the actor-critic paradigm, such COMA [19 and FACMAC II with a single centralised critic. MADDPG [8 is another well-established method, which does not learn single centralised critic, but one per agent and is designed for continuous action spaces. Another method, LII (70), aims to provide credit assessment with individual intrinsic rewards, while HATPRO and HAPPO 71 demonstrate that popular actor-critic methods like TRPO (72) and PPO 73 can be extended to cooperative MARL tasks. HATRPO and HAPPO could have been a great addition to our study but are unfortunately not implemented within the Para library.
Another approach for dealing with cooperative multi-agent settings is to link the recent success of sequence models and reinforcement learning by using a multi-agent transformer (MAT) that learns to transform a sequence of observations into a sequence of actions, one per agent [74].
E Experimental details
E.1
Description of the parameters set up in the experiments
In this section, we provide the information required to reproduce the results reported in this paper. Since the neural networks are trained via MARL using PyMarl's [9] library, the parameters are here described following PyMar'' convention. However, their purpose can be easily deduced from the names themselves. The experimental parameters set up equal across all experiments are presented in Table 9 while the parameters specific to each method are hereafter detailed. Note that in Table 9 some parameters are not used by all methods, c.g., RMS parameters. Besides, target update intervals,
21

---

Table Parameters set our experiments.
Parameter name
Parameters value
Parameter name
Parameters value
Y
0.95
Time max
2,050,000
Target update
200
RMS epsilon
10-5
RMS alpha
0.99
Grad norm clip
10
Learning rate
0.0005
Obs last action
True
Agent network
[
64 GRU []
Save model interval
20,000
Table 10: Exploration parameters.
Epsilon start
Epsilon finish
Epsilon anneal time
Value-base methods
0.5
.01
5000
FACMAC
0.3
0.005
50000
buffer size, and batch size are specified based on the number of episodes. When the number of agents increases, we only augment the number of trainable parameters of the mixer/critic networks, while the actor networks and other parameters are not modified, All the parameters can be found in the configuration files available on the GitHub repository and can be used launch any of the experiments conducted in this paper.
Regarding the agent network representation for CTDE methods and IQL, it consists of one GRU layer with a hidden state that includes 64 features. This means that the input is fully connected to the 64 hidden states, which then fully connected to the outputs, one per action. We represent it "[] 64 GRU - []". Since the number of actions, and hence the number of outputs, of the DQN network is 3", network with more representation capacity is needed. In that case, linear layers, whose number of output features are specified between brackets, are also included in the agent network surrounding the GRU layer. With
3 agents, the network is set "[128] 128 GRU [128,64]" while for n = 4 or n = 5 agents, is set as "[256] 256 GRU [256,256]". Note that the linear layers before the GRUs include Relu activation function and the last taken action also added to the observation of the agents as an additional input.
As mentioned previously, some parameters common to almost all methods. For instance, the optimiser selected is RMSProp for all methods, except for FACMAC, which is trained with ADAM. In nearly all methods, the buffer stores the latest 2, 000 episodes, and at each episode, 64 episodes are sampled to update the network. However, since COMA is an on-policy method, the networks updated with the episodes just played. Therefore, in our experiments, COMA's networks updated every four episodes based on these last experienced ones. This update is performed four times to ensure a fair amount of network updates with respect to the other methods, which are, in turn, updated every episode.
During training, the value-based methods rely epsilon-greedy policy, whose parameters are specified in Table 10 while they act following a greedy policy at testing. Note, however, that COMA and FACMAC utilise different training policies. FACMAC samples discrete actions through Gumbel softmax for its actor, whereas exploration is performed via an epsilon, whose values are specified in Table TO On the other hand, COMA follows classic stochastic policy during training. At the testing stage, FACMAC and COMA select actions adhering to a greedy approach, selecting the action associated with the maximum probability.
In the conducted experiments, the parameters set up in the environments differ with respect to the number of agents and we distinguish three cases (i) <= 10, (ii) 50, and (iii) = 100. Starting with QMIX, the parameters are all the same when increasing n, except for the architecture of the mixing network, whose embedding size in the middle of the mixer is: (i) 32, (ii) 64, and (iii) 128. QMIX relies on a double-Q feature, i.c., the loss computed to update differ from the original. While in the original loss function, the target Q value used for the update is selected with the action that maximises the target Q value parameterised by 0', in double-Q. the action is the one that maximises the Q value parameterised by 0. Therefore, have: L(0) E (re 7Q (((e+11, u=; 0¹)
(se.. uz; "] where u* argmax, ((5++1, u; O). This target network is updated every 200
22

---

Table 11: Number of trainable parameters in uncorrelated k-out-of-n systems.
Method
Network
n = 3
10
n 50
100
Agent
27,587
27,715
28.035
30,595
33,795
QMIX
Mixer
18,657
41,249
133,569
5,430,657
42,202,
Agent
27,587
27,715
28.035
30,595
33,795
QVMix
Mixer
64,771
110,083
295,043
10,891.779
84,437,891
Agent
27.587
27,715
28.035
30,595
33,795
QPLEX
Mixer
58,249
83,289
155,269
1,830,933
4,901,797
Agent
27,587
27,715
28.035
30,595
33,795
COMA
Critic
35,971
45,955
70,915
1,195,907
10,840,323
Agent
27,587
27,715
28,035
30,595
33,795
FACMAC
Critic
48,002
72,706
134,466
1,042,370
3,313,218
Agent
27,587
27,715
28,035
30,595
33,795
IQL
0
0
0
0
0
Agent
157,979
758,003
/
DQN
/
0
0
episodes. QVMix is variant of QMIX and the parameter values similarly specified. In partieular, QVMix contains two networks: (i) Q network with the same architecture as QMIX, and (ii) V network that is copy of the network, but with only one output. As for QPLEX, we try to be close to the parameters selected for the SMAC experiments they conduct in their paper. When increases, we change only the attention layer size composed of a layers and b heads. In particular, set up (i) 1L4H, (ii) 2L4H, and (iii) 2L10H. The mixer embedding is 64 for all experiments.
With respect to COMA, which is actor-critic method, few specific parameters should be addi- tionally set up. The agent network, here denoted as the actor, is the same as for the other previously deseribed. However, the eritic varies with n because the input of the critic becomes rather large, and its architecture is fully linear: (i) [128, 128]. (ii) [512, 256, 128. 128], and (iii) [2048, 1024, 512, 256, 128]. A TD-À used update the critic is set 0.8 and the learning rate to train the critic is the same as the one of the actor, specified in Table 9 In terms of FACMAC, the parameters of interest those at the critic, as its size increases with the number of agents n. FACMAC features 2-layer-mixing network and, therefore, we specify the size for both: (i) 64-64, (ii) 128-64, and (iii) 128-128. As in COMA, the TD-À parameter those included in value-based methods, that is also updated every 200 episodes with soft target update with 0.001. Moreover, we follow the optimiser selection made in FACMAC's original paper and used ADAM, with an epsilon equal to 10-8
to 0.8. The critic has target network, similar to
Regarding IQL, which is decentralised method, the parameters identical in all tested environ- ments. IQL also has the double-Q feature activated and learns independently individual Q values via DQN's algorithm. For DQN, we only experiments with less than five agents, i.e., <= 5. The main difference between tested environments is the size of their networks. Since DQN features 3" outputs, only 64 GRU cells are not enough in terms of network capacity. In our experiments, we increase the network size and confirm that DQN manages to achieve similar results as the other MARL methods. In DQN, the network architecture is the only parameter that is adjusted with the number of agents n.
Finally, list in Table 11 the number of trainable parameters of the networks. The input is slightly different between the tested sets of IMP-MARL environments and, therefore, we show in the table only the parameters for one of them. Note that there is an agent network taking actions for all agents and we purposely duplicate the agent network row to emphasise that all agent's ' networks are identical across experiments.
E.2 Hardware and experiments duration
They are executed with specific hardware requirements based on the number of agents: experiments with up to 10 agents are run on only CPUs, while we execute experiments on GPUs with 50 and 100 agents. The efficiency does not substantially improve when running experiments with less than 10 agents
Our experiments are run on different clusters managed by SLURM 75
23

---

Table 12: Hardware configurations for training and testing experiments.
Parameters
Train only CPUs
Train GPUs
Test only CPUs
Test GPUs
Number of CPU
4
2
8
5
RAM
5 Gb
6 Gb
5Gb
10 Gb
Experiments executed on CPUs only
agents
agents
agents
30
30-
12-
25
25
hours)
10-
hours)
Time (in hours)
20
20-
155
15-
10
2
0-
Intel(A) Xeen(R)
2.50GHz
Intel(R) Xean(R)
@
Intel(R) Xeon(R) Silver 4116 CPU 2.10GHZ
Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
Intel(Ri Xeon(R) Gold 5218 CPU @ 2.30GHZ
AMD EPVC 7763 64-Care Processor
Figure 4: Training duration for experiments with n <= 10 performed on only CPUs.
GPUs because training a GRU layer requires forwarding the whole episode sequentially. In contrast, the computational time can be reduced when running experiments with 50 and 100 agents on GPUs because we train all agents as single network and the batch size increases with n. We categorise the computational time required for the reported experiments according whether (i) the experiment (or is not) run only on CPUs, and (ii) the value reported corresponds to the training or the testing stage. In Table 12] we additionally provide the hardware requirements demanded during the training and testing phases. Note that we benefit from more resources during testing because 10 environments are running parallel. Moreover, we intentionally demand more RAM avoid problems. These reported RAM configurations are indicative and can be seen as requirements, yet not as exact memory usage numbers.
We represent the computational time required for the experiments during training in Figures and 5 as well as during testing in Figures 6 and 7 To avoid overloading the figures, the markers do not explicitly indicate which experiment they correspond to, yet the experiments are all vertically grouped based on the method and the environment. For each abscissa, 20 experiments, with and without campaign cost, are represented. The first three sets of experiments represent QMIX in the uncorrelated k-out-of-n setting, followed by QMIX in the correlated k-out-of-n environment which is followed by QMIX in the offshore wind farm one. The methods are ordered QMIX, QVMIX, QPLEX, COMA, FACMAC, IQL, and DQN, while the environments are ordered as k-out-of-n setting, correlated k-out-of-n, and offshore wind farm. It can be seen in Figures 4 and that the two plots with n < 10 additionally represent the three additional experiments related to DQN.
The first observation that may be addressed is the resulting high variance. The variation across runs logical because of the specific performance of the CPU/GPU models employed, but also due to the additional activity of clusters at the time of our experiments. With respect to the computational time required for training, testing episodes are not executed and can see that, by running the experiments on only CPUs, we manage to train the agents in less than 10 hours, except for some occasional outliers. For experiments with 50 agents, and also relying on GPUs, the computational time is overall very similar to those previously mentioned, requiring less than 10 hours, yet longer time is needed for those with 100 agents. The fastest training results correspond to COMA because we are running four environments in parallel, instead of one during training. We can see that IQL and QMIX follow closely, but QVMix, QPLEX, and FACMAC require additional computational time due to their architecture complexity. Naturally, the testing stage needs more time compared
24

---

Experiments
12-
20.0-
17.5-
15,00
(in
Time
12.5-
10.0-
Intel(R) XeenxR) Silver 4116 GPU 2.10GH2/ NVIDIA Geforce RTX 2060
Intek(i) XeeniR) CPU E5-28220 vi 2.10GHz NVICIA Geforce GTX 080
Intak(R) XeantRi Gold 5218 CPU @ 2,305Hz Quadro RTX 6000 IntekR) XeansRi 4216 2.10GHe NVIDI Geforce 2080
Intol(R) Xcon(R) Gold 6248 CPU @ 2.50GH2/ Testa V100 SXM2 32GB Intel(R) Xeon(R)
2.10 MIDIA RTX A5000
Figure 5: Training duration for experiments with >= 50 performed on CPUs and GPUs.
Experiments agents
only
2 agents
10 agents
304
25-
20
20.0-
20
time (In hours)
Time (in houss]
17.5-
12.5
10.
7.5-
5.0-
AMD EPYC 7763 64-Core Processor
Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
Intel[R] Xeen(R) Silver 4216 CPU 2.10GHZ
Intel(R) Xeon(R) CPU E5-2620 2.1.222.0 2.10GHZ
Silver 2.10GHZ
Intel(R)
Figure 6: Testing duration for experiments with
10 performed on only CPUs.
Experiments exccuted on CPUS/GPUS
agents
agents
60-
30
50-
60-
Time
30-
Intal(R) KeontRt Gold 5218 5218 CPU @ 2.30GHz Quadro RTX 6000
Inta(R) Keon(Ri Silver 4216 CPU @ 2.10HH// NVIDI Geforce RTX
InteloR) XeaniRt Stiver 4116 CPU 2.10GHZ/ NVIDI Geforce RTX 2080 TI
Intare) KcontR) Gold 6248 CPU @ 2.50GH2/ Testa V100 SXM2 3268
A5000
ES-2620
KeantR)
2.10GH2 GeForce
Figure 7: Testing duration for experiments with n > 50 performed on CPUs and GPUs.
25

---

k-nut-okn
40
correlated k-out-ofn
Time (nn heurs)
30
20
10
10
50
100
Number of components
Figure 8: Computational time required for executing expert-based heuristic policies function of the number of components. The experiments are run on 2 AMD EPYC Rome 7542 CPUs 2.9GHz.
epterdee
eppencee
10 eplendees
9125
Stondrrs oo
125
125
100
75
50
0.5
03
035
Timendace (X10
Figure 9: Variance analysis given set of neural networks. We report the standard deviation of the sum of discounted rewards obtained during the test phase. Each curve represents entire test experiment when executing 100, 1,000, or 10,000 test episodes.
to training for all experiments because executes 10, 000 test episodes per stored network. The correlated k-out-of-n environment slightly requires more time than the others because the correlation information updated at every inspection step.
Furthermore, represent in Figure the time required for the computation of expert- based heuristic policies. The experiments are plotted as a function of the number components and coloured based on their corresponding environment. In this case, all experiments are run CPUs. We can that heuristic policies can efficiently computed for environments with less than 50 components, yet the computational time significantly increases for experiments with 50 100 components. This result logical since the combination of evaluated parameters includes the number of components to be inspected each inspection interval. Besides, the overall computation time is directly influenced by the time needed to run an episode, with the k-out-of-n environments taking longer compared to the offshore wind farm ones because the episode's finite horizon spans over 10 additional time steps.
E.3 Statistical analysis of the variance associated with the number of test episodes
As previously explained, we conduct 10,000 test episodes to reduce the variance related to the expected sum of discounted rewards within a given environment. The choice is motivated by the direct relationship between the variance associated with E[Ro) and the number of test episodes. In our experiments, we average over 10,000 policy realisations, but here, we show that the standard deviation associated with [[] for each trainig time step can vary significantly if an insufficient number of test episodes is simulated. Figure 9 illustrates the standard deviations observed when executing with 100, 1, 1,000, and 10,000 test episodes. To produce this figure, we take the neural networks obtained with one FACMAC training run. These networks are trained over time in the offshore wind farm environment with 100 components. From this single set of networks, we execute 10 times the 100, 1,000, and 10,000 test episodes to observe how the variance evolves with this number of test episodes. We observe that standard deviations of E[[R)] obtained with 100 test episodes significantly vary over the investigated test runs. Naturally, the variation of the standard deviation reduced with an increasing number of test episodes, obtaining very similar standard deviations when testing with 10,000 episodes.
26

---

Reverd
-00% -3%
50%
Norwlled
-10% -0%
30%
Nomaited Hitchereeend
Figure 10: Visual description of the iterative process followed generate the boxplots showcased in Figure [Left] Learning curves corresponding to 10 QMIX training seeds in a k-out-of-n system with 50 agents. The markers highlight the policies that result in the highest expected sum of discounted rewards during evaluation, i.e., one policy per seed. [Right] The 10 selected policies are displayed the top as a function of the expected sum of discounted rewards, along with the heuristic score obtained in this environment. In the middle plot, we calculate and represent the 10 selected policies as a function of normalised relative rewards, i.c., (x h) h. Finally, a boxplot is constructed at the bottom based on the previously calculated ten normalised relative rewards, each representing different seed.
Moreover, one can also compute the largest difference that is observed between these 10 test runs at a specific time step i.e., the absolute difference of the estimated Ry] at a given time step. When testing 100 episodes, the absolute difference is 231.89 provided E Rn] is -2786.1.
time step where the maximum of the 10 1000 episodes are tested, the absolute difference is 99.8 where the at this time step, whereas by testing 10,000 episodes, the difference equals 24.8 with a maximum of - 2622.8. These numbers represent only a trained set of networks over 1920 training runs, and thus final conclusion cannot be claimed, yet it motivates the need of simulating 10,000 test episodes, as with only 100 test episodes, the absolute difference can reach up to 10 %.
maximum is - -2757.2 2
F Additional benchmark results
In this section, we present additional results and remarks beyond those reported in the main text. In the first place, we provide the values of the expected sum of discounted rewards achieved by the best runs over all experiments. We list the best policy for each conducted experiment in Tables T3 and 15 Note that the maximum values represented with markers at the right of each box in Figure can be retrieved from these values by applying the normalisation (x-H)/H, with being the value achieved by the heuristic policies. In Figure 10
we also visually illustrate the normalisation process that results the boxplots presented Figure
Additionally, we represent in Figures and 12 the learning curves corresponding to all our experi- ments. The learning curves showcase the evolution of the expected sum of discounted rewards every 20,000 training time steps, computed at the testing stage with 10,000 test episodes. Since the training conducted with 10 different seeds for each environment and method, we also plot the corresponding 25th-75th percentiles around the median. These results confirm the variance observed between the best results and presented Figure
Based on Tables 13 and [4] one may additionally infer that correlated environments result in lower costs with respect to those uncorrelated. This is especially true for environments with n>=10 agents, specified without campaign costs, and in all environments set up with campaign costs. While MARL methods profit from the additionally provided correlation information, this is not always the case for the heuristic policies.
One final remark is that the discrepancy between the expert-based heuristic policy and MARL methods more pronounced in offshore wind farm environments. This could be attributed to the shorter decision horizon or the higher cost per inspection in this particular case (see Table I5).
27

---

Table 13: k-out-of-n system best policies (* campaign cost).
QMIX
QVMix
QPLEX
COMA
FACMAC
IQL
DQN
Heuristics
3
-9.7
-9.8
-9.7
-10.6
-10.4
-35.3
-9,9
-12.5
5
-20.4
-20.7
-20.4
-21.8
-22.1
-108.7
-24.0
-25.2
10
-51.0
-51.5
-51.0
-54.3
-61.3
-404.5
-63.7
50
-229.7
-236.0
-212.8
-1190.6
-249.0
-1991.1
-268.1
100
222.6
-230.7
-220.6
11770.1
-225.7
-1770.1
-262.4
#3
-14.6
-14.7
-14.7
-15.0
-17.0
-35.3
-13.5
-15.1
#5
-27.4
-27.7
-27.4
-28.9
-33.0
-27.8
-26.6
-28.6
#0
-58.9
-63.0
-60.7
-70.0
-61.9
404.5
-64.5
#50
-169.5
-173.9
-168.4
-241.4
-160.7
-623.3
-232.7
#100
-167.2
-175.8
-160.2
-1770.1
-144.8
-1770.1
-231.5
Table 14: Correlated k-out-of-n system best policies ( campaign cost).
n
QMIX
QVMix
QPLEX
COMA
FACMAC
IQL
DQN
Heuristics
3
-9.7
-9.7
-9.6
-11.0
-10.6
-10.0
-10.0
-13.0
5
-20.4
-20.6
-18.4
-21.2
-21.6
-20.2
-23,4
-28.1
10
-47.6
-51.0
-45.2
-49.7
-46.1
-374.5
-67.7
50
-24.3
-233.0
-212.3
-419.3
113.44
-1339.9
-24..0
100
250.3
-289.0
-276.8
-486.9
-18.3
-1744.0
-218.1
#3
-1311
-12.9
-12.9
-14.8
-18.0
-34.7
-2266
-15.2
#5
23355
-24.7
-23.5
-28.2
-9.2
-23.9
-26.8
-30.5
10
-56.2
-53.4
-50.1
-52.8
-49.2
-56.0
-68.5
$50
-132.6
-157.1
-121.2
-159.3
-106.6
-814.9
-21.00
* #100
-14.77
-147.5
-121.0
-339.1
-71.3
-723.8
199.00
Table 15: Offishre wind farm best policies ( = campaign cost).
QMIX
QVMix
QPLEX
COMA
FACMAC
IQL
DQN
Heuristics
-2333
-23.3
-23.2
-23.7
-40.5
-23.7
-23.2
-58.3
4
47.1
.47.4
-4..1
.47.9
-122.4
-47.4
-47.7
-116.9
10
-118.4
-119.4
-118.5
-122.2
-235.2
-120.8
-292.3
50
-604.4
-613.9
-604.6
-2805.8
-627.3
-2892.5
-1463.8
100
1124.
-1238.8
-121..2
-5785.1
-1625.2
-5785.1
29255.0
#2
-5.88
-52.0
-1.9
-60.1
-60.3
-52.0
-8.99
-62.2
#4
80.5
-80.7
-80.7
-122.2
-118.6
-85.6
-76.0
-115.2
#10
-19.33
-133.3
-13.00
-314.5
196.44
-132.0
-267.2
*50
-432.9
-436.9
-434.5
-2892.5
-50288
-1709.7
-1248.2
* #100
-808.1
-829.0
-852.3
-5785.1
-1280.5
-5785.1
-2436.3
28

---

COMA
-25
-07
Roward
-10
590
-150
Timestess (616
Row33
-200
-200
-400
-100
-100
-100
300
-300
-200
-500
-500
-500
800
600
600
(x10")
unsonslatdd Ko
sondlred swr.ttnnn
-500
-000
-100
Roward
-200
4000
-500
-503
Roward
-4000
1100
-0000
-2000
-2000
1000
-2500
Tinedlepe (x10
Figure 11: Learning curves in all environments with no campaign cost. Curves represent the sum of discounted rewards obtained during test time. The bold line the median while the error bands are delimited by the 25th and 75th percentiles. Colours represent the different methods and the parameters of each environment can be inferred from the title above its corresponding graph.
29

---

COMA
-07
Roword
-10
-125
590
-190
Timestess
Rowa3
-20
-200
-400
k
-100
-100
-100
300
-300
-200
-500
-500
-500
600
unsonslated kostoolnnnn
sonsland -or-d--
-500
-000
-100
Roward
-2000
-2000
4000
-500
-503
Roward
-4000
1000
-6000
-2000
-2000
1000
155
Thredape (x10
Figure 12: Learning curves in all environments with campaign cost. Curves represent the sum of discounted rewards obtained during test time. The bold line the median while the error bands are delimited by the 25th and 75th percentiles. Colours represent the different methods and the parameters of each environment can be inferred from the title above its corresponding graph.
30


Reviews:
[{"review_id": "OpeJiK87NR", "reviewer": "Reviewer_quZJ", "summary": "", "questions": "", "limitations": "The connection between this paper to the mean-field game is obvious. The authors may want to emphasize this connection in the paper.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The reinforcement learning framework emphasizes the cooperation among different components in the complicated system. The cooperation reflects the complex network structure in the large-scale infrastructure system.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JS3jjAy2tk", "reviewer": "Reviewer_3EU2", "summary": "", "questions": "", "limitations": "Yes, the authors have mentioned limitations in discussion and conclusions.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "First, extensive benchmarking is done of the environments on several MARL algorithms. These are also compared with expert-based heuristic policies to analyze scenarios when learned policies fail. This shows the benefit of benchmarking exercise on new environments to highlight further avenues for RL research.\nSecond, the environments are released with Gym and PyMARL wrappers which allows easy use with RL packages. This is essential to facilitate fast adoption by the RL community.\nThird, it is a good evidence of real-world utility of MARL (for reliability engineering) which is critical as we seek to move beyond game play.\n\nI think this a strong paper which deserves to be accepted.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "YKXNiEoXEA", "reviewer": "Reviewer_y9Yt", "summary": "", "questions": "", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "RL and online planning in sequential decision-making hold significant promise in real-world applications. Yet, benchmark environments are often toy settings that simplify several real-world assumptions. As a result, this set of benchmark environments is an important step in establishing a suit of large-scale real-world domains where cooperative RL can be particularly useful. The paper is well-written and motivated, but in my opinion, the breadth of coverage of problems is not enough for this track at this moment. I would encourage the authors to include more problems and resubmit.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "6pcb4DK8zp", "reviewer": "Reviewer_FPDs", "summary": "", "questions": "", "limitations": "Generally, I think there are few limitations besides from the obvious ones that could be the number of MARL methods and environments provided out-of-the-box. However, as this is not my area of expertise, I am not aware of any obvious omissions. \n\nI only view the social impact as being positive and cannot see any negative effects, which is the view of the authors as well.\n\nI am not sure I think the simulation of the offshore wind farms is done very well. Dividing the wind turbine into three different zones do not make much sense to me. It is not clear to me how one would do service to the middle part that is lowered into the water. Usually, moving and electric parts in the nacelle are the ones that are monitored for defects and maintained, as the non-moving parts are really hard to maintain, maybe with an exception for maintenance related to corrosion. \n\nAnother way the model seems too simplistic is that the progressions of the probability vectors utilize on physics-based models and not monitoring of real-time measurement. This can of course be improved upon, and I mean that this is not important for this paper but would be important for the applicability of models developed using this framework.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is well written, mostly clear and the topic is timely, as no other similar benchmark of multiagent RL for infrastructure management planning exist.\n\nThe paper is thorough with several appendices detailing all parts of the paper. \n\nI consider the contribution high, as predictive maintenance will be increasingly prevalent in industry and the importance of improving planning of inspections and service for large-scale manufacturing plants will increase. Hence, identifying promising methods will be important so to reduce the risk of performing costly real-world experiments on less promising methods.\n\nI expect that this benchmarking suite could be relevant for the broader community focusing on planning industrial service operations.\n\nThe quality of the research is high.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: 3 of the four reviewers were generally quite supportive of this benchmark based on infrastructure planning (IMP), and recommended acceptance. 

One reviewer (y9Yt) does bring up some valid criticisms--especially in terms of the kinds of assumptions underlying the benchmark. 

In accordance with the majority opinion, I will recommend acceptance. 

Having looked at the reviews, the discussion and the  paper, I was sympathetic to the concerns of y9Yt. In particular, I think more attention needs to be paid to the connection to the planning/MDP literature. The International Planning Competition--especially the stochastic track--had looked at infrastructure planning like benchmarks (including such old chestnuts as the SysAdmin problem). It would be useful to put benchmark baselines from that model-based inference literature too--instead of just going with simple heuristic approaches. I say this in part because there was at least one case--during IPC 2021/ICAPS 2021--when a competition benchmark-- The Flatland Challenge -- designed specifically showcase MARL approaches for a class of inference problems wound up being won by model-based approaches. See this link for details:  https://icaps21.icaps-conference.org/Competitions/
Decision: Accept (Poster)