SlowGuess/NeurIPS_2023_OCR huggingface dataset:
*Pascal Leroy<sup>1‡</sup>, *Pablo G. Morato<sup>2b</sup>, Jonathan Pisane<sup>3</sup>, Athanasios Kolios<sup>2</sup>, Damien Ernst<sup>1,4</sup>
<sup>1</sup>Montefiore Institute, University of Liège, <sup>2</sup>Technical University of Denmark, <sup>3</sup>Thales Belgium
$^{4}$ LTCI, Telecom Paris, Institut Polytechnique de Paris
$\mathbb{P}$  pleroy@uliege.be, pgmdo@dtu.dk
*Authors contributed equally
We introduce IMP-MARL, an open-source suite of multi-agent reinforcement learning (MARL) environments for large-scale Infrastructure Management Planning (IMP), offering a platform for benchmarking the scalability of cooperative MARL methods in real-world engineering applications. In IMP, a multi-component engineering system is subject to a risk of failure due to its components' damage condition. Specifically, each agent plans inspections and repairs for a specific system component, aiming to minimise maintenance costs while cooperating to minimise system failure risk. With IMP-MARL, we release several environments including one related to offshore wind structural systems, in an effort to meet today's needs to improve management strategies to support sustainable and reliable energy systems. Supported by IMP practical engineering environments featuring up to 100 agents, we conduct a benchmark campaign, where the scalability and performance of state-of-the-art cooperative MARL methods are compared against expert-based heuristic policies. The results reveal that centralised training with decentralised execution methods scale better with the number of agents than fully centralised or decentralised RL approaches, while also outperforming expert-based heuristic policies in most IMP environments. Based on our findings, we additionally outline remaining cooperation and scalability challenges that future MARL methods should still address. Through IMP-MARL, we encourage the implementation of new environments and the further development of MARL methods.
Intelligent agents trained with reinforcement learning (RL) have proven successful in solving complex decision-making tasks, e.g., games [1-3], autonomous driving [4, 5], human healthcare [6], nuclear fusion [7], among others. RL training approaches where multiple agents interact together are commonly denoted as multi-agent reinforcement learning (MARL) methods. In certain applications, these agents must cooperate to accomplish a common goal, leading to the special case of cooperative MARL. To support the advancement of cooperative MARL methods, multiple environments based on games and simulators have served as benchmark testbeds, e.g., the particle environment [8], StarCraft Multi-Agent Challenge (SMAC) [9, 10], and MaMuJoCo [11]. Benchmarking on environments based on games and simulators is useful for the development of MARL methods in specific collaborative/competitive tasks, but additional challenges may still be encountered when deploying MARL methods in real-world applications [12].
Infrastructure Management Planning (IMP) is a contemporary application that responds to current societal and environmental concerns. In IMP, inspections, repairs, and/or retrofits should be timely planned in order to control the risk of potential system failures, e.g., bridge and wind turbine failures,
Figure 1: Overarching representation of an infrastructure management planning (IMP) problem. The system failure risk is defined as a function of the probability distribution over the components' damage condition. To control the system failure risk, components can be inspected or repaired at each time step  $t$  and, typically, an agent controls one component. The objective of IMP's problem is the maximisation of the expected sum of discounted rewards, by balancing the system failure risk  $R_{f}$  against inspections  $R_{ins}$  and repairs  $R_{rep}$ , all three being negative rewards. Here, we show three components with the same damage probability at time step  $t$ . When a component is not inspected nor repaired, its damage probability evolves according to a deterioration process. If a component is inspected, information from the inspection is also considered when updating the damage probability, and if a component is repaired, the damage probability resets to its initial damage distribution.
among many others [13]. Formally, the system failure risk is defined as the system failure probability multiplied by the consequences associated with a failure event, typically defined in monetary units. Due to model and measurement uncertainties, the components' damage is not perfectly known, and decisions are made based on a probability distribution over the damage condition, henceforth denoted as damage probability. The system failure probability is defined as a function of components' damage probabilities. Starting from its initial damage distribution, each component's damage probability transitions according to a deterioration stochastic process, but also according to the decisions made [13]. Naturally, the damage probability transitions based on its deterioration model when the component is neither inspected nor repaired, i.e., do-nothing action. If a component is inspected, its damage probability is updated with respect to the inspection outcome. When a component is repaired, its damage condition is directly improved and the damage probability resets to its initial damage distribution. A schematic of a typical IMP problem is shown in Figure 1.
In an effort to generate more efficient strategies for managing engineering systems through the application of cooperative MARL methods, we introduce IMP-MARL, a novel open-source suite of multi-agent environments. In IMP-MARL, each agent is responsible for managing one constituent component in a system, making decisions based on the damage probability of the component. Besides seeking to reduce component inspection and maintenance costs, agents should effectively cooperate to minimise the system failure risk. With IMP-MARL, our goal is to facilitate the definition and implementation of new customisable environments. By jointly minimising system failure risks and inspection/maintenance costs, more effective IMP policies contribute to a better allocation of resources from a societal perspective. Furthermore, additional societal impact is also made by controlling the risk of system failure events. For example, the failure of a wind turbine may affect the available electricity production. Beyond economic considerations, our proposed IMP-MARL framework can also be used to include sustainability and societal metrics within the objective function by accounting for those directly in the reward model.
To assess the capability of cooperative MARL methods for generating effective policies for IMP problems involving many components, we additionally benchmark here state-of-the-art cooperative MARL methods in terms of scalability and optimality. Most of the benchmarked methods are centralised training with decentralised execution (CTDE) methods [14, 15], in which each agent
acts based on only local information, while global information can be utilised during training. Specifically, we benchmark five CTDE methods: QMIX [16], QVMix [17], QPLEX [18], COMA [19], and FACMAC [11], along with a decentralised method, i.e., IQL [20], and a centralised one, i.e., DQN [1]. All tested MARL methods are compared against expert-based heuristic policies, which can be categorised as a state-of-the-art method to deal with IMP problems in the reliability engineering community [13, 21]. In our study, three sets of IMP environments are investigated, including one related to offshore wind structural systems, where MARL methods are tested with up to 100 agents. Additionally, these environments can be set up with two distinct reward models, and one of them incorporates explicit cooperative objectives. For the sake of enabling the reproduction of any published result, we have made our best effort to ensure that the necessary code is publicly available.
Our contributions can be outlined as follows:
- We introduce IMP-MARL, a novel open-source suite of environments, motivating the development of scalable MARL methods as well as the creation of new IMP environments, enabling the effective management of multi-component engineering systems and, as such, leading to a positive societal impact.
- In an extensive benchmark campaign, we test state-of-the-art cooperative MARL methods in very high-dimensional IMP environments featuring up to 100 agents. The resulting management strategies are evaluated against expert-based heuristic policies. We publicly provide the source code for reproducing our reported results and for easing direct comparisons with future developments.
- Based on our results, we draw relevant insights for both machine learning and reliability engineering communities, further highlighting important challenges that must still be resolved. While cooperative MARL methods can learn superior strategies compared to expert-based heuristic policies, the relative performance benefit decreases in environments with over 50 agents. In certain environments, cooperative MARL policies are characterised by a high variance and sometimes underperform expert-based heuristic policies, suggesting the need for further research efforts.
MARL environments Cooperative MARL has a long-standing history, and decentralised approaches such as IQL were already originally proposed in 1993 [20]. There has been a recent interest in the development of CTDE methods [14, 15] (see Section 4.1), inducing the creation of new environments with cooperative tasks. With continuous action spaces, popular environments include the particle environment [8], a suite of communication oriented environments with cooperative scenarios, and MaMuJoCo [11], which aims at factorising the decision of MuJoCo [22], a physics-based simulator. In contrast, the StarCraft multi-agent challenge (SMAC) [9] and its upgraded version SMACv2 [10] are probably the most studied environments with discrete action spaces. SMAC is based on the StarCraft II Learning Environment [23] with a suite of micro-management challenges where each game unit is an independent agent. Other cooperative environments based on game simulators include the Hanabi Challenge [24], a "cooperative solitaire" between two and five players, and Google Research Football [25], a football game simulator. Cooperative MARL methods are mostly benchmarked on these games and simulators, but also on real-world applications: target coverage control (MATE) [26], train scheduling (Flatland-RL) [27], traffic control (CityFlow) [28], multi-robot warehouse (RWARE) [29][30]. Oroojlooy and Hajinezhad [12] provide a review of cooperative MARL, including a more detailed list of applications. IMP-MARL introduces two key advancements: support for environments with up to 100 agents and seamless creation of diverse IMP environments. This enables the utilisation of RL in real-world scenarios, ranging from complex factories with heterogeneous components to offshore wind farms with multiple homogeneous components.
Infrastructure management planning methods Recent heuristic-based inspection and maintenance (I&M) planning methods generate IMP policies based on an optimised set of predefined decision rules [21, 31]. By evaluating only a set of decision rules out of the entire policy space, the previously mentioned approaches might yield suboptimal policies [13]. In the literature, one can also find POMDP-based methods applied to the I&M planning of engineering components, in most cases, relying on efficient point-based solvers [13, 32, 33]. When dealing with multi-component engineering systems, solving point-based POMDPs becomes computationally complex. In that case, the policy and value function can be approximated by neural networks, enabling the treatment of high-dimensional engineering systems. Specifically, actor-critic, and value function-based methods have been proposed in the literature for the management of engineering systems [34-36] with some of them relying on
CTDE methods [37, 38]. Note that no open-source methods nor publicly available environments are provided in the above-mentioned references. This emphasises the importance of our efforts to enhance comparison and reproducibility within the reliability engineering community.
3 IMP-MARL: A suite of Infrastructure Management Planning environments
In IMP, the damage condition of multiple components deteriorates stochastically over time, inducing a system failure risk that is penalised at each time step. To control the system failure risk, components can be inspected or repaired, yet, incurring additional costs. The objective is the minimisation of the expected sum of discounted costs, including inspections, repairs, and system failure risk. This can be achieved through the agents' cooperative behaviour, assigning component inspections and repairs while jointly controlling the system failure risk. The introduced IMP decision-making problem can be modelled as a decentralised partially observable Markov decision process (Dec-POMDP).
A Dec-POMDP [39] can be defined by a tuple  $[S, \mathcal{Z}, \mathcal{U}, n, O, R, P, \gamma]$ , where  $n$  agents simultaneously choose an action at every time step  $t$ . The state of the environment is  $s_t \in S$  where  $S$  is the set of states. The observation function  $O: \mathcal{S} \times \{1,.., n\} \to \mathcal{Z}$  maps the state to an observation  $o_t^a \in \mathcal{Z}$  perceived by agent  $a$  at time  $t$ , where  $\mathcal{Z}$  is the observation space. Each agent  $a \in \{1,.., n\}$  selects an action  $u_t^a \in \mathcal{U}_a$ , and the joint action space is  $\mathcal{U} = \mathcal{U}_1 \times .. \times \mathcal{U}_n$ . After the joint action  $\boldsymbol{u}_t \in \mathcal{U}$  is executed, the transition function determines the new state with probability  $P(s_{t+1} | s_t, \boldsymbol{u}_t): \mathcal{S}^2 \times \mathcal{U} \to \mathbb{R}^+$ , and  $r_t = R(s_{t+1}, s_t, \boldsymbol{u}_t): \mathcal{S}^2 \times \mathcal{U} \to \mathbb{R}$  is the team reward obtained by all agents. An agent's policy is a function  $\pi^a(u_t^a | \tau_t^a, o_t^a): (\mathcal{Z} \times \mathcal{U}_a)^t \to \mathbb{R}^+$ , which maps its history  $\tau_t^a \in (\mathcal{Z} \times \mathcal{U}_a)^{t-1}$  and its observation  $o_t^a$  to the probability of taking action  $u_t^a$ . The joint policy is denoted by  $\pi = (\pi^1,.., \pi^n)$ . The cumulative discounted reward obtained from time step  $t$  over the next  $T$  time steps is defined by  $R_t = \sum_{k=0}^{T-1} \gamma^k r_{t+k}$  and  $\gamma \in [0,1)$  is the discount factor. The goal of agents is to find the optimal joint policy that maximises the expected  $R_t$  during the entire episode:  $\pi^* = \operatorname{argmax}_{\pi} \mathbb{E}[R_0 | \pi]$ .
3.2 Environments formulation
States and observations As introduced, each agent in IMP perceives  $o_{t}^{a}$ , an observation corresponding to its respective component damage probability and the current time step. Each component damage probability transitions based on a deterioration model, defined according to physics-based engineering models, e.g., numerical simulators and/or analytical laws [32]. The damage probability is also updated based on maintenance decisions, as explained in Section 1. Since the components' damage is not perfectly known, the state of the Dec-POMDP is defined as the collection of all components' damage probabilities along with the current time step:  $s_{t} = (o_{t}^{1},..,o_{t}^{n},t)$ .
Actions and rewards Each agent controls a component and collaborates with other agents in order to minimise the system failure risk while minimising local costs associated with individual repair and/or inspection actions. At each time step  $t$ , an agent decides  $u_{t}^{a}$  between (i) do-nothing, (ii) inspect, or (iii) repair actions, as described in Section 1. Both inspection and repair actions incur significant costs, formally included in the Dec-POMDP framework as negative rewards,  $R_{ins}$  and  $R_{rep}$ , respectively. Moreover, the system failure risk is defined as  $R_{f} = c_{F} \cdot p_{F_{sys}}$  where  $p_{F_{sys}}$  is the system failure probability and  $c_{F}$  is the associated consequences of a failure event, encompassing economic, environmental, and societal losses. In IMP, we include two reward models. The first is a campaign cost model where a global cost,  $R_{camp}$ , is incurred if at least one component is inspected or repaired, plus a surplus,  $R_{ins} + R_{rep}$ , per inspected/ repaired component. This campaign cost explicitly incentivises agents to cooperate. The second is a no campaign cost model, where the campaign cost is set equal to 0 (i.e.,  $R_{camp} = 0$ ), and only component inspections and repairs costs are considered. Acting on finite-horizon episodes that span over  $T$  time steps, all agents aim at maximising the expected sum of discounted rewards  $\mathbb{E}[R_0] = \mathbb{E}\left[\sum_{t=0}^{T-1} \gamma^t [R_{t,f} + \sum_{a=1}^{n} (R_{t,ins}^a + R_{t,rep}^a) + R_{t,camp}]\right]$ .
Real-world data While IMP policies are trained based on simulated data, they policies can then be deployed to applications where real-world streams of data are available. In that case, the damage condition of the components is updated based on collected real-world data, e.g., inspections.
Q insp
campaign cost
component cost
(a) A k-out-of-n system environment.
(b) An offshore wind farm environment.
(c) Uncorrelated and correlated initial damage distribution.
(d) A campaign cost environment.
Figure 2: Visual representation of available IMP-MARL environment sets and options. In 2a, a 4-out-of-5 system fails if 2 or more components fail. In 2b, a wind turbine fails if any constituent component fails. In 2c, when the environment is under deterioration correlation, the information collected by inspecting one component also influences uninspected components. In 2d campaign cost environments, a global cost is incurred if any component is inspected and/or repaired plus a surplus per inspected/ repaired component.
3.3 IMP-MARL environments
With IMP-MARL, we provide three sets of environments to benchmark cooperative MARL methods. For all three, components are exposed to fatigue deterioration during a finite-horizon episode, inducing the growth of a crack over  $T$  time steps. The first set of environments is named  $k$ -out-of-  $n$  system and refers to systems for which a system failure occurs if (n-k+1) components fail. Those systems have been widely studied in the reliability engineering community [40]. The second type of generic environment is named correlated  $k$ -out-of-  $n$  system and is a variation of the first one for which the initial components' damage distributions are correlated. The last one is named offshore wind farm and allows the definition of environments for which a group of offshore wind turbines must be maintained. The proposed IMP-MARL environment sets and options are graphically illustrated in Figure 2, and we hereafter provide details about these sets of environments. Additionally, the deterioration processes and implementation details are formally described in Appendices B and C.
k-out-of-n system In this set of environments, the components' damage probability distribution,  $p(d_t^a)$ , is defined as a vector of 30 bins, with each bin representing a crack size interval. Here, the failure probability of one component is defined as the probability indicated in the last bin. The specificity of a k-out-of-n system is that it fails if (n-k+1) components fail, establishing a direct link between the system failure probability and the component failure probabilities. For this first system, the initial damage distribution is statistically independent among components and the time horizon is  $T = 30$  time steps. Since it is finite, we normalise each time step input and we define  $s_t = (p(d_t^1), \dots, p(d_t^n), t / T)$  and  $o_t^a = (p(d_t^a), t / T)$ . The interest of this system is that, in many practical scenarios, the reliability of an engineering system can be modelled as a  $k$ -out-of-  $n$  system.
Correlated k-out-of-n system The second set of environments is the same as the first one previously defined, with the difference that the initial damage distribution is correlated among all components. Therefore, inspecting one component also provides information about other uninspected components, depending on the specified degree of correlation. This setting is particularly challenging when approached from a decentralised scheme without providing components' correlation information to individual agents. To further address this issue, and in addition to their 30-bin local damage probability, the agents perceive correlation information  $\alpha_{t}$  common to all, which is updated based on inspection outcomes collected from all components. We thus have:  $s_t = (p(d_t^1),\dots,p(d_t^n),\alpha_t,t / T)$
and  $o_{t}^{a} = (p(d_{t}^{a}), \alpha_{t}, t / T)$ . This damage correlation structure is inspired by practical engineering applications where initial defects among components are statistically correlated due to the fact that components undergo similar manufacturing processes [13].
Offshore wind farm The third set of environments is different from the previous ones as it considers a system with a certain number of wind turbines. Specifically, each wind turbine contains three representative components: (i) the top component located in the atmospheric zone, (ii) the middle component in the underwater zone, and (iii) the mudline component submerged under the seabed. In this case, we consider that the mudline component cannot be inspected nor repaired, as it is installed under the seabed in an inaccessible region and, since only the top and middle components can be inspected or repaired, two agents are assigned for each wind turbine. Furthermore, the damage probability,  $p(d_t^a)$ , is a vector with 60 bins and transitions differently depending on the component location in the wind turbine, as corrosion-induced effects accelerate deterioration in certain areas. Besides individual component damage models, inspection techniques and their associated costs also depend on the component location: it is cheaper to inspect or repair the top components than the middle one [41]. Moreover, while the mudline component cannot be directly maintained, its damage probability also impacts the failure risk of a wind turbine. In offshore wind farm environments, a wind turbine fails if any of its three constituent components fails, and the overall system failure risk is defined as the sum of all individual wind turbine failure risks. In this case,  $p(d_t^a)$  is modelled as a 60-bin vector and the time horizon is  $T = 20$ . In this set of environments  $s_t = (p(d_t^1), \dots, p(d_t^n), t / T)$  and  $o_t^a = (p(d_t^a), t / T)$ .
Implementation All defined IMP environments are integrated with well-known MARL ecosystems, i.e., Gym [42], Gymnasium [43], PettingZoo [44] and PyMarl [9], through wrappers. The tested MARL methods are adopted from PyMarl's library, but other libraries are also compatible with our wrappers, e.g., RLlib [45], CleanRL [46], MARLlib [47], or TorchRL [48]. All developments are available on a public GitHub repository, https://github.com/moratodpg/imp_marl, featuring an open-source Apache v2 license.
4 Benchmark campaign of MARL methods
4.1 Tested methods