SlowGuess/NeurIPS_2023_OCR huggingface dataset:
GPUs because training a GRU layer requires forwarding the whole episode sequentially. In contrast, the computational time can be reduced when running experiments with 50 and 100 agents on GPUs because we train all agents as a single network and the batch size increases with  $n$ . We categorise the computational time required for the reported experiments according to whether (i) the experiment is (or is not) run only on CPUs, and (ii) the value reported corresponds to the training or the testing stage. In Table 12, we additionally provide the hardware requirements demanded during the training and testing phases. Note that we benefit from more resources during testing because 10 environments are running in parallel. Moreover, we intentionally demand more RAM to avoid problems. These reported RAM configurations are indicative and can be seen as requirements, yet not as exact memory usage numbers.
We represent the computational time required for the experiments during training in Figures 4 and 5 as well as during testing in Figures 6 and 7. To avoid overloading the figures, the markers do not explicitly indicate which experiment they correspond to, yet the experiments are all vertically grouped based on the method and the environment. For each abscissa, 20 experiments, with and without campaign cost, are represented. The first three sets of experiments represent QMIX in the uncorrelated k-out-of-n setting, followed by QMIX in the correlated k-out-of-n environment which is followed by QMIX in the offshore wind farm one. The methods are ordered as QMIX, QVMIX, QPLEX, COMA, FACMAC, IQL, and DQN, while the environments are ordered as k-out-of-n setting, correlated k-out-of-n, and offshore wind farm. It can be seen in Figures 4 and 6 that the two plots with  $n < 10$  additionally represent the three additional experiments related to DQN.
The first observation that may be addressed is the resulting high variance. The variation across runs is logical because of the specific performance of the CPU/GPU models employed, but also due to the additional activity of clusters at the time of our experiments. With respect to the computational time required for training, testing episodes are not executed and we can see that, by running the experiments on only CPUs, we manage to train the agents in less than 10 hours, except for some occasional outliers. For experiments with 50 agents, and also relying on GPUs, the computational time is overall very similar to those previously mentioned, requiring less than 10 hours, yet a longer time is needed for those with 100 agents. The fastest training results correspond to COMA because we are running four environments in parallel, instead of one during training. We can see that IQL and QMIX follow closely, but QVMix, QPLEX, and FACMAC require additional computational time due to their architecture complexity. Naturally, the testing stage needs more time compared
Figure 5: Training duration for experiments with  $n > = 50$  performed on CPUs and GPUs.
Figure 6: Testing duration for experiments with  $n < = 10$  performed on only CPUs.
Figure 7: Testing duration for experiments with  $n \geq 50$  performed on CPUs and GPUs.
Figure 8: Computational time required for executing expert-based heuristic policies as a function of the number of components. The experiments are run on 2 AMD EPYC Rome 7542 CPUs @ 2.9GHz.
Figure 9: Variance analysis of a given set of neural networks. We report the standard deviation of the sum of discounted rewards obtained during the test phase. Each curve represents an entire test experiment when executing 100, 1,000, or 10,000 test episodes.
to training for all experiments because it executes 10,000 test episodes per stored network. The correlated k-out-of-n environment slightly requires more time than the others because the correlation information is updated at every inspection step.
Furthermore, we represent in Figure 8 the time required for the computation of expert-based heuristic policies. The experiments are plotted as a function of the number of components and coloured based on their corresponding environment. In this case, all experiments are run on CPUs. We can see that heuristic policies can be efficiently computed for environments with less than 50 components, yet the computational time significantly increases for experiments with 50 or 100 components. This result is logical since the combination of evaluated parameters includes the number of components to be inspected at each inspection interval. Besides, the overall computation time is directly influenced by the time needed to run an episode, with the k-out-of-n environments taking longer compared to the offshore wind farm ones because the episode's finite horizon spans over 10 additional time steps.
E.3 Statistical analysis of the variance associated with the number of test episodes
As previously explained, we conduct 10,000 test episodes to reduce the variance related to the expected sum of discounted rewards within a given environment. The choice is motivated by the direct relationship between the variance associated with  $\mathbb{E}[R_0]$  and the number of test episodes. In our experiments, we average over 10,000 policy realisations, but here, we show that the standard deviation associated with  $\mathbb{E}[R_0]$  for each training time step can vary significantly if an insufficient number of test episodes is simulated. Figure 9 illustrates the standard deviations observed when executing with 100, 1,000, and 10,000 test episodes. To produce this figure, we take the neural networks obtained with one FACMAC training run. These networks are trained over time in the offshore wind farm environment with 100 components. From this single set of networks, we execute 10 times the 100, 1,000, and 10,000 test episodes to observe how the variance evolves with this number of test episodes. We observe that the standard deviations of  $\mathbb{E}[R_0]$  obtained with 100 test episodes significantly vary over the investigated test runs. Naturally, the variation of the standard deviation is reduced with an increasing number of test episodes, obtaining very similar standard deviations when testing with 10,000 episodes.
Figure 10: Visual description of the iterative process followed to generate the boxplots showcased in Figure 3. [Left] Learning curves corresponding to 10 QMIX training seeds in a k-out-of-n system with 50 agents. The markers highlight the policies that result in the highest expected sum of discounted rewards during evaluation, i.e., one policy per seed. [Right] The 10 selected policies are displayed at the top as a function of the expected sum of discounted rewards, along with the heuristic score obtained in this environment. In the middle plot, we calculate and represent the 10 selected policies as a function of normalised relative rewards, i.e.,  $(\mathrm{x - h}) / \mathrm{h}$ . Finally, a boxplot is constructed at the bottom based on the previously calculated ten normalised relative rewards, each representing a different seed.
Moreover, one can also compute the largest difference that is observed between these 10 test runs at a specific time step i.e., the absolute difference of the estimated  $\mathbb{E}[R_0]$  at a given time step. When testing 100 episodes, the absolute difference is 231.89 at a time step where the maximum of the 10 provided  $\mathbb{E}[R_0]$  is  $-2786.1$ . If 1000 episodes are tested, the absolute difference is 99.8 where the maximum is  $-2757.2$  at this time step, whereas by testing 10,000 episodes, the difference equals 24.8 with a maximum of  $-2622.8$ . These numbers represent only a trained set of networks over 1920 training runs, and thus a final conclusion cannot be claimed, yet it motivates the need of simulating 10,000 test episodes, as with only 100 test episodes, the absolute difference can reach up to  $10\%$ .
F Additional benchmark results
In this section, we present additional results and remarks beyond those reported in the main text. In the first place, we provide the values of the expected sum of discounted rewards achieved by the best runs over all experiments. We list the best policy for each conducted experiment in Tables 13, 14, and 15. Note that the maximum values represented with markers at the right of each box in Figure 3 can be retrieved from these values by applying the normalisation (x-H)/H, with H being the value achieved by the heuristic policies. In Figure 10, we also visually illustrate the normalisation process that results in the boxplots presented in Figure 3.
Additionally, we represent in Figures 11 and 12 the learning curves corresponding to all our experiments. The learning curves showcase the evolution of the expected sum of discounted rewards every 20,000 training time steps, computed at the testing stage with 10,000 test episodes. Since the training is conducted with 10 different seeds for each environment and method, we also plot the corresponding 25th-75th percentiles around the median. These results confirm the variance observed between the best results and presented in Figure 3.
Based on Tables 13 and 14, one may additionally infer that correlated environments result in lower costs with respect to those uncorrelated. This is especially true for environments with  $n >= 10$  agents, specified without campaign costs, and in all environments set up with campaign costs. While MARL methods profit from the additionally provided correlation information, this is not always the case for the heuristic policies.
One final remark is that the discrepancy between the expert-based heuristic policy and MARL methods is more pronounced in offshore wind farm environments. This could be attributed to the shorter decision horizon or the higher cost per inspection in this particular case (see Table 15).
Table 13: k-out-of-n system best policies ( $^{*}$  = campaign cost).
n	QMIX	QVMix	QPLEX	COMA	FACMAC	IQL	DQN	Heuristics
3	-9.7	-9.8	-9.7	-10.6	-10.4	-35.3	-9.9	-12.5
5	-20.4	-20.7	-20.4	-21.8	-22.1	-108.7	-24.0	-25.2
10	-51.0	-51.5	-51.0	-54.3	-61.3	-404.5	/	-63.7
50	-229.7	-236.0	-212.8	-1190.6	-249.0	-1991.1	/	-268.1
100	-222.6	-230.7	-220.6	-1770.1	-225.7	-1770.1	/	-262.4
*3	-14.6	-14.7	-14.7	-15.0	-17.0	-35.3	-13.5	-15.1
*5	-27.4	-27.7	-27.4	-28.9	-33.0	-27.8	-26.6	-28.6
*10	-58.9	-63.0	-60.7	-70.0	-61.9	-404.5	/	-64.5
*50	-169.5	-173.9	-168.4	-241.4	-160.7	-623.3	/	-232.7
*100	-167.2	-175.8	-160.2	-1770.1	-144.8	-1770.1	/	-231.5
Table 14: Correlated k-out-of-n system best policies ( $^{*} =$  campaign cost).
n	QMIX	QVMix	QPLEX	COMA	FACMAC	IQL	DQN	Heuristics
3	-9.7	-9.7	-9.6	-11.0	-10.6	-10.0	-10.0	-13.0
5	-20.4	-20.6	-18.4	-21.2	-21.6	-20.2	-23.4	-28.1
10	-47.6	-51.0	-45.2	-49.7	-46.1	-374.5	/	-67.7
50	-214.3	-233.0	-212.3	-419.3	-143.4	-1339.9	/	-240.0
100	-250.3	-289.0	-276.8	-486.9	-118.3	-1744.0	/	-218.1
*3	-13.1	-12.9	-12.9	-14.8	-18.0	-34.7	-12.6	-15.2
*5	-23.5	-24.7	-23.5	-28.2	-29.2	-23.9	-26.8	-30.5
*10	-56.2	-53.4	-50.1	-52.8	-49.2	-56.0	/	-68.5
*50	-132.6	-157.1	-121.2	-159.3	-106.6	-814.9	/	-211.0
*100	-147.7	-147.5	-121.0	-339.1	-71.3	-723.8	/	-194.0
Table 15: Offshore wind farm best policies ( $^{*}$  = campaign cost).
n	QMIX	QVMix	QPLEX	COMA	FACMAC	IQL	DQN	Heuristics
2	-23.3	-23.3	-23.2	-23.7	-40.5	-23.7	-23.2	-58.3
4	-47.1	-47.4	-47.1	-47.9	-122.4	-47.4	-47.7	-116.9
10	-118.4	-119.4	-118.5	-122.2	-235.2	-120.8	/	-292.3
50	-604.4	-613.9	-604.6	-2805.8	-627.3	-2892.5	/	-1463.8
100	-1224.1	-1238.8	-1213.2	-5785.1	-1625.2	-5785.1	/	-2925.0
*2	-51.8	-52.0	-51.9	-60.1	-60.3	-52.0	-48.9	-62.2
*4	-80.5	-80.7	-80.7	-122.2	-118.6	-85.6	-76.0	-115.2
*10	-129.3	-133.3	-130.0	-314.5	-196.4	-132.0	/	-267.2
*50	-432.9	-436.9	-434.5	-2892.5	-502.8	-1709.7	/	-1248.2
*100	-808.1	-829.0	-852.3	-5785.1	-1280.5	-5785.1	/	-2436.3
Figure 11: Learning curves in all environments with no campaign cost. Curves represent the sum of discounted rewards obtained during test time. The bold line is the median while the error bands are delimited by the 25th and 75th percentiles. Colours represent the different methods and the parameters of each environment can be inferred from the title above its corresponding graph.
Figure 12: Learning curves in all environments with campaign cost. Curves represent the sum of discounted rewards obtained during test time. The bold line is the median while the error bands are delimited by the 25th and 75th percentiles. Colours represent the different methods and the parameters of each environment can be inferred from the title above its corresponding graph.