SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Domain	S	Seval	Ns	Ns_eval	Na	Na_eval	Ne	b	Neval
Warehouse (even)	36 × 33	101 × 102	240	2,250	200	1,400	5	50	10,000
Warehouse (uneven)	36 × 33	101 × 102	240	2,250	200	1,000	5	50	10,000
Manufacturing	36 × 33	101 × 102	N/A	N/A	200	1,800	5	50	10,000
Maze	18 × 18	66 × 66	N/A	N/A	1	1	50	150	100,000
Table 1: Summary of the experiment setup. Columns 2-7 show the configurations related to the environment, and columns 8-10 show the parameters of CMA-MAE.  $S$ ,  $N_{s}$ , and  $N_{a}$  are the size of the environments, the number of shelves in the warehouse domain, and the number of agents used in training, respectively.  $S_{eval}$ ,  $N_{s\_eval}$ , and  $N_{a\_eval}$  are their counterparts used in evaluation. We choose  $N_{a\_eval}$  to be large enough such that the human-designed environments of the same size are congested while our NCA-generated ones are not. Therefore, the value of  $N_{a\_eval}$  depends on the tasks of the robots in each domain.
6 Experimental Evaluation
Table 1 summarizes the experiment setup. We train the NCA generators with environments of size  $S$  and then evaluate them in sizes of both  $S$  and  $S_{eval}$ . In addition, we set the number of NCA iterations  $C = 50$  for environments of size  $S$  and  $C_{eval} = 200$  for those of size  $S_{eval}$  for all domains.  $C_{eval}$  is larger than  $C$  because the NCA generators need more iterations to converge to a stable state while generating environments of size  $S_{eval}$ . For the warehouse and manufacturing domains, we use Rolling-Horizon Collision Resolution (RHCR) [28], a state-of-the-art centralized lifelong MAPF planner, in the simulation. We run each simulation for  $T = 1,000$  timesteps during training and  $T_{eval} = 5,000$  timesteps during evaluation and stop early in case of congestion, which happens if more than half of the agents take wait actions at the same timestep. For the maze domain, we use a trained ACCEL agent [9] in the simulations. We include more details of the experiment setup, compute resources, and implementation in Appendix C.
6.1 Multi-Agent Domains
Baseline Environments. We consider two types of baseline environments for the multi-agent domains (warehouse and manufacturing), namely those designed by human and those optimized by DSAGE [51], the state-of-the-art environment optimization method. In DSAGE, we use  $f_{res}$  as the objective function for the result archive and  $f_{opt}$  with  $\alpha = 5$  for the surrogate archive. With size  $S_{eval}$ , however, it is impractical to run DSAGE due to the computational limit. One single run of MILP repair, for example, could take 8 hours in an environment of size  $S_{eval}$ . We use the human-designed warehouse environments from previous work [29, 28, 51]. The manufacturing domain is new, so we create human-designed environments for it.
NCA-generated Environments. We take the best NCA generator from the result archive to generate environments of size  $S$  and  $S_{eval}$ . In the manufacturing domain, we additionally take the best NCA generator that generate an environment of size  $S$  with a similar number of workstations as the DSAGE-optimized environment. This is to ensure a fair comparison, as we do not constrain the number of workstations, which is correlated with throughput, in the manufacturing domain.
We show more details of the baselines and NCA-generated environments in Appendix D.
6.1.1 Environments of Size  $S$
Columns 3 and 4 of Table 2 show the numerical results with  $N_{a} = 200$  agents in the warehouse and manufacturing domains with environment size  $S$ . All NCA-generated environments have significantly better throughput than the baseline environments. In addition, we observe that larger  $\alpha$  values have no significant impact on the throughput with  $N_{a}$  agents.
We also analyze the scalability of the environments by running simulations with varying numbers of agents and show the throughput in Figures 3a to 3c. In both the warehouse (even) and manufacturing domains, our NCA-generated environments have higher throughput than the baseline environments with an increasingly larger gap with more agents. In the warehouse (uneven) domain, however, the DSAGE-optimized environment displays slightly better scalability than those generated by NCA. This could be due to a need for more traversable tiles near the popular left-side workstations to alleviate congestion, a pattern creation task that poses a significant challenge for NCA generators
Size S with Na agents	Size Seval with Na_eval agents
Domain	Algorithm	Success Rate	Throughput	Success Rate	Throughput
warehouse (even)	CMA-MAE + NCA (α = 0)	100%	6.79 ± 0.00	0%	N/A
CMA-MAE + NCA (α = 1)	100%	6.73 ± 0.00	0%	N/A
CMA-MAE + NCA (α = 5)	100%	6.74 ± 0.00	90%	16.01 ± 0.00
DSAGE (α = 5)	100%	6.35 ± 0.00	N/A	N/A
Human	0%	N/A	0%	N/A
warehouse (uneven)	CMA-MAE + NCA (α = 0)	100%	6.89 ± 0.00	62%	12.32 ± 0.00
CMA-MAE + NCA (α = 1)	100%	6.70 ± 0.00	8%	11.56 ± 0.01
CMA-MAE + NCA (α = 5)	100%	6.82 ± 0.00	84%	12.03 ± 0.00
DSAGE (α = 5)	100%	6.40 ± 0.00	N/A	N/A
Human	0%	N/A	0%	N/A
Manufacturing	CMA-MAE + NCA (α = 5, opt)	94%	6.82 ± 0.00	100%	23.11 ± 0.01
CMA-MAE + NCA (α = 5, comp DSAGE)	98%	6.61 ± 0.00	N/A	N/A
DSAGE (α = 5)	28%	5.61 ± 0.12	N/A	N/A
Human	100%	5.92 ± 0.00	0%	N/A
Table 2: Success rates and throughput of environments of sizes  $S$  and  ${S}_{\text{eval }}$  . We run 50 simulations for all environments except for the manufacturing environments of size  ${S}_{\text{eval }}$  ,for which we run 20 simulations. The success rate is calculated as the percentage of simulations that end without congestion. We measure the throughput of only successful simulations and report both its average and standard error.
(d)  $S_{eval}$ : Warehouse (even)
(e)  $S_{eval}$ : Warehouse (uneven)
(f)  $S_{eval}$ : Manufacturing
Figure 3: Throughput with an increasing number of agents in environments of size  $S$  (a-c) and  $S_{eval}$  (d-f). For size  $S$ , we run 50 simulations and increase the number of agents by a step size of 10. For  $S_{eval}$ , we run 50 and 20 simulations and increase the number of agents by step sizes of 25 and 50 in warehouse and manufacturing domains, respectively. The solid lines are the average throughput while the shaded area shows the  $95\%$  confidence interval.
compared to DSAGE's direct tile search. In addition, in the manufacturing domain, the optimal NCA generator (green line) creates an environment with top throughput for 200 agents, but it is slightly less scalable than the sub-optimal one used for DSAGE comparison (grey line). We conjecture this is due to the optimizer being caught in a local optimum region due to the complexity of the tasks. On the effect of  $\alpha$  values, we observe that larger  $\alpha$  values enhance the scalability of the generated
Figure 4: Example manufacturing environment.
(a)  $18\times 18$
Figure 5: Maze environments with similar patterns of different sizes generated by the same NCA generator.
(b)  $66\times 66$
environments in the warehouse (even) domain, as shown in Figure 3a. We include further discussion of  $\alpha$  values in Appendix E.1.
To further understand the gap of throughput among different environments of size  $S$  with  $N_{a}$  agents, we present additional results on the number of finished tasks over time in Appendix E.2. All NCA-generated environments maintain a stable number of finished tasks throughout the simulation.
6.1.2 Environments of Size  $S_{eval}$
Columns 5 and 6 of Table 2 compare environments of size  $S_{eval}$  with  $N_{a\_eval}$  agents. In the warehouse (even) domain, all environments, except for the one generated by CMA-MAE + NCA  $(\alpha = 5)$ , run into congestion. In the warehouse (uneven) domain, CMA-MAE + NCA  $(\alpha = 5)$  also achieves the best success rate, even though it does not have the highest throughput. This shows the benefit of using a larger  $\alpha$  value with CMA-MAE in creating large and scalable environments.
To further understand the scalability, Figures 3d to 3f show the throughput of the environments with an increasing number of agents. We observe that all our NCA-generated environments are much more scalable than the human-designed ones. In particular, the environments generated by CMA-MAE + NCA ( $\alpha = 5$ ) yield the best scalability in all domains. This is because a higher value of  $\alpha$  makes CMA-MAE focus more on minimizing the disparity between the unrepaired and repaired environments. As a result, the optimized NCA generators are more capable of generating environments with desired regularized patterns directly. On the other hand, a smaller value of  $\alpha$  makes the algorithm rely more on the MILP solver to generate patterns, introducing more randomness as the MILP solver arbitrarily selects an optimal solution with minimal hamming distance when multiple solutions exist. We show additional results of scaling the same NCA generators to more environment sizes in Appendix E.3. We also include a more competitive baseline by tiling environments of size  $S$  to create those of size  $S_{eval}$  in Appendix E.4.
6.2 Scaling Single-Agent RL Policy
In the maze domain, we show that it is possible to scale a single-agent RL policy to larger environments with similar regularized patterns. We use the NCA generator trained from  $18 \times 18$  maze environments to generate a small  $18 \times 18$  and a large  $66 \times 66$  maze environment, shown in Figure 5. Running the ACCEL [9] agent 100 times in these environments results in success rates of  $90\%$  and  $93\%$ , respectively. The high success rate comes from the fact that the similar local observation space of the RL policy enables the agent to make the correct local decision and eventually arrive at the goal.
For comparison, we generate 100 baseline  $66 \times 66$  maze environments by restricting the wall counts to be the same and the path lengths to range between  $80\%$  to  $120\%$  of the one in Figure 5b, to maintain similar difficulty levels. Figure 15 in Appendix D.2 shows two example baseline environments. Testing the ACCEL agent in these environments yields an average success rate of  $22\%$  (standard error:  $3\%$ ), significantly lower than that achieved in the NCA-generated environments. This demonstrates
Size S with Na agents	Size Seval with Na_eval agents
Domain	Algorithm	TNCA	TMILP	Δ	TNCA	TMILP	Δ
Warehouse (even)	CMA-MAE (α = 0)	0.06	2.72	0.94	0.12	228.80	0.95
CMA-MAE (α = 1)	0.06	2.97	0.86	0.13	9278.18	0.84
CMA-MAE (α = 5)	0.06	2.55	0.97	0.13	19.79	0.99
Warehouse (uneven)	CMA-MAE (α = 0)	0.06	2.41	0.85	0.13	1,376.62	0.85
CMA-MAE (α = 1)	0.06	2.36	0.95	0.13	26.54	0.96
CMA-MAE (α = 5)	0.06	13.42	0.84	0.13	15,656.04	0.86
Manufacturing	CMA-MAE (α = 5, opt)	0.07	3.04	0.17	0.18	15,746.10	0.18
Table 3: CPU runtime measured (in seconds) for the trained NCA generators to generate environments ( $T_{NCA}$ ), and for the MILP solver to subsequently repair the generated environments ( $T_{MILP}$ ) of size  $S$  and  $S_{eval}$ .  $\Delta$  refers to the similarity score between the repaired and unrepaired environments. We measure the CPU runtime in machine (1) listed in Appendix C.3 and compute constraints in Appendix C.4.
the ability of our method to generate arbitrarily large environments with similar patterns in which the trained RL agent performs well.
6.3 NCA Generation Time
We show the total CPU runtime of generating and repairing the environments of sizes  $S$  and  $S_{eval}$  in Table 3. In both sizes, larger similarity scores are correlated with shorter MILP runtime. The similarity scores of the manufacturing environments are significantly lower because we use a larger normalization factor  $P$  than that in the warehouse domains. Nevertheless, we note that even the longest combined runtime of the NCA generation and MILP repair (15, 746.10 + 0.18 = 15, 746.28 seconds for the manufacturing domain) on the environments of size  $S_{eval}$  is significantly shorter than optimizing the environments of that size directly. This reduction in time is attributed to our method, which involves running the MILP repair  $N_{eval}$  times with an environment size of  $S$  while training the NCA generators, and then using the MILP solver only once with size  $S_{eval}$  while generating the environments. Our method stands in contrast to the previous direct environment optimization method [51] which requires running the MILP repair with the size of  $S_{eval}$  for  $N_{eval}$  times.
6.4 On the Benefit of QD Algorithms
QD algorithms can train a diverse collection of NCA generators. In Appendix E.5, we compare the QD-score and archive coverage of CMA-MAE + NCA and DSAGE. We also benchmark CMA-MAE against MAP-Elites [34, 47] to highlight the benefit of using CMA-MAE in training NCA generators. We observe that CMA-MAE + NCA has the best QD-score and archive coverage in all domains.
Furthermore, QD algorithms can train more scalable NCA generators than single-objective optimizers. In Appendix E.6 we compare CMA-MAE with a single-objective optimizer CMA-ES [20]. We observe that CMA-MAE is less prone to falling into local optima and thus better NCA generators.
We present a method to train a diverse collection of NCA generators that can generate environments of arbitrary sizes within a range defined by finite compute resources. In small environments of size  $S$ , our method generates environments that have competitive or better throughput compared to the state-of-the-art environment optimization method [51]. In large environments of size  $S_{eval}$ , where existing optimization methods are inapplicable, our generated environments achieve significantly higher throughput and better scalability compared to human-designed environments.
Our work is limited in many ways, yielding many directions for future work. First, our method is only arbitrarily scalable under a range determined by finite compute resources. For example, with the compute resource in Appendices C.3 and C.4, the upper bound of the range is about  $S_{eval} = 101 \times 102$ . Although this upper bound is significantly higher than previous environment optimization methods, future work can seek to further increase it. Second, we focus on generating 4-neighbor grid-based environments with an underlying undirected movement graph. Future work can consider generating similar environments with directed movement graphs or irregular-shaped environments. We additionally discuss social impacts of our work in Appendix F.
This work used Bridge-2 at Pittsburgh Supercomputing Center (PSC) through allocation CIS220115 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. In addition, this work was supported by the CMU Manufacturing Futures Institute, made possible by the Richard King Mellon Foundation, and was partially supported by the NSF CAREER Award (#2145077).
[1] Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 37762-37777, 2022.
[2] Alan S. Brown. How Amazon robots navigate congestion. https://www.amazon.science/latest-news/how-amazon-robots-navigate-congestion, 2022. Accessed: 2023-05-09.
[3] Bert Wang-Chak Chan. Lenia and expanded universe. In Proceedings of the Conference on Artificial Life (ALIFE), pages 221-229, 2020.
[4] Zhe Chen, Javier Alonso-Mora, Xiaoshan Bai, Daniel D Harabor, and Peter J Stuckey. Integrated task assignment and path planning for capacitated multi-agent pickup and delivery. IEEE Robotics and Automation Letters, 6(3):5816-5823, 2021.
[5] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018.
[6] Antonello Contini and Alessandro Farinelli. Coordination approaches for multi-item pickup and delivery in logistic scenarios. Robotics and Autonomous Systems, 146:103871, 2021.
[7] Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):503-507, 2015.
[8] Mehul Damani, Zhiyao Luo, Emerson Wenzel, and Guillaume Sartoretti. PRIMAL₂: Pathfinding via reinforcement and imitation multi-agent learning - lifelong. IEEE Robotics and Automation Letters, 6(2):2666-2673, 2021.
[9] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 13049-13061, 2020.
[10] Sam Earle, Justin Snider, Matthew C. Fontaine, Stefanos Nikolaidis, and Julian Togelius. Illuminating diverse neural cellular automata for level generation. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), pages 68-76, 2022.
[11] Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 10040–10052, 2021.
[12] Matthew C. Fontaine and Stefanos Nikolaidis. A quality diversity approach to automatically generating human-robot interaction scenarios in shared autonomy. In Proceedings of the Robotics: Science and Systems (RSS), 2021.
[13] Matthew Fontaine and Stefanos Nikolaidis. Covariance matrix adaptation map-annealing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), pages 456-465, 2023.
[14] Matthew C. Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), pages 94-102, 2020.
[49] Hejia Zhang, Matthew C. Fontaine, Amy K. Hoover, Julian Togelius, Bistra Dilkina, and Stefanos Nikolaidis. Video game level repair via mixed integer linear programming. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), pages 151-158, 2020.
[50] Yulun Zhang, Matthew C. Fontaine, Amy K. Hoover, and Stefanos Nikolaidis. Deep surrogate assisted map-elites for automated hearthstone deckbuilding. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), pages 158-167, 2022.
[51] Yulun Zhang, Matthew C. Fontaine, Varun Bhatt, Stefanos Nikolaidis, and Jiaoyang Li. Multirobot coordination and layout design for automated warehousing. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 5503-5511, 2023.