SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yulun Zhang
Matthew C. Fontaine
Varun Bhatt
Stefanos Nikolaidis
Jiaoyang Li
$^{1}$ Robotics Institute, Carnegie Mellon University
$^{2}$ Thomas Lord Department of Computer Science, University of Southern California
yulunzhang@cmu.edu, {mfontain, vsbhatt, nikolaid}@usc.edu,
jiaoyangli@cmu.edu
We study the problem of generating arbitrarily large environments to improve the throughput of multi-robot systems. Prior work proposes Quality Diversity (QD) algorithms as an effective method for optimizing the environments of automated warehouses. However, these approaches optimize only relatively small environments, falling short when it comes to replicating real-world warehouse sizes. The challenge arises from the exponential increase in the search space as the environment size increases. Additionally, the previous methods have only been tested with up to 350 robots in simulations, while practical warehouses could host thousands of robots. In this paper, instead of optimizing environments, we propose to optimize Neural Cellular Automata (NCA) environment generators via QD algorithms. We train a collection of NCA generators with QD algorithms in small environments and then generate arbitrarily large environments from the generators at test time. We show that NCA environment generators maintain consistent, regularized patterns regardless of environment size, significantly enhancing the scalability of multi-robot systems in two different domains with up to 2,350 robots. Additionally, we demonstrate that our method scales a single-agent reinforcement learning policy to arbitrarily large environments with similar patterns. We include the source code at https://github.com/lunjohnzhang/warehouse_env_gen_nca_public.
We study the problem of generating arbitrarily large environments to improve the throughput of multi-robot systems. As a motivating example, consider a multi-robot system for automated warehousing where thousands of robots transport inventory pods in a shared warehouse environment. While numerous works have studied the underlying Multi-Agent Path Finding (MAPF) problem [42] to more effectively coordinate the robots to improve the throughput [4, 6, 8, 24, 27, 29, 31, 35, 46], we can also optimize the throughput by designing novel warehouse environments. A well-optimized environment can alleviate traffic congestion and reduce the travel distances for the robots to fulfill their tasks in the warehouse.
A recent work [51] formulates the environment optimization problem as a Quality Diversity (QD) optimization problem and optimizes the environments by searching for the best allocation of shelf and endpoint locations (where endpoints are locations where the robots can interact with the shelves). It uses a QD algorithm to iteratively generate new environments and then repairs them with a Mixed Integer Linear Programming (MILP) solver to enforce domain-specific constraints, such as the storage capacity and connectivity of the environment. The repaired environments are evaluated with an
(a) Optimized environment with no patterns.
(b) NCA-generated environment with patterns.
Figure 1: Examples of optimized warehouse environments. Robots move in between workstations (pink) and endpoints (blue) without traveling through shelves (black) to transport goods. In this environment, the workstations on the left border are used 5 times more frequently than those on the right border. (a) shows a warehouse environment optimized directly with QD algorithms from previous work [51]. It has no obvious regularized patterns. (b) shows an environment generated by our NCA generator with regularized patterns. (c) shows a much larger environment generated by the same NCA generator in (b) with similar patterns.
(c) Scaling patterns from the NCA-generated environment to a larger environment.
agent-based simulator [28] that simulates the movements of the robots in the warehouse. Figure 1a shows an example optimized environment, which is shown to have much higher throughput and is more scalable than human-designed environments.
However, with the aforementioned method, the search space of the QD algorithm grows exponentially with the size of the environment. To optimize arbitrarily large environments, QD algorithms require a substantial number of runs in both the agent-based simulator and the MILP solver. Each run also takes more time to finish. For example, in the aforementioned work [51], it took up to 24 hours on a 64-core local machine to optimize a warehouse environment of size only  $36 \times 33$  with 200 robots, which is smaller than many warehouses in reality or warehouse simulations used in the literature. For example, Amazon fulfillment centers are reported to have more than 4,000 robots [2]. Previous research motivated by Amazon sorting centers run simulations in environments of size up to  $179 \times 69$  with up to 1,000 robots [48, 28].
Therefore, instead of optimizing the environments directly, we propose to train Neural Cellular Automata (NCA) environment generators capable of scaling their generated environments arbitrarily. Cellular automata (CA) [19] are well suited to arbitrary scaling as they incrementally construct environments through local interactions between cells. Each cell observes the state of its neighbors and incorporates that information into changing its own state. NCA [33] represents the state change rules via a neural network that can be trained with any optimization method.
We follow the insight from prior work [10] and use QD algorithms to efficiently train a diverse collection of NCA generators in small environments. We then use the NCA generators to generate arbitrarily large environments with consistent and regularized patterns. In case the generated environments are invalid, we adopt the MILP solver from the previous works [49, 15, 51] to repair the environments. Figure 1b shows an example environment with regularized patterns generated by our NCA generator and then repaired by the MILP solver. Figure 1c shows a much larger environment generated by the same NCA generator with similar patterns and then repaired by MILP. Similar to previous environment optimization methods [51], we need to run the MILP solver repeatedly on small environments (Figure 1b) to train the NCA generator. However, once the NCA generators are trained, we only run the MILP solver once after generating large environments (Figure 1c).
We show that our generated environments have competitive or better throughput and scalability compared to the existing methods [51] in small environments. In large environments where the existing
methods are not applicable due to computational limits, we show that our generated environments have significantly better throughput and scalability than human-designed environments.
We make the following contributions: (1) instead of directly searching for the best environments, we present a method to train a collection of NCA environment generators on small environments using QD algorithms, (2) we then show that the trained NCA generators can generate environments in arbitrary sizes with consistent regularized patterns, which leads to significantly higher throughput than the baseline environments in two multi-robot systems with large sizes, (3) we additionally demonstrate the general-purpose nature of our method by scaling a single-agent RL policy to arbitrarily large environments with similar patterns, maintaining high success rates.
2 Background and Related Work
2.1 Neural Cellular Automata (NCA)
Cellular automata (CA) [3, 39] originated in the artificial life community as a way to model incremental cell development. A CA consists of grid cells and an update rule for how to change a cell based on the state of its neighbors. Later work [33] showed that these rules could be encoded by a convolutional neural network. By iteratively updating a grid of cells, an NCA is capable of producing complex environments. Figure 6 in Appendix A shows an example NCA environment generation process of 50 iterations from an initial environment shown in Figure 6a.
Representing the rules of CAs by a neural network facilitates learning useful rules for the cellular automata. Several works have trained NCA generators to generate images [33], textures [36, 32], and 3D structures [43]. Other works have replaced the convolutional network with other model architectures [44, 18]. In the case of environment optimization, the objective is computed from a non-differentiable agent-based simulator. Therefore, we choose derivative-free optimization methods to train the NCA generators for environment optimization.
Prior work [10] shows that derivative-free QD algorithms can efficiently train a diverse collection of NCA video game environment generators. However, our work differs in the following ways: (1) we show how environments generated by NCA generators can scale to arbitrary sizes with similar regularized patterns, (2) in addition to encoding the constraints as part of the objective function of QD algorithms, we also use MILP to enforce the constraints on the generated environments, (3) the objective function of the previous work focuses on the reliability, diversity, and validity of the generated environments, while we primarily focus on optimizing throughput, which is a simulated agent-based metric of the multi-robot systems.
2.2 Quality Diversity (QD) Algorithms
Although derivative-free single-objective optimization algorithms such as Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) [20] have been used to optimize derivative-free objective functions, we are interested in QD algorithms because they generate a diverse collection of solutions, providing users a wider range of options in terms of the diversity measure functions.
Inspired by evolutionary algorithms with diversity optimization [7, 25, 26], QD algorithms simultaneously optimize an objective function and diversify a set of diversity measure functions to generate a diverse collection of high-quality solutions. QD algorithms have many different variants that incorporate different optimization methods, such as model-based approaches [1, 17, 50], Bayesian optimization [23], and gradient descent [11].
MAP-Elites. MAP-Elites [7, 34] constructs a discretized measure space, referred to as archive, from the user-defined diversity measure functions. It tries to find the best solution, referred to as elite, in each discretized cell. The QD-score measures the quality and diversity of the elites by summing the objective values of elites in the archive. MAP-Elites generates new solutions either randomly or by mutating existing elites. It then evaluates the solutions and replaces existing elites in the corresponding cells with superior ones. After a fixed number of iterations, MAP-Elites returns the archive with elites.
CMA-MAE. We select Covariance Matrix Adaptation MAP-Annealing (CMA-MAE) [13, 14] as our QD method for training NCA generators because of its state-of-the-art performance in continuous domains. CMA-MAE extends MAP-Elites by incorporating the self-adaptation mechanisms of
CMA-ES [20]. CMA-ES maintains a Gaussian distribution, samples from it for new solutions, evaluates them, and then updates the distribution towards the high-objective region of the search space. CMA-MAE incorporates this mechanism to optimize the QD-score. In addition, CMA-MAE implements an archive updating mechanism to balance exploitation and exploration of the measure space. The mechanism introduces a threshold value to each cell in the archive, which determines whether a new solution should be added. The threshold values are iteratively updated via an archive learning rate, with lower learning rates focusing more on exploitation. Because of the updating mechanism, some high-quality solutions might be thrown away. Therefore, CMA-MAE maintains two archives, an optimization archive implementing the updating mechanism, and a separate result archive that does not use the mechanism and stores the actual elites.
2.3 Automatic Environment Generation and Optimization
Automatic environment generation has emerged in different research communities for various purposes such as generating diverse game content [41], training more robust RL agents [40, 22], and generating novel testing scenarios [1, 12].
In the MAPF community, Zhang et al. [51] have proposed an environment optimization method based on QD algorithms MAP-Elites [7, 34] and Deep Surrogate Assisted Generation of Environments (DSAGE) [1] to improve the throughput of the multi-robot system in automated warehouses. In particular, they represent environments as tiled grids and use QD algorithms to repeatedly propose new warehouse environments by generating random environments or mutating existing environments by randomly replacing the tiles. Then they repair the proposed environments with a MILP solver and evaluate them in an agent-based simulator. The evaluated environments are added to an archive to create a collection of high-throughput environments.
However, since the proposed method directly optimizes warehouse environments, it is difficult to use the method to optimize arbitrarily large environments due to (1) the exponential growth of the search space and (2) the CPU runtime of the MILP solver and the agent-based simulator increase significantly. In comparison, we optimize environment generators instead of directly optimizing environments. In addition, we explicitly take the regularized patterns into consideration which allows us to generate arbitrarily large environments with high throughput.
3 Problem Definition
We define the environments and the corresponding environment optimization problem as follows.
Definition 1 (Environment). We represent each environment as a 2D four-neighbor grid, where each tile can be one of  $N_{type}$  tile types.  $N_{type}$  is determined by the domains.
Definition 2 (Valid Environment). An environment is valid iff the assignment of tile types satisfies its domain-specific constraints.
For example, the warehouse environments shown in Figure 1 have  $N_{type} = 4$  (endpoints, workstations, empty spaces, and shelves). One example domain-specific constraint for warehouse environments is that all non-shelf tiles should be connected so that the robots can reach all endpoints.
Definition 3 (Environment Optimization). Given an objective function  $f: \mathbf{X} \to \mathbb{R}$  and a measure function  $\mathbf{m}: \mathbf{X} \to \mathbb{R}^m$ , where  $\mathbf{X}$  is the space of all possible environments, the environment optimization problem searches for valid environments that maximize the objective function  $f$  while diversifying the measure function  $\mathbf{m}$ .
We extend previous works [10, 51] to use CMA-MAE to search for a diverse collection of NCA generators with the objective and diversity measures computed from an agent-based simulator that runs domain-specific simulations in generated environments. Figure 2 provides an overview of our method. We start by sampling a batch of  $b$  parameter vectors  $\theta$  from a multivariate Gaussian distribution, which form  $b$  NCA generators. Each NCA generator then generates one environment from a fixed initial environment, resulting in  $b$  environments. We then repair the environments using a MILP solver to enforce domain-specific constraints. After getting the repaired environments, we
Figure 2: Overview of our method of using CMA-MAE to train diverse NCA generators.
evaluate each of them by running an agent-based simulator for  $N_{e}$  times, each with  $T$  timesteps, and compute the average objective and measures. We add the evaluated environments and their corresponding NCA generators to both the optimization archive and the result archive. Finally, we update the parameters of the multivariate Gaussian distribution (i.e.,  $\mu$  and  $\Sigma$ ) and sample a new batch of parameter vectors, starting a new iteration. We run CMA-MAE iteratively with batch size  $b$ , until the total number of evaluations reaches  $N_{eval}$ .
NCA Generator. We define an NCA generator as a function  $\mathbf{g}(\mathbf{s};\theta ,C):\mathbf{X}\rightarrow \mathbf{X}$ , where  $\mathbf{X}$  is the space of all possible environments,  $\mathbf{s}\in \mathbf{X}$  is a fixed initial environment, and  $C\in \mathbb{Z}^{+}$  is a fixed number of iterations for the NCA generator.  $\mathbf{g}$  is parameterized by  $\theta \in \Theta$ . Since both  $\mathbf{s}$  and  $C$  are fixed, each parameter vector  $\theta \in \Theta$  corresponds to an NCA generator, which corresponds to an environment  $\mathbf{x}\in \mathbf{X}$ . We discretize the measure space into  $M$  cells to create an archive and attempt to find the best NCA generator in each cell to optimize the QD-score of the archive.
Our NCA generator is a convolutional neural network (CNN) with the same architecture as the previous work [10] with 3 convolutional layers of kernel size  $3 \times 3$  followed by ReLU or sigmoid activations. Figure 8 in Appendix C.1.1 shows the architecture. We represent the 2D tile-based environments as 3D tensors, with each tile as a one-hot encoded vector. Depending on the tile types, our NCA model has about  $1,500 \sim 3,000$  parameters. The input and output of the generator are one-hot environments of the same size, facilitating iterative environment generation by feeding the generator's output back into it. By using a CNN, we can use the same NCA generator to generate environments of arbitrary sizes.
MILP Repair. The environments generated by the NCA generators might be invalid. Therefore, we follow previous works [49, 15, 51] to repair the invalid environments using a MILP solver. For each unrepaired environment  $\mathbf{x}_{in} = \mathbf{g}(\mathbf{s};\boldsymbol {\theta},C)$ , we find  $\mathbf{x}_{out}\in \mathbf{X}$  such that (1) the hamming distance between  $\mathbf{x}_{in}$  and  $\mathbf{x}_{out}$  is minimized, and (2) the domain-specific constraints are satisfied. We introduce the domain-specific constraints in detail in Section 5 and Appendix B.
Objective Functions. We have two objective functions  $f_{opt}$  and  $f_{res}$  for the optimization archive and the result archive, respectively.  $f_{res}$  runs an agent-based simulator for  $N_e$  times and returns the average throughput.  $f_{opt}$  optimizes the weighted sum of  $f_{res}$  and a similarity score  $\Delta$  between the unrepaired and repaired environments  $\mathbf{x}_{in}$  and  $\mathbf{x}_{out}$ . Specifically,  $f_{opt} = f_{res} + \alpha \cdot \Delta$ , where  $\alpha$  is a hyperparameter that controls the weight between  $f_{res}$  and  $\Delta$ . We incorporate the similarity score to  $f_{opt}$  so that the NCA generators are more inclined to directly generate valid environments with desired regularized patterns, reducing reliance on the MILP solver for pattern generation. We show in Section 6 that incorporating the similarity score to  $f_{opt}$  significantly improves the scalability of the environments generated by the NCA generators.
Let  $n$  be the number of tiles in the environment, the similarity score  $\Delta$  is computed as:
$$
\Delta \left(\mathbf {x} _ {i n}, \mathbf {x} _ {o u t}\right) = \frac {\sum_ {i = 1} ^ {n} e _ {i} p _ {i}}{P}, \text {w h e r e} e _ {i} = \left\{ \begin{array}{l l} 1 & \text {i f} \left(\mathbf {x} _ {i n}\right) _ {i} = \left(\mathbf {x} _ {o u t}\right) _ {i} \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \text {a n d} P = n \cdot \max  _ {i} p _ {i} \tag {1}
$$
$e_i$  encodes if the tile type of tile  $i$  in  $\mathbf{x}_{in}$  and  $\mathbf{x}_{out}$  are the same.  $p_i$  assigns a weight to each tile. As a result, the numerator computes the unnormalized similarity score.  $P$  is the theoretical upper bound of the unnormalized similarity score of all environments of a domain. Dividing by  $P$  normalizes the score to the range of 0 to 1. Intuitively, the similarity score quantifies the level of similarity between the unrepaired and repaired environments  $\mathbf{x}_{in}$  and  $\mathbf{x}_{out}$ , with each tile weighted according to  $p_i$ .
Environment Entropy Measure. We want the NCA generators to generate environments with regularized patterns. To quantify these patterns, we introduce environment entropy as a diversity measure, inspired by previous work [30, 16]. We define a tile pattern as one possible arrangement of a  $2 \times 2$  grid in the environment. Then we count the occurrence of all possible tile patterns in the given environment, forming a tile pattern distribution. The environment entropy is calculated as the entropy of this distribution. In essence, the environment entropy measures the degree of regularization of patterns within a given environment. A lower value of environment entropy indicates a higher degree of pattern regularization. We use environment entropy as a diversity measure to find NCA generators that can generate a broad spectrum of environments of varying patterns.
5 Domains
We perform experiments in three different domains: (1) a multi-agent warehouse domain [28, 51], (2) a multi-agent manufacturing domain, and (3) a single-agent maze domain [16, 5, 1].
Warehouse. The warehouse domain simulates automated warehousing scenarios, where robots transport goods between shelves in the middle and human-operated workstations on the left and right borders. We use two variants of the warehouse domain, namely warehouse (even) and warehouse (uneven). In warehouse (even), all workstations are visited with equal frequency, while in warehouse (uneven), the workstations on the left border are visited 5 times more frequently than those on the right border. We constrain the warehouse environments such that all non-shelf tiles are connected and the number of shelves is  $N_{s}$  to keep consistent storage capability. We use  $f_{opt}$  and  $f_{res}$  introduced in Section 4 as the objective functions, and environment entropy as one of the measures. Inspired by previous work [51], we use the number of connected shelf components as the other measure. Figure 1 and Figure 7a in Appendix B show example warehouse environments.
Manufacturing. The manufacturing domain simulates automated manufacturing scenarios, where robots go to different kinds of manufacturing workstations in a fixed order to manufacture products. We create this domain to show that our method can generate environments with more tile types for more complex robot tasks. We assume 3 kinds of workstations to which the robots must visit in a fixed order and stay for different time duration. We constrain the manufacturing environments such that all non-workstation tiles are connected and there is at least one workstation of each type. We use  $f_{opt}$  and  $f_{res}$  introduced in Section 4 as the objective functions, and the environment entropy and the number of workstations as the measures. The number of workstations can approximate the cost of the manufacturing environment. By diversifying the number of workstations, our method generates a collection of environments with different price points. Figure 4 and Figure 7b in Appendix B show example manufacturing environments.
Maze. We conduct additional experiments in the maze domain to show that our method can scale a single-agent reinforcement learning (RL) policy trained in small environments to larger environments with similar patterns. The maze domain does not have domain-specific constraints. We use the same objective and measure functions in the previous work [1]. Specifically,  $f_{opt}$  and  $f_{res}$  are both binary functions that return 1 if the environment is solvable and 0 otherwise. The measures are the number of walls and the average path length of the agent over  $N_e$  simulations. Figure 5 and Figure 7c in Appendix B show example maze environments.
We include more detailed information on the domains in Appendix B.