Title: Arbitrarily Scalable Environment Generators via Neural Cellular Automata
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-robot systems

quality diversity

automatic environment generation

neural cellular automata
Abstract: We study the problem of generating arbitrarily large environments to improve the throughput of multi-robot systems. Prior work proposes Quality Diversity (QD) algorithms as an effective method for optimizing the environments of automated warehouses. However, these approaches optimize only relatively small environments, falling short when it comes to replicating real-world warehouse sizes. The challenge arises from the exponential increase in the search space as the environment size increases. Additionally, the previous methods have only been tested with up to 350 robots in simulations, while practical warehouses could host thousands of robots. In this paper, instead of optimizing environments, we propose to optimize Neural Cellular Automata (NCA) environment generators via QD algorithms. We train a collection of NCA generators with QD algorithms in small environments and then generate arbitrarily large environments from the generators at test time. We show that NCA environment generators maintain consistent, regularized patterns regardless of environment size, significantly enhancing the scalability of multi-robot systems in two different domains with up to 2,350 robots. Additionally, we demonstrate that our method scales a single-agent reinforcement learning policy to arbitrarily large environments with similar patterns. We include the source code at https://github.com/lunjohnzhang/warehouse_env_gen_nca_public.

Arbitrarily Scalable Environment Generators via Neural Cellular Automata

Abstract

We study the problem of generating arbitrarily large environments to improve the throughput of multi-robot systems. Prior work proposes Quality Diversity (QD) algorithms as an effective method for optimizing the environments of automated warehouses. However, these approaches optimize only relatively small environments, falling short when it comes to replicating real-world warehouse sizes. The challenge arises from the exponential increase in the search space as the environment size increases. Additionally, the previous methods have only been tested with up to 350 robots in simulations, while practical warehouses could host thousands of robots. In this paper, instead of optimizing environments, we propose to optimize Neural Cellular Automata (NCA) environment generators via QD algorithms. We train a collection of NCA generators with QD algorithms in small environments and then generate arbitrarily large environments from the generators at test time. We show that NCA environment generators maintain consistent, regularized patterns regardless of environment size, significantly enhancing the scalability of multi-robot systems in two different domains with up to 2,350 robots. Additionally, we demonstrate that our method scales a single-agent reinforcement learning policy to arbitrarily large environments with similar patterns. We include the source code at https://github.com/lunjohnzhang/warehouse_ env_gen_nca_public .

1 Introduction

We study the problem of generating arbitrarily large environments to improve the throughput of multirobot systems. As a motivating example, consider a multi-robot system for automated warehousing where thousands of robots transport inventory pods in a shared warehouse environment. While numerous works have studied the underlying Multi-Agent Path Finding (MAPF) problem [42] to more effectively coordinate the robots to improve the throughput [4, 6, 8, 24, 27, 29, 31, 35, 46], we can also optimize the throughput by designing novel warehouse environments. A well-optimized environment can alleviate traffic congestion and reduce the travel distances for the robots to fulfill their tasks in the warehouse.

A recent work [51] formulates the environment optimization problem as a Quality Diversity (QD) optimization problem and optimizes the environments by searching for the best allocation of shelf and endpoint locations (where endpoints are locations where the robots can interact with the shelves). It uses a QD algorithm to iteratively generate new environments and then repairs them with a Mixed Integer Linear Programming (MILP) solver to enforce domain-specific constraints, such as the storage capacity and connectivity of the environment. The repaired environments are evaluated with an

2

(a) Optimized environ-

ment with no patterns.

Empty space

Endpoint

Shelf

Workstation

(b) NCA-generated en- vironment with patterns.

(c) Scaling patterns from the NCA-generated envi- ronment to a larger environment.

agent-based simulator [28] that simulates the movements of the robots in the warehouse. Figure 1a shows an example optimized environment, which is shown to have much higher throughput and is more scalable than human-designed environments.

However, with the aforementioned method, the search space of the QD algorithm grows exponentially with the size of the environment. To optimize arbitrarily large environments, QD algorithms require a substantial number of runs in both the agent-based simulator and the MILP solver. Each run also takes more time to finish. For example, in the aforementioned work [51], it took up to 24 hours on a 64-core local machine to optimize a warehouse environment of size only 36 × 33 with 200 robots, which is smaller than many warehouses in reality or warehouse simulations used in the literature. For example, Amazon fulfillment centers are reported to have more than 4,000 robots [2]. Previous research motivated by Amazon sortation centers run simulations in environments of size up to 179 × 69 with up to 1,000 robots [48, 28].

Therefore, instead of optimizing the environments directly, we propose to train Neural Cellular Automata (NCA) environment generators capable of scaling their generated environments arbitrarily. Cellular automata (CA) [19] are well suited to arbitrary scaling as they incrementally construct environments through local interactions between cells. Each cell observes the state of its neighbors and incorporates that information into changing its own state. NCA [33] represents the state change rules via a neural network that can be trained with any optimization method.

We follow the insight from prior work [10] and use QD algorithms to efficiently train a diverse collection of NCA generators in small environments. We then use the NCA generators to generate arbitrarily large environments with consistent and regularized patterns. In case the generated environments are invalid, we adopt the MILP solver from the previous works [49, 15, 51] to repair the environments. Figure 1b shows an example environment with regularized patterns generated by our NCA generator and then repaired by the MILP solver. Figure 1c shows a much larger environment generated by the same NCA generator with similar patterns and then repaired by MILP. Similar to previous environment optimization methods [51], we need to run the MILP solver repeatedly on small environments (Figure 1b) to train the NCA generator. However, once the NCA generators are trained, we only run the MILP solver once after generating large environments (Figure 1c).

We show that our generated environments have competitive or better throughput and scalability compared to the existing methods [51] in small environments. In large environments where the existing

methods are not applicable due to computational limits, we show that our generated environments have significantly better throughput and scalability than human-designed environments.

We make the following contributions: (1) instead of directly searching for the best environments, we present a method to train a collection of NCA environment generators on small environments using QD algorithms, (2) we then show that the trained NCA generators can generate environments in arbitrary sizes with consistent regularized patterns, which leads to significantly higher throughput than the baseline environments in two multi-robot systems with large sizes, (3) we additionally demonstrate the general-purpose nature of our method by scaling a single-agent RL policy to arbitrarily large environments with similar patterns, maintaining high success rates.

2 Background and Related Work

2.1 Neural Cellular Automata (NCA)

Cellular automata (CA) [3, 39] originated in the artificial life community as a way to model incremental cell development. A CA consists of grid cells and an update rule for how to change a cell based on the state of its neighbors. Later work [33] showed that these rules could be encoded by a convolutional neural network. By iteratively updating a grid of cells, an NCA is capable of producing complex environments. Figure 6 in Appendix A shows an example NCA environment generation process of 50 iterations from an initial environment shown in Figure 6a.

Representing the rules of CAs by a neural network facilitates learning useful rules for the cellular automata. Several works have trained NCA generators to generate images [33], textures [36, 32], and 3D structures [43]. Other works have replaced the convolutional network with other model architectures [44, 18]. In the case of environment optimization, the objective is computed from a non-differentiable agent-based simulator. Therefore, we choose derivative-free optimization methods to train the NCA generators for environment optimization.

Prior work [10] shows that derivative-free QD algorithms can efficiently train a diverse collection of NCA video game environment generators. However, our work differs in the following ways: (1) we show how environments generated by NCA generators can scale to arbitrary sizes with similar regularized patterns, (2) in addition to encoding the constraints as part of the objective function of QD algorithms, we also use MILP to enforce the constraints on the generated environments, (3) the objective function of the previous work focus on the reliability, diversity, and validity of the generated environments, while we primarily focus on optimizing throughput, which is a simulated agent-based metric of the multi-robot systems.

2.2 Quality Diversity (QD) Algorithms

Although derivative-free single-objective optimization algorithms such as Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) [20] have been used to optimize derivative-free objective functions, we are interested in QD algorithms because they generate a diverse collection of solutions, providing users a wider range of options in terms of the diversity measure functions.

Inspired by evolutionary algorithms with diversity optimization [7, 25, 26], QD algorithms simultaneously optimize an objective function and diversify a set of diversity measure functions to generate a diverse collection of high-quality solutions. QD algorithms have many different variants that incorporate different optimization methods, such as model-based approaches [1, 17, 50], Bayesian optimization [23], and gradient descent [11].

MAP-Elites. MAP-Elites [7, 34] constructs a discretized measure space, referred to as archive , from the user-defined diversity measure functions. It tries to find the best solution, referred to as elite , in each discretized cell. The QD-score measures the quality and diversity of the elites by summing the objective values of elites in the archive. MAP-Elites generates new solutions either randomly or by mutating existing elites. It then evaluates the solutions and replaces existing elites in the corresponding cells with superior ones. After a fixed number of iterations, MAP-Elites returns the archive with elites.

CMA-MAE. We select Covariance Matrix Adaptation MAP-Annealing (CMA-MAE) [13, 14] as our QD method for training NCA generators because of its state-of-the-art performance in continuous domains. CMA-MAE extends MAP-Elites by incorporating the self-adaptation mechanisms of

CMA-ES [20]. CMA-ES maintains a Gaussian distribution, samples from it for new solutions, evaluates them, and then updates the distribution towards the high-objective region of the search space. CMA-MAE incorporates this mechanism to optimize the QD-score. In addition, CMA-MAE implements an archive updating mechanism to balance exploitation and exploration of the measure space. The mechanism introduces a threshold value to each cell in the archive, which determines whether a new solution should be added. The threshold values are iteratively updated via an archive learning rate, with lower learning rates focusing more on exploitation. Because of the updating mechanism, some high-quality solutions might be thrown away. Therefore, CMA-MAE maintains two archives, an optimization archive implementing the updating mechanism, and a separate result archive that does not use the mechanism and stores the actual elites.

2.3 Automatic Environment Generation and Optimization

Automatic environment generation has emerged in different research communities for various purposes such as generating diverse game content [41], training more robust RL agents [40, 22], and generating novel testing scenarios [1, 12].

In the MAPF community, Zhang et al. [51] have proposed an environment optimization method based on QD algorithms MAP-Elites [7, 34] and Deep Surrogate Assisted Generation of Environments (DSAGE) [1] to improve the throughput of the multi-robot system in automated warehouses. In particular, they represent environments as tiled grids and use QD algorithms to repeatedly propose new warehouse environments by generating random environments or mutating existing environments by randomly replacing the tiles. Then they repair the proposed environments with a MILP solver and evaluate them in an agent-based simulator. The evaluated environments are added to an archive to create a collection of high-throughput environments.

However, since the proposed method directly optimizes warehouse environments, it is difficult to use the method to optimize arbitrarily large environments due to (1) the exponential growth of the search space and (2) the CPU runtime of the MILP solver and the agent-based simulator increase significantly. In comparison, we optimize environment generators instead of directly optimizing environments. In addition, we explicitly take the regularized patterns into consideration which allows us to generate arbitrarily large environments with high throughput.

3 Problem Definition

We define the environments and the corresponding environment optimization problem as follows.

Definition 1 (Environment) . We represent each environment as a 2D four-neighbor grid, where each tile can be one of N type tile types. N type is determined by the domains.

Definition 2 (Valid Environment) . An environment is valid iff the assignment of tile types satisfies its domain-specific constraints.

For example, the warehouse environments shown in Figure 1 have N type = 4 (endpoints, workstations, empty spaces, and shelves). One example domain-specific constraint for warehouse environments is that all non-shelf tiles should be connected so that the robots can reach all endpoints.

Definition 3 (Environment Optimization) . Given an objective function f : X → R and a measure function m : X → R m , where X is the space of all possible environments, the environment optimization problem searches for valid environments that maximize the objective function f while diversifying the measure function m .

4 Methods

We extend previous works [10, 51] to use CMA-MAE to search for a diverse collection of NCA generators with the objective and diversity measures computed from an agent-based simulator that runs domain-specific simulations in generated environments. Figure 2 provides an overview of our method. We start by sampling a batch of b parameter vectors θ from a multivariate Gaussian distribution, which form b NCA generators. Each NCA generator then generates one environment from a fixed initial environment, resulting in b environments. We then repair the environments using a MILP solver to enforce domain-specific constraints. After getting the repaired environments, we

Run agent-based simulator

Update optimization archive

Update result archive

Repair the environment with MILP

Form NCA generator and

generate the environment

Update

...

Sample

evaluate each of them by running an agent-based simulator for N e times, each with T timesteps, and compute the average objective and measures. We add the evaluated environments and their corresponding NCA generators to both the optimization archive and the result archive. Finally, we update the parameters of the multivariate Gaussian distribution (i.e., µ and Σ ) and sample a new batch of parameter vectors, starting a new iteration. We run CMA-MAE iteratively with batch size b , until the total number of evaluations reaches N eval .

NCA Generator. We define an NCA generator as a function g ( s ; θ , C ) : X → X , where X is the space of all possible environments, s ∈ X is a fixed initial environment, and C ∈ Z + is a fixed number of iterations for the NCA generator. g is parameterized by θ ∈ Θ . Since both s and C are fixed, each parameter vector θ ∈ Θ corresponds to an NCA generator, which corresponds to an environment x ∈ X . We discretize the measure space into M cells to create an archive and attempt to find the best NCA generator in each cell to optimize the QD-score of the archive.

Our NCA generator is a convolutional neural network (CNN) with the same architecture as the previous work [10] with 3 convolutional layers of kernel size 3 × 3 followed by ReLU or sigmoid activations. Figure 8 in Appendix C.1.1 shows the architecture. We represent the 2D tile-based environments as 3D tensors, with each tile as a one-hot encoded vector. Depending on the tile types, our NCA model has about 1,500 ∼ 3,000 parameters. The input and output of the generator are one-hot environments of the same size, facilitating iterative environment generation by feeding the generator's output back into it. By using a CNN, we can use the same NCA generator to generate environments of arbitrary sizes.

MILP Repair. The environments generated by the NCA generators might be invalid. Therefore, we follow previous works [49, 15, 51] to repair the invalid environments using a MILP solver. For each unrepaired environment x in = g ( s ; θ , C ) , we find x out ∈ X such that (1) the hamming distance between x in and x out is minimized, and (2) the domain-specific constraints are satisfied. We introduce the domain-specific constraints in detail in Section 5 and Appendix B.

Objective Functions. We have two objective functions f opt and f res for the optimization archive and the result archive, respectively. f res runs an agent-based simulator for N e times and returns the average throughput. f opt optimizes the weighted sum of f res and a similarity score ∆ between the unrepaired and repaired environments x in and x out . Specifically, f opt = f res + α · ∆ , where α is a hyperparameter that controls the weight between f res and ∆ . We incorporate the similarity score to f opt so that the NCA generators are more inclined to directly generate valid environments with desired regularized patterns, reducing reliance on the MILP solver for pattern generation. We show in Section 6 that incorporating the similarity score to f opt significantly improves the scalability of the environments generated by the NCA generators.

Let n be the number of tiles in the environment, the similarity score ∆ is computed as:

e i encodes if the tile type of tile i in x in and x out are the same. p i assigns a weight to each tile. As a result, the numerator computes the unnormalized similarity score. P is the theoretical upper bound of the unnormalized similarity score of all environments of a domain. Dividing by P normalizes the score to the range of 0 to 1. Intuitively, the similarity score quantifies the level of similarity between the unrepaired and repaired environments x in and x out , with each tile weighted according to p i .

Environment Entropy Measure. We want the NCA generators to generate environments with regularized patterns. To quantify these patterns, we introduce environment entropy as a diversity measure, inspired by previous work [30, 16]. We define a tile pattern as one possible arrangement of a 2 × 2 grid in the environment. Then we count the occurrence of all possible tile patterns in the given environment, forming a tile pattern distribution. The environment entropy is calculated as the entropy of this distribution. In essence, the environment entropy measures the degree of regularization of patterns within a given environment. A lower value of environment entropy indicates a higher degree of pattern regularization. We use environment entropy as a diversity measure to find NCA generators that can generate a broad spectrum of environments of varying patterns.

5 Domains

We perform experiments in three different domains: (1) a multi-agent warehouse domain [28, 51], (2) a multi-agent manufacturing domain, and (3) a single-agent maze domain [16, 5, 1].

Warehouse. The warehouse domain simulates automated warehousing scenarios, where robots transport goods between shelves in the middle and human-operated workstations on the left and right borders. We use two variants of the warehouse domain, namely warehouse (even) and warehouse (uneven) . In warehouse (even), all workstations are visited with equal frequency, while in warehouse (uneven), the workstations on the left border are visited 5 times more frequently than those on the right border. We constrain the warehouse environments such that all non-shelf tiles are connected and the number of shelves is N s to keep consistent storage capability. We use f opt and f res introduced in Section 4 as the objective functions, and environment entropy as one of the measures. Inspired by previous work [51], we use the number of connected shelf components as the other measure. Figure 1 and Figure 7a in Appendix B show example warehouse environments.

Manufacturing. The manufacturing domain simulates automated manufacturing scenarios, where robots go to different kinds of manufacturing workstations in a fixed order to manufacture products. We create this domain to show that our method can generate environments with more tile types for more complex robot tasks. We assume 3 kinds of workstations to which the robots must visit in a fixed order and stay for different time duration. We constrain the manufacturing environments such that all non-workstation tiles are connected and there is at least one workstation of each type. We use f opt and f res introduced in Section 4 as the objective functions, and the environment entropy and the number of workstations as the measures. The number of workstations can approximate the cost of the manufacturing environment. By diversifying the number of workstations, our method generates a collection of environments with different price points. Figure 4 and Figure 7b in Appendix B show example manufacturing environments.

Maze. We conduct additional experiments in the maze domain to show that our method can scale a single-agent reinforcement learning (RL) policy trained in small environments to larger environments with similar patterns. The maze domain does not have domain-specific constraints. We use the same objective and measure functions in the previous work [1]. Specifically, f opt and f res are both binary functions that return 1 if the environment is solvable and 0 otherwise. The measures are the number of walls and the average path length of the agent over N e simulations. Figure 5 and Figure 7c in Appendix B show example maze environments.

We include more detailed information on the domains in Appendix B.

×

×

6 Experimental Evaluation

Table 1 summarizes the experiment setup. We train the NCA generators with environments of size S and then evaluate them in sizes of both S and S eval . In addition, we set the number of NCA iterations C = 50 for environments of size S and C eval = 200 for those of size S eval for all domains. C eval is larger than C because the NCA generators need more iterations to converge to a stable state while generating environments of size S eval . For the warehouse and manufacturing domains, we use Rolling-Horizon Collision Resolution (RHCR) [28], a state-of-the-art centralized lifelong MAPF planner, in the simulation. We run each simulation for T = 1 , 000 timesteps during training and T eval = 5 , 000 timesteps during evaluation and stop early in case of congestion , which happens if more than half of the agents take wait actions at the same timestep. For the maze domain, we use a trained ACCEL agent [9] in the simulations. We include more details of the experiment setup, compute resources, and implementation in Appendix C.

6.1 Multi-Agent Domains

Baseline Environments. We consider two types of baseline environments for the multi-agent domains (warehouse and manufacturing), namely those designed by human and those optimized by DSAGE [51], the state-of-the-art environment optimization method. In DSAGE, we use f res as the objective function for the result archive and f opt with α = 5 for the surrogate archive. With size S eval , however, it is impractical to run DSAGE due to the computational limit. One single run of MILP repair, for example, could take 8 hours in an environment of size S eval . We use the human-designed warehouse environments from previous work [29, 28, 51]. The manufacturing domain is new, so we create human-designed environments for it.

NCA-generated Environments. We take the best NCA generator from the result archive to generate environments of size S and S eval . In the manufacturing domain, we additionally take the best NCA generator that generate an environment of size S with a similar number of workstations as the DSAGE-optimized environment. This is to ensure a fair comparison, as we do not constrain the number of workstations, which is correlated with throughput, in the manufacturing domain.

We show more details of the baselines and NCA-generated environments in Appendix D.

6.1.1 Environments of Size S

Columns 3 and 4 of Table 2 show the numerical results with N a = 200 agents in the warehouse and manufacturing domains with environment size S . All NCA-generated environments have significantly better throughput than the baseline environments. In addition, we observe that larger α values have no significant impact on the throughput with N a agents.

We also analyze the scalability of the environments by running simulations with varying numbers of agents and show the throughput in Figures 3a to 3c. In both the warehouse (even) and manufacturing domains, our NCA-generated environments have higher throughput than the baseline environments with an increasingly larger gap with more agents. In the warehouse (uneven) domain, however, the DSAGE-optimized environment displays slightly better scalability than those generated by NCA. This could be due to a need for more traversable tiles near the popular left-side workstations to alleviate congestion, a pattern creation task that poses a significant challenge for NCA generators

GLYPH<c=24,font=/AAAABC+DejaVuSans>GLYPH<c=19,font=/AAAABC+DejaVuSans>

GLYPH<c=21,font=/AAAABC+DejaVuSans>GLYPH<c=19,font=/AAAABC+DejaVuSans>GLYPH<c=19,font=/AAAABC+DejaVuSans>

GLYPH<c=22,font=/AAAABC+DejaVuSans>GLYPH<c=24,font=/AAAABC+DejaVuSans>GLYPH<c=19,font=/AAAABC+DejaVuSans>

1XPEHUGLYPH<c=3,font=/AAAABC+DejaVuSans>RIGLYPH<c=3,font=/AAAABC+DejaVuSans>$JHQWV

GLYPH<c=19,font=/AAAABC+DejaVuSans>

GLYPH<c=24,font=/AAAABC+DejaVuSans>

GLYPH<c=20,font=/AAAABC+DejaVuSans>GLYPH<c=19,font=/AAAABC+DejaVuSans>

7KURXJKSXW

(a)

S

: Warehouse (even)

GLYPH<c=24,font=/AAAABD+DejaVuSans>GLYPH<c=19,font=/AAAABD+DejaVuSans>

GLYPH<c=21,font=/AAAABD+DejaVuSans>GLYPH<c=19,font=/AAAABD+DejaVuSans>GLYPH<c=19,font=/AAAABD+DejaVuSans>

GLYPH<c=22,font=/AAAABD+DejaVuSans>GLYPH<c=24,font=/AAAABD+DejaVuSans>GLYPH<c=19,font=/AAAABD+DejaVuSans>

1XPEHUGLYPH<c=3,font=/AAAABD+DejaVuSans>RIGLYPH<c=3,font=/AAAABD+DejaVuSans>$JHQWV

GLYPH<c=19,font=/AAAABD+DejaVuSans>

GLYPH<c=24,font=/AAAABD+DejaVuSans>

GLYPH<c=20,font=/AAAABD+DejaVuSans>GLYPH<c=19,font=/AAAABD+DejaVuSans>

7KURXJKSXW

(b)

S

: Warehouse (uneven)

GLYPH<c=24,font=/AAAABE+DejaVuSans>GLYPH<c=19,font=/AAAABE+DejaVuSans>

GLYPH<c=21,font=/AAAABE+DejaVuSans>GLYPH<c=19,font=/AAAABE+DejaVuSans>GLYPH<c=19,font=/AAAABE+DejaVuSans>

GLYPH<c=22,font=/AAAABE+DejaVuSans>GLYPH<c=24,font=/AAAABE+DejaVuSans>GLYPH<c=19,font=/AAAABE+DejaVuSans>

1XPEHUGLYPH<c=3,font=/AAAABE+DejaVuSans>RIGLYPH<c=3,font=/AAAABE+DejaVuSans>$JHQWV

GLYPH<c=19,font=/AAAABE+DejaVuSans>

GLYPH<c=24,font=/AAAABE+DejaVuSans>

GLYPH<c=20,font=/AAAABE+DejaVuSans>GLYPH<c=19,font=/AAAABE+DejaVuSans>

7KURXJKSXW

(c)

S

: Manufacturing

GLYPH<c=20,font=/AAAABF+DejaVuSans>GLYPH<c=19,font=/AAAABF+DejaVuSans>GLYPH<c=19,font=/AAAABF+DejaVuSans>

GLYPH<c=28,font=/AAAABF+DejaVuSans>GLYPH<c=19,font=/AAAABF+DejaVuSans>GLYPH<c=19,font=/AAAABF+DejaVuSans>

GLYPH<c=20,font=/AAAABF+DejaVuSans>GLYPH<c=26,font=/AAAABF+DejaVuSans>GLYPH<c=19,font=/AAAABF+DejaVuSans>GLYPH<c=19,font=/AAAABF+DejaVuSans>

1XPEHUGLYPH<c=3,font=/AAAABF+DejaVuSans>RIGLYPH<c=3,font=/AAAABF+DejaVuSans>$JHQWV

GLYPH<c=19,font=/AAAABF+DejaVuSans>

GLYPH<c=26,font=/AAAABF+DejaVuSans>GLYPH<c=17,font=/AAAABF+DejaVuSans>GLYPH<c=24,font=/AAAABF+DejaVuSans>

GLYPH<c=20,font=/AAAABF+DejaVuSans>GLYPH<c=24,font=/AAAABF+DejaVuSans>

7KURXJKSXW

(d)

S

eval

: Warehouse (even)

GLYPH<c=20,font=/AAAABG+DejaVuSans>GLYPH<c=19,font=/AAAABG+DejaVuSans>GLYPH<c=19,font=/AAAABG+DejaVuSans>

GLYPH<c=25,font=/AAAABG+DejaVuSans>GLYPH<c=24,font=/AAAABG+DejaVuSans>GLYPH<c=19,font=/AAAABG+DejaVuSans>

GLYPH<c=20,font=/AAAABG+DejaVuSans>GLYPH<c=21,font=/AAAABG+DejaVuSans>GLYPH<c=19,font=/AAAABG+DejaVuSans>GLYPH<c=19,font=/AAAABG+DejaVuSans>

1XPEHUGLYPH<c=3,font=/AAAABG+DejaVuSans>RIGLYPH<c=3,font=/AAAABG+DejaVuSans>$JHQWV

GLYPH<c=19,font=/AAAABG+DejaVuSans>

GLYPH<c=26,font=/AAAABG+DejaVuSans>GLYPH<c=17,font=/AAAABG+DejaVuSans>GLYPH<c=24,font=/AAAABG+DejaVuSans>

GLYPH<c=20,font=/AAAABG+DejaVuSans>GLYPH<c=24,font=/AAAABG+DejaVuSans>

7KURXJKSXW

(e)

S

eval

: Warehouse (uneven)

GLYPH<c=20,font=/AAAABH+DejaVuSans>GLYPH<c=19,font=/AAAABH+DejaVuSans>GLYPH<c=19,font=/AAAABH+DejaVuSans>

GLYPH<c=20,font=/AAAABH+DejaVuSans>GLYPH<c=21,font=/AAAABH+DejaVuSans>GLYPH<c=21,font=/AAAABH+DejaVuSans>GLYPH<c=24,font=/AAAABH+DejaVuSans>

GLYPH<c=21,font=/AAAABH+DejaVuSans>GLYPH<c=22,font=/AAAABH+DejaVuSans>GLYPH<c=24,font=/AAAABH+DejaVuSans>GLYPH<c=19,font=/AAAABH+DejaVuSans>

1XPEHUGLYPH<c=3,font=/AAAABH+DejaVuSans>RIGLYPH<c=3,font=/AAAABH+DejaVuSans>$JHQWV

GLYPH<c=19,font=/AAAABH+DejaVuSans>

GLYPH<c=20,font=/AAAABH+DejaVuSans>GLYPH<c=21,font=/AAAABH+DejaVuSans>GLYPH<c=17,font=/AAAABH+DejaVuSans>GLYPH<c=24,font=/AAAABH+DejaVuSans>

GLYPH<c=21,font=/AAAABH+DejaVuSans>GLYPH<c=24,font=/AAAABH+DejaVuSans>

7KURXJKSXW

(f)

S

eval

: Manufacturing

compared to DSAGE's direct tile search. In addition, in the manufacturing domain, the optimal NCA generator (green line) creates an environment with top throughput for 200 agents, but it is slightly less scalable than the sub-optimal one used for DSAGE comparison (grey line). We conjecture this is due to the optimizer being caught in a local optimum region due to the complexity of the tasks. On the effect of α values, we observe that larger α values enhance the scalability of the generated

(a) 18 × 18.

(b) 66 × 66.

environments in the warehouse (even) domain, as shown in Figure 3a. We include further discussion of α values in Appendix E.1.

To further understand the gap of throughput among different environments of size S with N a agents, we present additional results on the number of finished tasks over time in Appendix E.2. All NCA-generated environments maintain a stable number of finished tasks throughout the simulation.

6.1.2 Environments of Size S eval

Columns 5 and 6 of Table 2 compare environments of size S eval with N a _ eval agents. In the warehouse (even) domain, all environments, except for the one generated by CMA-MAE + NCA ( α = 5 ), run into congestion. In the warehouse (uneven) domain, CMA-MAE + NCA ( α = 5 ) also achieves the best success rate, even though it does not have the highest throughput. This shows the benefit of using a larger α value with CMA-MAE in creating large and scalable environments.

To further understand the scalability, Figures 3d to 3f show the throughput of the environments with an increasing number of agents. We observe that all our NCA-generated environments are much more scalable than the human-designed ones. In particular, the environments generated by CMA-MAE + NCA ( α = 5 ) yield the best scalability in all domains. This is because a higher value of α makes CMA-MAE focus more on minimizing the disparity between the unrepaired and repaired environments. As a result, the optimized NCA generators are more capable of generating environments with desired regularized patterns directly. On the other hand, a smaller value of α makes the algorithm rely more on the MILP solver to generate patterns, introducing more randomness as the MILP solver arbitrarily selects an optimal solution with minimal hamming distance when multiple solutions exist. We show additional results of scaling the same NCA generators to more environment sizes in Appendix E.3. We also include a more competitive baseline by tiling environments of size S to create those of size S eval in Appendix E.4.

6.2 Scaling Single-Agent RL Policy

In the maze domain, we show that it is possible to scale a single-agent RL policy to larger environments with similar regularized patterns. We use the NCA generator trained from 18 × 18 maze environments to generate a small 18 × 18 and a large 66 × 66 maze environment, shown in Figure 5. Running the ACCEL [9] agent 100 times in these environments results in success rates of 90% and 93%, respectively. The high success rate comes from the fact that the similar local observation space of the RL policy enables the agent to make the correct local decision and eventually arrive at the goal.

For comparison, we generate 100 baseline 66 × 66 maze environments by restricting the wall counts to be the same and the path lengths to range between 80% to 120% of the one in Figure 5b, to maintain similar difficulty levels. Figure 15 in Appendix D.2 shows two example baseline environments. Testing the ACCEL agent in these environments yields an average success rate of 22% (standard error: 3%), significantly lower than that achieved in the NCA-generated environments. This demonstrates

the ability of our method to generate arbitrarily large environments with similar patterns in which the trained RL agent performs well.

6.3 NCA Generation Time

We show the total CPU runtime of generating and repairing the environments of sizes S and S eval in Table 3. In both sizes, larger similarity scores are correlated with shorter MILP runtime. The similarity scores of the manufacturing environments are significantly lower because we use a larger normalization factor P than that in the warehouse domains. Nevertheless, we note that even the longest combined runtime of the NCA generation and MILP repair ( 15 , 746 . 10 + 0 . 18 = 15 , 746 . 28 seconds for the manufacturing domain) on the environments of size S eval is significantly shorter than optimizing the environments of that size directly. This reduction in time is attributed to our method, which involves running the MILP repair N eval times with an environment size of S while training the NCA generators, and then using the MILP solver only once with size S eval while generating the environments. Our method stands in contrast to the previous direct environment optimization method [51] which requires running the MILP repair with the size of S eval for N eval times.

6.4 On the Benefit of QD Algorithms

QD algorithms can train a diverse collection of NCA generators. In Appendix E.5, we compare the QD-score and archive coverage of CMA-MAE + NCA and DSAGE. We also benchmark CMA-MAE against MAP-Elites [34, 47] to highlight the benefit of using CMA-MAE in training NCA generators. We observe that CMA-MAE + NCA has the best QD-score and archive coverage in all domains.

Furthermore, QD algorithms can train more scalable NCA generators than single-objective optimizers. In Appendix E.6 we compare CMA-MAE with a single-objective optimizer CMA-ES [20]. We observe that CMA-MAE is less prone to falling into local optima and thus better NCA generators.

7 Limitations and Future Work

We present a method to train a diverse collection of NCA generators that can generate environments of arbitrary sizes within a range defined by finite compute resources. In small environments of size S , our method generates environments that have competitive or better throughput compared to the state-of-the-art environment optimization method [51]. In large environments of size S eval , where existing optimization methods are inapplicable, our generated environments achieve significantly higher throughput and better scalability compared to human-designed environments.

Our work is limited in many ways, yielding many directions for future work. First, our method is only arbitrarily scalable under a range determined by finite compute resources. For example, with the compute resource in Appendices C.3 and C.4, the upper bound of the range is about S eval = 101 × 102 . Although this upper bound is significantly higher than previous environment optimization methods, future work can seek to further increase it. Second, we focus on generating 4-neighbor grid-based environments with an underlying undirected movement graph. Future work can consider generating similar environments with directed movement graphs or irregular-shaped environments. We additionally discuss social impacts of our work in Appendix F.

References

Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , pages 37762-37777, 2022.

Alan S. Brown. How Amazon robots navigate congestion. https://www.amazon. science/latest-news/how-amazon-robots-navigate-congestion , 2022. Accessed: 2023-05-09.

Bert Wang-Chak Chan. Lenia and expanded universe. In Proceedings of the Conference on Artificial Life (ALIFE) , pages 221-229, 2020.

Zhe Chen, Javier Alonso-Mora, Xiaoshan Bai, Daniel D Harabor, and Peter J Stuckey. Integrated task assignment and path planning for capacitated multi-agent pickup and delivery. IEEE Robotics and Automation Letters , 6(3):5816-5823, 2021.

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid , 2018.

Antonello Contini and Alessandro Farinelli. Coordination approaches for multi-item pickup and delivery in logistic scenarios. Robotics and Autonomous Systems , 146:103871, 2021.

Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature , 521(7553):503-507, 2015.

Mehul Damani, Zhiyao Luo, Emerson Wenzel, and Guillaume Sartoretti. PRIMAL 2 : Pathfinding via reinforcement and imitation multi-agent learning - lifelong. IEEE Robotics and Automation Letters , 6(2):2666-2673, 2021.

Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , pages 13049-13061, 2020.

Sam Earle, Justin Snider, Matthew C. Fontaine, Stefanos Nikolaidis, and Julian Togelius. Illuminating diverse neural cellular automata for level generation. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 68-76, 2022.

Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , pages 10040-10052, 2021.

Matthew C. Fontaine and Stefanos Nikolaidis. A quality diversity approach to automatically generating human-robot interaction scenarios in shared autonomy. In Proceedings of the Robotics: Science and Systems (RSS) , 2021.

Matthew Fontaine and Stefanos Nikolaidis. Covariance matrix adaptation map-annealing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 456-465, 2023.

Matthew C. Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 94-102, 2020.

Matthew C. Fontaine, Ya-Chuan Hsu, Yulun Zhang, Bryon Tjanaka, and Stefanos Nikolaidis. On the importance of environments in human-robot coordination. In Proceedings of the Robotics: Science and Systems (RSS) , 2021.

Matthew C. Fontaine, Ruilin Liu, Ahmed Khalifa, Jignesh Modi, Julian Togelius, Amy K. Hoover, and Stefanos Nikolaidis. Illuminating Mario scenes in the latent space of a generative adversarial network. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 5922-5930, 2021.

Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret. Data-efficient design exploration through surrogate-assisted illumination. Evolutionary computation , 26(3):381-410, 2018.

Gennaro Gala, Daniele Grattarola, and Erik Quaeghebeur. E(n)-equivariant graph neural cellular automata. ArXiv , abs/2301.10497, 2023.

Martin Gardner. Mathematical games - the fantastic combinations of john conway's new solitaire game 'life'. Scientific American , 223(4):120-123, 1970.

Nikolaus Hansen. The CMA evolution strategy: A tutorial. ArXiv , abs/1604.00772, 2016.

IBM Corp. IBM ILOG CPLEX Optimization Studio. https://www.ibm.com/ products/ilog-cplex-optimization-studio , 2022.

Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. In NeurIPS Workshop on Deep Reinforcement Learning , 2018.

Paul Kent and Juergen Branke. Bayesian quality diversity search with interactive illumination. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 10191026, 2023.

Ngai Meng Kou, Cheng Peng, Hang Ma, T. K. Satish Kumar, and Sven Koenig. Idle time optimization for target assignment and path finding in sortation centers. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 9925-9932, 2020.

Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation , 19(2):189-223, 2011.

Joel Lehman and Kenneth O. Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 211-218, 2011.

Jiaoyang Li, Kexuan Sun, Hang Ma, Ariel Felner, T. K. Satish Kumar, and Sven Koenig. Moving agents in formation in congested environments. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages 726-734, 2020.

Jiaoyang Li, Andrew Tinka, Scott Kiesel, Joseph W. Durham, T. K. Satish Kumar, and Sven Koenig. Lifelong multi-agent path finding in large-scale warehouses. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 11272-11281, 2021.

Minghua Liu, Hang Ma, Jiaoyang Li, and Sven Koenig. Task and path planning for multi-agent pickup and delivery. In Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) , pages 1152-1160, 2019.

Simon M. Lucas and Vanessa Volz. Tile pattern KL-divergence for analysing and evolving game levels. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 170-178, 2019.

Hang Ma, Jiaoyang Li, T. K. Satish Kumar, and Sven Koenig. Lifelong multi-agent path finding for online pickup and delivery tasks. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages 837-845, 2017.

Alexander Mordvintsev and Eyvind Niklasson. µ NCA: Texture generation with ultra-compact neural cellular automata. ArXiv , abs/2111.13545, 2021.

Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, and Michael Levin. Growing neural cellular automata. Distill , 2020.

Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. ArXiv , abs/1504.04909, 2015.

Van Nguyen, Philipp Obermeier, Tran Cao Son, Torsten Schaub, and William Yeoh. Generalized target assignment and path finding using answer set programming. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , pages 1216-1223, 2017.

Eyvind Niklasson, Alexander Mordvintsev, Ettore Randazzo, and Michael Levin. Selforganising textures. Distill , 2021.

Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In Proceedings of the International Conference on Machine Learning (ICML) , pages 17473-17498, 2022.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , pages 8024-8035, 2019.

Chris Reinke, Mayalen Etcheverry, and Pierre-Yves Oudeyer. Intrinsically motivated discovery of diverse patterns in self-organizing systems. In International Conference on Learning Representations (ICLR) , 2020.

Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence , 2:428-436, 2020.

Noor Shaker, Julian Togelius, and Mark J. Nelson. Procedural Content Generation in Games: A Textbook and an Overview of Current Research . Springer, 2016.

Roni Stern, Nathan R. Sturtevant, Ariel Felner, Sven Koenig, Hang Ma, Thayne T. Walker, Jiaoyang Li, Dor Atzmon, Liron Cohen, T. K. Satish Kumar, Roman Barták, and Eli Boyarski. Multi-agent pathfinding: Definitions, variants, and benchmarks. In Proceedings of the International Symposium on Combinatorial Search (SoCS) , pages 151-159, 2019.

Shyam Sudhakaran, Djordje Grbic, Siyan Li, Adam Katona, Elias Najarro, Claire Glanois, and Sebastian Risi. Growing 3D artefacts and functional machines with neural cellular automata. In Proceedings of the Conference on Artificial Life (ALIFE) , page 108, 2021.

Mattie Tesfaldet, Derek Nowrouzezahrai, and Chris Pal. Attention-based neural cellular automata. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , pages 8174-8186, 2022.

Bryon Tjanaka, Matthew C. Fontaine, David H. Lee, Yulun Zhang, Nivedit Reddy Balam, Nathaniel Dennler, Sujay S. Garlanka, Nikitas Dimitri Klapsis, and Stefanos Nikolaidis. pyribs: A bare-bones python library for quality diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 220-229, 2023.

Sumanth Varambally, Jiaoyang Li, and Sven Koenig. Which MAPF model works best for automated warehousing? In Proceedings of the Symposium on Combinatorial Search (SoCS) , pages 190-198, 2022.

Vassiiis Vassiliades and Jean-Baptiste Mouret. Discovering the elite hypervolume by leveraging interspecies correlation. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 149-156, 2018.

Ge Yu and Michael Wolf. Congestion prediction for large fleets of mobile robots. In Proceedings of the International Conference on Robotics and Automation (ICRA) , pages 7642-7649, 2023.

Hejia Zhang, Matthew C. Fontaine, Amy K. Hoover, Julian Togelius, Bistra Dilkina, and Stefanos Nikolaidis. Video game level repair via mixed integer linear programming. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE) , pages 151-158, 2020.

Yulun Zhang, Matthew C. Fontaine, Amy K. Hoover, and Stefanos Nikolaidis. Deep surrogate assisted map-elites for automated hearthstone deckbuilding. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages 158-167, 2022.

Yulun Zhang, Matthew C. Fontaine, Varun Bhatt, Stefanos Nikolaidis, and Jiaoyang Li. Multirobot coordination and layout design for automated warehousing. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , pages 5503-5511, 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors' goal is to optimize large floorplans for warehouses or manufacturing facilities, where robots need to drive during completion of tasks, while avoiding a congestion of the system.\nTheir innovation is to use the framework of neural cellular automata (NCA) to generate optimized patterns from an initial pattern. Optimized here means maximizing the number of robots in the system.\nThe training of the NCA's parameters is done using an unsupervised objective, the troughput of the system for the robot drivers, and simoultaneaously optimizing for the diversity of the solution using quality diversity algorithms.\nIt is shown that the NCA produce regular floor plans for envirnments larger than seen in their training and also perform well under the desired metrics, while being much more efficient to generate than what existing optimizers can achieve.\n\nSTRENGTHS:\n1. Interesting unsupervised objective for training environments of NCA with a combination of different optimization techniques.\n2. The authors found a way of optimizing patterns from small environments that seems to scale/generalize to larger environments of a similar structure.\n3. I find the evaluation strategy with the RL agent that scales with the NCA interesting and creative. I have not seen jointly scaling agents and environments before and would find this a generally interesting point to investigate independent of this work (although I might not be aware of existing literature on this topic).\n\nWEAKNESSES:\n1. An ablation or discussion is missing towards what can break the scalability and generalization ability to new settings: What happens for non-rectangular environments? What if the positioning of the stations is not as in the training set? More diverse test samples would strengthen the argument, that NCAs truly scale well. Also, at which size does the NCA start to fail?\n2. I think the setup explored here has big potential for introspecting the solutions. Can one derive the strategy the NCA learned from its produced examples and directly apply it? I.e. find the scalable simple rules that seem to underly the regular patterns generated?\n3. Since quality diversity is a big part of the main draft, it would be interesting to see several examples of generated environments in the main draft.\n4. To me, the design choices of experimenting with NCA and quality diversity objectives for warehouse plan optimization are not clearly justified (see question section).\n\nQUESTIONS:\n1. Why should a bigger warehouse plan be more difficult to optimize overall? I understand that the complexity of the search space grows, but why is simply tiling good solutions for small environments not mentioned as a (possibly) good baseline? After all, this is how the NCA environments seem to behave (but I might also be missing an obvious problem here).\n2. Isn’t a cool thing about your approach that it possibly also scales to arbitrarily shaped environments? I think it would be interesting to see what happens when the constraints of the warehouse are more intricate, e.g. integrating an old warehouse plan with a new one.\n3. How ‘broken’ are the NCA’s plans typically? Is there a lot of extra work that is done by the extra solver?\n4. Why is it necessary to use quality diversity as an algorithm for warehouse plan optimization? I can imagine how a diversity of environments is interesting in video games, but why should the diversity of the tiling types (which is the entropy measure here) be necessary? I would rather expect a low entropy to be useful, as repeated patterns in the warehouse probably make manufacturing the hall and plans cheaper and less complex.\n5. What was the process of manually designing the human-created solutions?\n6. Does congestion occur only during testing on the 5,000 time steps? If not, how do you handle it during training?\n7. Do you have an idea of how the environment size scale with respect to the maximum number of robots/tasks achievable without congestion? \n\nMinor:\n1. L.14 regularized -> regular (?) … if you truly mean regularized, please explain what is being regularized here.\n2. L.41 sortation -> sorting (?)\n3. L.49 it would be helpful to clarify that your objective is unsupervised and simply comes from the different losees you define.\n4. Overall, an (at least informal) definition of the throughput objective missing in the main text.\n5. L.168 what kind of object is $\\Theta$ and $\\mathbf{\\theta}$?\n6. I think in L.178 it should be “Due to the locality of the NCA operation…” rather than “By using a CNN …” Whatever mechanism you use to implement, the NCA will behave locally as it is its inherent property. The CNN is just an efficient way of implementing it in ML frameworks.\n\nLIMITATIONS:\nOverall, I think the ideas presented in the paper are interesting and a good work of engineering.\nHowever, to make the paper more relevant and useful to a broad audience, it is missing a more abstract perspective on the interesting techniques it proposes.\nFor this, it should be possible for other researchers to better understand the working principle so that they can estimate how well the techniques could work in different settings. I think this requires more introspection on the results than in the current draft.\nMoreover, to make the claim that the NCAs are ‘arbitrarily scalable’, as in the title, the evaluation is lacking different levels of floor plan scales, NCA time scales and an analysis of the regular patterns obtained from the NCA.\nFor the application to the warehouses, I do not find it very comprehensible that quality diversity plays a big role as emphasized in the paper.\nThese factors are what lead me to my score.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors train neural cellular automata with a quality diversity evolutionary algorithm to grow the 2D environment of a multi-agent automated warehouse, manufacturing domain, and single agent maze domain. They show that the approach is able to generate environments that can scale to different sizes, while keeping important regularities.\n\nSTRENGTHS:\n- NCAs can be trained to generate small environments and then scaled to larger environments without further training, thereby saving computational costs\n- Compared to previous methods, the Mixed Integer Linear Programming (MILP) has to only be run once to fix the larger environments\n- the combination of NCa and MILP is an interesting idea that could be extended to many other domains that NCAs already have been successfully applied to\n\nWEAKNESSES:\n- The approach is only shown in similar 2D domains\n- The paper could include more environment generation baselines, beyond DSAGE. For example, a comparison to Compositional Pattern Producing Networks (which could also be scaled to larger environments without further training) could elucidate the importance of growth over time\n- It would also be useful to include a baseline in which a larger map is created by just concatenating smaller environments and running MILP. Would those perform equally well?\n\nQUESTIONS:\n- \"we define a tile pattern as one possible arrangement of a 2- 2 grid in the environment.\" -> What about the distribution of global patterns? I assume they would not be captured this way?\n- One important question is, how easily could this approach be applied to other domains? Does it work particularly well for the domains in this paper because of their types of patterns? Which other domains could it be applied to in the future and what type of patterns would they need to benefit of, in order to allow the scaling to happen? What about environments that need more global pattern than local ones, which the NCA is particularly good at?\n- How important is the MILP in the process? At some point in the paper, it says that the process can take 8 hours, which suggests it can also become a bottleneck. It would be good to include the number of errors that MILP fixed (including the number of enironments that were initially not functional) in the result tables for all methods.\n\nLIMITATIONS:\nYes for the most part. It would be good to talk a bit more about the potential limitation of NCAs to capture more global patterns.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for generating diverse sets of environments and solving arbitrarily large environments. The main difference of this method is that the individual in QD algorithms serves as an environment generator rather than an environment, making it capable of handling very large-scale environment generation tasks.\n\nSTRENGTHS:\n1/ The proposed method can generate large environments and significantly improve the throughput of multi-robot system. \n\n2/ Using QD algorithms to solve this problems make sense, and the effectiveness are verified.\n\nWEAKNESSES:\n1/ The paper is somewhat difficult to read and lacks sufficient background information. To effectively convey the significance of the problem, the authors should provide a more detailed introduction to the research and add more references. This will help readers who are unfamiliar with the content to better understand the paper\n\n2/ The technical contribution of this work is somewhat limited. The proposed method appears to be an ad-hoc combination of existing techniques, and the authors should emphasize their unique technical contributions or demonstrate the method's importance in real-world applications.\n\n3/ To further strengthen the results, it would be beneficial to include more challenging types of environments and competitive baselines.\n\nQUESTIONS:\n1/ Why are there only two lines in Figure 3-f? \n\n2/ Could you show the diverse behaviors of the different generators? \n\n3/ The problem formulation, i.e., geneate a set of environment generators rather than environments, significantly increases the optimization overhead during the training process. How to balance the cimputation cost and the final performance? \n\n4/ How about using QD algorithms to directly generate a set of environments with different scales, and taking the scales as a part of the diversity measure?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors have used the Covariance Matrix Adaptation MAP-Annealing algorithm to generate warehouse environments that are efficient for robots to roam around and do their task of moving packages from one place to the other. The authors have presented the idea very nicely in step by step manner, making the concepts clear gradually. \n\nIt appears that the work is an improvement over previous works by the same authors. So this cannot be called a novel idea or work. Although, the authors have successfully applied the proposed algorithm to the problem at hand, i.e. organizing a warehouse for swift movement of robots.\n\nSTRENGTHS:\nThe results presented by the authors are comprehensive and promising. The appendices cover the areas that are left unhandled in the paper. The authors have provided the code to execute. Although, I could not run it due to the time required for it.\n\nWEAKNESSES:\nIncluding Human design as a metric of comparison is odd, because this is very subjective. An expert human being might be able to design an environment that is far better than a machine. Then there will be a question how have you sampled a human being, have you conducted a survey, or have designed a test that can measure the knowledge of a human being? The questions will keep popping up. I am not saying that you cannot compare with a human being, but the idea is when a scientific study is done including human beings, then the appropriate representatives are picked carefully, which is missing here.\n\nThe authors should have mentioned which hardware they used for executing the algorithm. They should have also mentioned how much time it took for them to execute all the scenarios. It appears that the simulation step would have taken quite a large amount of time, then have they considered any parallelism?\n\n\nFollowing are some mistakes \n\n\t- Figure 1\n\t\t○ shows an warehouse environment optimized directly with QD algorithms\n\t\t○ [correction]\n\t\t○  shows a warehouse environment optimized directly with QD algorithms\n\t- Line 45\n\t\t○ are well suited to arbitrary scaling as they incrementally contruct\n\t\t○ [correction]are well suited to arbitrary scaling as they incrementally construct\n\t- Line 56,57\n\t\t○ we only run the MILP solver once after generating the large environments.\n\t\t○ [correction]\n\t\t○ we only run the MILP solver once after generating large environments.\n\t- Line 80\n\t\t○ However, in the case of the environment optimization\n\t\t○ [correction]\n\t\t○ However, in the case of environment optimization\n\t- Line 177\n\t\t○ The generator’s input and output are indentical sized one-hot environments\n\t\t○ [correction]\n\t\t○ The generator’s input and output are identical sized one-hot environments\n\t- Line 223 and onwards\n\t\t○ Repeatedly the work \"Manufacture\" has been used, instead in most places \"Manufacturing\" should have been used\n\t- Line 319\n\t\t○ of arbitrarily sizes.\n\t\t○ [correction]\nof arbitrary sizes.\n\nQUESTIONS:\nWhat is a measure space that you have mentioned at different places starting from line 115, apparently, you have not defined it. Please provide a reference where this term is defined and an appropriate meaning relevant to your context.\n\nLIMITATIONS:\nYes, the authors have dedicated a complete section to this. The present work is only limited to the square-shaped cells each having 2 to 4 neighbors. They are aiming to find a strategy for organizing irregularly shaped objects, which would be quite challenging.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research paper addresses the problem of generating large environments to improve the throughput of multi-robot systems. While previous work has proposed Quality Diversity (QD) algorithms for optimizing small-scale environments in automated warehouses, these approaches fall short when replicating real-world warehouse sizes. The challenge arises from the exponential increase in the search space as the environment size grows. Additionally, the previous methods have only been tested with up to 350 robots in simulations, whereas practical warehouses could host thousands of robots.\n\nTo overcome these limitations, the authors propose a novel approach using Neural Cellular Automata (NCA) environment generators optimized through QD algorithms. Instead of directly optimizing environments, they train a collection of NCA generators with QD algorithms in small environments and then generate arbitrarily large environments from these generators at test time. The key advantage of using NCA generators is that they maintain consistent, regularized patterns regardless of the environment size, significantly enhancing the scalability of multi-robot systems.\n\nThe research is divided into three domains: multi-agent warehouses, multi-agent manufacturing scenarios, and single-agent maze environments. In the warehouse and manufacturing domains, the authors compare their NCA-generated environments with human-designed and state-of-the-art optimized environments, demonstrating higher throughput and better scalability in the NCA-generated environments. In the maze domain, they show that their method can scale a single-agent reinforcement learning (RL) policy to larger environments with similar patterns, outperforming baseline environments.\n\nThe paper's strengths lie in its originality, proposing a novel combination of NCA generators and QD algorithms to address the scalability challenge, its quality in presenting a systematic evaluation across multiple domains, clarity in the presentation of algorithms and results, and the significance of its potential impact on various multi-robot systems applications.\n\nSTRENGTHS:\nOriginality:\n\nThe paper introduces a novel approach to address the problem of generating large environments for multi-robot systems, which is distinct from prior work that focused on optimizing relatively small-scale environments.\nThe utilization of Neural Cellular Automata (NCA) environment generators, optimized through Quality Diversity (QD) algorithms, is a unique and creative combination of techniques, enabling the generation of large environments with regularized patterns.\nThe introduction of environment entropy as a diversity measure to quantify regularized patterns is an innovative concept, which contributes to understanding and evaluating the environments' structural characteristics.\nQuality:\n\nThe research paper thoroughly describes the methodology, algorithms, and experiments, providing comprehensive details for replicability and understanding.\nThe authors present a systematic evaluation across multiple domains, comparing their NCA-generated environments with human-designed and state-of-the-art optimized environments. The results are statistically analyzed and presented, demonstrating the quality of the proposed approach.\nThe utilization of CMA-MAE, a state-of-the-art optimization algorithm, adds rigor to the paper's technical foundation.\nClarity:\n\nThe paper is well-written and organized, making it easy for readers to follow the methodology and results.\nMathematical formulations and notations are clear and appropriately explained, enhancing the paper's accessibility to researchers in the field.\nThe use of illustrative figures and tables helps in visualizing the concepts and experimental outcomes.\nSignificance:\n\nThe paper addresses an important problem in the field of multi-robot systems, i.e., scalability and efficient generation of large environments.\nThe proposed NCA environment generators have the potential to impact various applications, such as automated warehouses and manufacturing systems, by providing optimized and scalable environments.\nThe demonstrated scalability of single-agent RL policies to larger environments with similar patterns indicates broader implications for other agent-based systems and RL tasks.\n\nWEAKNESSES:\nLack of Novelty in the Proposed Method: The paper claims to propose a new method for optimizing Neural Cellular Automata (NCA) environment generators using Quality Diversity (QD) algorithms. However, the paper does not adequately demonstrate the novelty of the proposed method compared to prior works that have used similar optimization techniques for environment generation. The lack of a comprehensive literature review and clear differentiation from previous approaches weakens the paper's contribution.\n\nInsufficient Evaluation of Scalability: While the paper demonstrates improved scalability of NCA-generated environments compared to human-designed environments, it lacks a thorough analysis of scalability as the environment size increases. The experiments focus on relatively small environments (e.g., 36x33) and then evaluate the same NCA generators on larger environments (e.g., 101x102). A more systematic study with varying environment sizes and intermediate steps would provide a clearer understanding of scalability.\n\nLimited Comparison with Baseline Models: The paper compares the NCA-generated environments with human-designed environments and a state-of-the-art optimization method, DSAGE, but does not include comparisons with other baseline models or state-of-the-art methods in related areas of research. Including more comprehensive comparisons would strengthen the paper's findings.\n\nLack of Real-world Implementation and Validation: The research focuses on simulated environments and agent-based simulations. While these are suitable for initial validation, the lack of real-world implementation and validation on physical multi-robot systems in actual automated warehouses or manufacturing scenarios reduces the practical significance of the proposed method.\n\nLack of Discussion on Generalizability: The paper focuses on three specific domains (multi-agent warehouse, multi-agent manufacturing, and single-agent maze), but it lacks a discussion of the generalizability of the proposed method to other domains or applications. Addressing the potential limitations and extensions to different scenarios would enhance the paper's broader impact.\n\nInsufficient Analysis of Hyperparameters: The paper briefly mentions the use of hyperparameters, such as ↵, but does not provide a detailed analysis of how these hyperparameters affect the performance of the proposed method. A thorough sensitivity analysis and tuning of hyperparameters would provide more insights into the model's behavior and stability.\n\nLack of Open-Source Implementation: While the paper describes the proposed method in detail, it does not provide an open-source implementation or a publicly available codebase. Providing access to the code would allow researchers to reproduce the experiments and build upon the proposed method.\n\nQUESTIONS:\nClarification on Novelty: Can the authors provide a more detailed explanation of the novel aspects of their proposed method compared to prior works in the field of environment generation and optimization? How does their approach differ from existing approaches in terms of algorithmic design and optimization techniques?\n\nScalability Analysis: Could the authors elaborate on the reasons behind the observed differences in scalability between the warehouse (even) and warehouse (uneven) domains? What factors contribute to the better scalability in one domain compared to the other? Is it possible to improve scalability in the domain with poorer performance?\n\nComparison with Baseline Models: While the authors have compared their NCA-generated environments with human-designed environments and DSAGE, could they provide additional comparisons with other baseline models or state-of-the-art methods in related research areas? This would help establish the broader significance of their proposed method.\n\nReal-World Applicability: How feasible is the implementation of the proposed method in actual automated warehouse or manufacturing environments? What are the potential challenges or limitations when applying the method to real-world scenarios? Are there any specific requirements or modifications needed for practical implementation?\n\nGeneralization to Other Domains: Could the authors discuss the potential applicability of their proposed method to domains beyond the ones tested in this paper? How well does the approach generalize to diverse multi-agent systems and different task settings? What are the potential obstacles or adaptations required for transferring the method to other scenarios?\n\nSensitivity to Hyperparameters: How sensitive is the proposed method to the choice of hyperparameters, such as ↵? Can the authors provide insights into the effects of different hyperparameter settings on the quality and scalability of the generated environments? Are there specific guidelines for tuning these hyperparameters?\n\nOpen-Source Code Availability: Is the authors' codebase publicly available for reproducibility and further research? If not, would the authors consider releasing the code to the research community? Open-sourcing the implementation would facilitate collaboration and promote further advancements in the field.\n\nRealistic Simulation Assumptions: How well do the agent-based simulators used in the experiments represent real-world multi-agent systems? Are there any limitations or simplifications in the simulators that might affect the validity of the results or the generalizability of the findings?\n\nImpact of Environment Size: In the experiments, the authors evaluate the NCA-generated environments on larger sizes (Seval) compared to the training environments (Strain). Could the authors discuss how increasing environment size affects the performance and efficiency of the NCA generators? Are there any specific challenges or benefits associated with generating larger environments?\n\nPractical Use Cases: Can the authors provide examples of potential real-world applications or use cases where the proposed method could be applied to improve multi-robot systems' performance and scalability? What are the envisioned practical benefits of adopting the proposed approach in such scenarios?\n\nLIMITATIONS:\nSimplified Simulations: The agent-based simulators used in the experiments might oversimplify real-world scenarios, leading to potential discrepancies between simulated and actual multi-robot system behaviors. The authors should discuss the implications of these simplifications and any potential deviations from real-world performance.\n\nLack of Real-World Validation: The proposed method's evaluation is primarily based on simulations, and there is no direct validation in real-world warehouse or manufacturing environments. It would be beneficial to include experiments in real-world scenarios to assess the transferability and effectiveness of the NCA-generated environments.\n\nGeneralization to Other Environments: While the paper shows promising results in the warehouse and manufacturing domains, it remains unclear how well the proposed method generalizes to more complex and diverse environments, such as outdoor spaces, multi-floor warehouses, or environments with dynamic obstacles.\n\nHyperparameter Sensitivity: The method's performance might be sensitive to the choice of hyperparameters, such as the weight ↵ used to balance similarity and objective functions. The authors should provide a more comprehensive analysis of the sensitivity of the method to different hyperparameter settings.\n\nScalability in All Domains: While the paper demonstrates superior scalability in some domains, the method's performance in other domains, such as the warehouse (uneven) scenario, does not show the same level of improvement over baseline environments. The authors should address the limitations that might hinder scalability in certain domains and propose potential solutions.\n\nEthical and Societal Considerations: Although the paper focuses on technical aspects, it is essential to discuss the broader societal impacts and potential ethical considerations of deploying large-scale multi-robot systems in various real-world applications. Addressing ethical implications and potential negative societal consequences can provide a more well-rounded perspective on the research.\n\nReproducibility: While the paper outlines the method's details, there is no mention of whether the authors' codebase is available for replication and further research. Providing open-source code or detailed instructions for replication would improve the study's transparency and reproducibility.\n\nComparison with More Baseline Models: The paper compares the NCA-generated environments with human-designed environments and DSAGE. Including additional comparisons with other state-of-the-art environment generation and optimization methods would strengthen the research's credibility and position it in the broader context of related work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}