Powderworld: A Platform for Understanding Generalization via Rich Task Distributions
Authors: Kevin Frans
Phillip Isola
Venue: ICLR 2023 notable top 25% (2023)
Keywords: reinforcement learning
environment
generalization
out-of-distribution
multi-task

Abstract: One of the grand challenges of reinforcement learning is the ability to generalize to new tasks. However, general agents require a set of rich, diverse tasks to train on. Designing a `foundation environment' for such tasks is tricky -- the ideal environment would support a range of emergent phenomena, an expressive task space, and fast runtime. To take a step towards addressing this research bottleneck, this work presents Powderworld, a lightweight yet expressive simulation environment running directly on the GPU. Within Powderworld, two motivating task distributions are presented, one for world-modelling and one for reinforcement learning. Each contains hand-designed test tasks to examine generalization. Experiments indicate that increasing the environment's complexity improves generalization for world models, yet causes reinforcement learning agents to struggle. Powderworld aims to support the study of generalization by providing a source of diverse tasks arising from the same core rules.

POWDERWORLD: A PLATFORM FOR UNDERSTANDING GENERALIZATION VIA RICH TASK DISTRIBUTIONS
Kevin Frans MIT CSAIL
Phillip Isola
MIT CSAIL phi 11lpppieee @mit.eeee.......e.m edu
eyfrans@ml. edu
ABSTRACT
One of the grand challenges of reinforcement learning is the ability to generalize new tasks. However, general agents require set of rich, diverse tasks train on. Designing a 'foundation environment' for such tasks is tricky - the ideal envi- ronment would support range of emergent phenomena, an expressive task space. and fast runtime. To take step towards addressing this research bottleneck, this work presents Powderworld, a lightweight yer expressive simulation environment running directly on the GPU. Within Powderworld, two motivating challenges distributions are presented, one for wordd-modelling and one for reinforcement learning. Each contains hand-designed test tasks to examine generalization. Ex- periments indicate that increasing the environment's complexity improves gen- eralization for world models and certain reinforcement learning agents, yet may inhibit learning in high-variance environments. Powderworld aims to support the study of generalization by providing source of diverse tasks arising from the same core rules. Try an interactable demo at Lvfrrans.com/static/oooo
INTRODUCTION
One of the grand challenges of reinforcement learning (RL), and of decision- makinn in general, is the ability to generalize to new tasks. RL agents have shown incredible performance on single task settings (Berner et al.) 2019 Lillicrap et al., [015)] Mnih et al. 2013), yet frequently stumble when presented with unseen challenges. Single-task RL agents are largely overfit on the tasks they are trained on (Kirk et al. 202T), limiting their practical use. In contrast, a general agent, which can robustly perform well wide range of novel tasks, can then be adapted to solve downstream tasks and unseen challenges.
General agents greatly depend on diverse set of tasks to train on. Recent progress in deep learning has shown that as the amount of data increases, so do generalization capabilities of trained models
(Brown al. 2020; Ramesh al.) 2021 Bommasani et al.) 2021: Radford et al. 2021). Agents trained on environments with domain randomization or procedural generation capabilities transfer better to unseen test tasks Cobbe et al. (2020); Tobin al. (2017); Risi & Togelius (2020); Khalifa et al. 22020). However, as creating training tasks is expensive and challenging, most standard en- vironments are inherently over-specific limited by their focus on single task type, c.g. robotic control or gridworld movement.
As the need study the relationships between training tasks and generalization increases, the RL community would benefit greatly from a 'foundation environment" supporting diverse tasks arising from the same core rules. The benefits of expansive task spaces have been showcased in Unsuper- vised Environment Design (Wang et al. [219 Dennis et al. 2020 Jiang et al. 2021 Parker-Holder ct al. 2022), but gridworld domains fail to display how such methods scale up. Previous works have proposed specialized task distributions for multi-task training (Samvelyan et al. 202 Suarez al.) 2019 Fan et al., 2022: Team et al.) 2021), each focusing on a specific decision-makinn problem. To further investigate generalization, is beneficial to have environment where many variations of training tasks can easily be compared.
As step toward lightweight yet expressive environments, this paper presents Powderworld, sim- ulation environment geared to support procedural data generation, agent learning, and multi-task generalization. Powderworld aims to efficiently provide environment dynamics by running directly

---

Build Arch
Transport Sand
Freeze Path
Gas
Figure 1: Examples of tasks created in the Powderworld engine. Powderworld provides a physics-inspired simulation over which many distributions of tasks can be defined. Pictured above are human-designed challenges where a player must construct unstable arches, transport sand through tunnel, freeze water create bridge, and draw a path with plants. Tasks in Powder- world creates challenges from set of core rules, allowing agents to learn generalizable knowledge. Try an interactive Powderworld simulation at Kwwrans.com/stattii com/static/powder
on the GPU. Elements (e.g. sand, water, fire) interact in modular manner within local neighbor- hoods, allowing for efficient runtime. The free-form nature Powderworld enables construction of tasks ranging from simple manipulation objectives to complex multi-step goals. Powderworld aims 1) be modular and supportive of emergent interactions, 2) allow for expressive design capability, and 3) support efficient runtime and representations.
Additionally presented are two motivating frameworks for defining world-modelling and reinforce- ment learning tasks within Powderworld. World models trained on increasingly complex environ- ments show superior transfer performance. In addition, models trained over more element types show stronger fine-tuming on novel rulesets, demonstrating that robust representation has been learned. the reinforcement learning case, increases task complexity benefit generalization up to task-specific inflection point, at which performance decreases. This point may mark when vari- ance the resulting reward signal becomes too high, inhibiting learning. These findings provide a starting point for future directions in studying generalization using Powderworld foundation.
2 RELATED WORK
Task Distributions for RL. Video games popular setting for studying multi-tate task RL, and en- vironments have been built off NetHack (Samvelyan et al., 2021 Küttler et al. 2020). Minecraft
(Fan al. 2022 Johnson al. 2016 Guss al. [209)). Doom (Kempka et al. 2016). and Atari
[Bellemare et al. 2013). Team et al. 2021) Yu et al. )0 Cobbe et al.) (2020) describe task
distributions focused on meta-learning, and Fan al. (2022); Suarez al. (2019); Hafner (2021); Perez-Liebana et al. (2016) detail more open-ended environments containing multiple task types. Most similar this work may be ProcGen (Cobbe et al. 2020), platform that supports infinite procedurally generated environments. However, while ProcGen games each have their own rulesets, Powderworld aims to share core rules across all tasks. Powderworld focuses specifically on run- time and expressivity, taking inspiration from online "powder games" where players build ranges of creations out of simple elements (bal) pow Bittker).
Generalization in RL. Multi-task reinforcement learning agents generally valued for their abil- ity to perform on unseen training tasks (Packer et al., 20188) Kirk et al., 2021). The sim2real prob- lem requires agents aim generalize to out-of-disttiibutio real world domains (Tobin et al. 2017 Sadeghi & Levine 2016). The platforms cited above also target generalization, often within the context of solving unseen levels within game. This work aims to study generalization within physics-inspires simulated setting, and creates outto---iitri challenges by sand-designing a set of unseen test tasks.
2

---

Simulation Speed vs. Batch Size
Sim. Time Components
Timesteps/Sec
Powderworld
10000
Powderworld (64x64)
35%
Gravity Fluid Flow Elements Velocity
Procgen
Minigrid
8%
23%
5000
Atari
34%
0
Minecraft
100
200
300
400
500
Batch Size
Figure 2: Powderworld runs on the GPU and can simulate many worlds parallel. GPU simu- lation provides a significant speedup and allows simulation time to scale with batch size. Simulation speed guaranteed to remain constant regardless of how many elements are present in the world.
POWDERWORLD ENVIRONMENT
The main contribution of this work is environment specifically for training generalizable agents over easily customizable distributions of tasks. Powderworld is designed to feature:
Modularity and support for emergent phenomena. The core of Powderworld is a set of fundamental rules defining how two neighboring elements interact. The consistent nature these rules is key to agent generalization; e.g. fire will always burn wood, and agents can learn these inherent properties of the environment. Furthermore, local interactions can build up to form emergent wider-scale phenomena, e.g. fire spreading throughout the world. This capacity for emergence enables tasks to be diverse yet share consistent properties. Thus, fundamental Powderworid priors exist that agents can take advantage of to generalize.
Expressive task design capability. major challenge in the study of RI. generaliza- tion is that tasks are often nonadjustable. Instead, an ideal environment should present an explorable space of tasks, capable of representing interesting challenges, goals, and con- straints. Tasks should be parametrized allows for automated design and interpretable control. Powderworld represents each task as 2D array of elements, enabling a variety of procedural generation methods. Many ways exist to test specific agent capability, e.g. "burn plants to create a gap", increasing the chance that agents encounter these challenges.
Fast runtime and representation. As multi-task learning can be computationally ex- pensive, it is important that the underlying environment runs efficiently. Powerworld is designed run on the GPU, enabling large batches of simulation to be run in parallel. Ad- ditionally, Powderworld employs neurallnettwor--friendd matrix representation for both task design and agent observations. To simplify the training of decision-making agents, the Powderworld representation is fully-observable and runs discrete timescale (but partial-observabiliyy is an easy modification if desired).
3.1 ENGINE
Described below is an overview of the engine used for the Powderworld simulator. Additional technical details can be founded in the Appendix.
World matrix. The core structure of Powderworld world. Each location W., z.y holds a vector of information representing that location in the world. Namely, each vector contains one-hot encoding of the occupying element, plus additional values indicating gravity, density, and velocity. The W matrix is Markovian state of the world, and thus past W matrices are unnecessary for state transitions. Every timestep, a new W matrix is generated via a stochastic update function, as described below.
matrix of elements W representing the
Gravity. Certain elements are affected by gravity, as noted by the IsGravity flag in Figure 3 Each affvity-affected element also holds density value, which determines the element's priority dur- ing the gravity calculation. Every timestep, each element checks with its neighbor below. both elements are affeete,, and the neighbor below has lower density, then the two elements swap positions. This interaction functions as a core rule in the Powderworld simulation and allows elements to stack, displace, and block each other.
3

---

Elemsnt, loGravity, Demsity
Reactants, Effect
Empty
Fire
Sand
bottom-side empty, fall.
Fire
Gas
Burn quickly.
Wall
Plant
Water
Fluid: If space, move to side.
Fire
Plant
Burn quickly.
Sand
3
Stone
Ice
2% chance: melt to water.
Fire
Wood
Burn slowly.
Water
2
Lava
Stone
a'yy side empty, fall,,
Fire
Dust
Burn quickly + explosion.
Gats
Acid
Water Ice
5% chance; freeze to ice.
Acid
Dissolve acid neighbor.
Wood
4
Dust
Plant
Water
If 3 plant neighbor, grow plant.
Cloner
Clone first element touched.
Ice
Cloner
Fire If not burning. 30% fade away.
Lava
If neighhor empty, create fire.
Figure 3: A list of elements and reactions in the Powderworld simulation. Elements each contain gravity and density information. A set of element-specific reactions dictates how each element behaves and reacts to neighbors. Certain reactions manipulate the world's ss velocity field, which can push further elements away. Together, the gravity, velocity, and reaction systems create core set of rules by which interesting simulations arise.
Element-specific reactions. The behavior of Powderworld arises from set of modular, local ele- ment reactions. Element reactions can occur either within a single element, or as a reaction when two elements are neighbors each other. These reactions are designed to facilitate larger-scale behaviors; e.g. the sand element falls to neighboring locations, thus areas of sand form pyramid- like structures. Elements such as water, gas, and lava are fluids, and move horizontally to occupy available space. Finally, pairwise reactions provide interactions between specific elements, e.g. fire spreads to flammable elements, and plants grow when water nearby. See Figure a description of the Powderworld reactions, and full documentation is given in the appendix and code.
Velocity system. Another interaction method is applying movement through the velocity system. Certain reactions, such as fire burning or dust exploding, add to the velocity field. Velocity is rep- resented via an two-component Vz.y vector at each world location. If the magnitude of the velocity field at a location is greater than a threshold, elements are moved in one of eight cardinal directions, depending on the velocity angle. Velocity naturally diffuses and spreads in its own direction, thus a velocity difference will spread outwards before fading away. Walls are immune to velocity affects. Additionally, the velocity field can be directly manipulated by an interacting agent.
All operators are local and translation equivariant, yielding simple implementation in terms of (nonlinear) convolutional kernels. To exploit GPU-optimized operators, Powderworld imple- mented in Pytorch (Paszke et al. 2019), and performance scales with GPU capacity (Figure()).
EXPERIMENTS
The following section presents series of motivating experiments showcasing task distributions within Powderworld. These tasks intend to provide two frameworks for accessing the richness of the Powderworld simulation, one through supervised learning and one through reinforcement learning. While these tasks aim to specifically highlight how Powderworld can be used to generate diverse task distributions, the presented tasks are by means exhaustive, and future work may easily define modifications or additional task objectives as needed.
In all tasks, the model is provided the W E Hx x20 vian state containing element, gravity, density, and velocity information. All task distributions also include a procedural generation algorithm for generating training tasks, well as tests used to measure transfer learning.
x20 matrix an observation, which is Marko-
In all experiments below, evaluation on out-of-disrribution tests which are unseen during training.
4.1 WORLD MODELLING TASK
This section examines a world-modelling objective in which neural network is given observation of the world, and must then predict a future observation. World models can be seen as learning how to encode an environment's 's dynamics, and have proven to hold great practical value in downstream decision making (Ha & Schmidhuber] 20188 Hafner et al. 2019b]a). a). A model which can correctly predict the future of any observation can be seen as thoroughly understanding the core rules of
4

---

Sand Fa
Stone Bridge
Ice and Plant
Burning Vine
Acid on Wood
Water Flow
Dust Leva
Figure 4: World modelling test states are designed to showcase specifie element interactions. Test states are out-of-distribution and unseen during training. Model generalization capability is measured by how accurate its future predictions on all eight tests.
PCG Params
Start States
Future States
# Elems Squares A Start States
PCG Algorithm
Powderworld Simulator
World Model
Evaluate on Test Tasks
Figure 5: Training states are generated via procedural content generation (PCG) algorithm followed by Powderworld simulation. Experiments examine the affect increasing complexity in PCG parameters.
the environment. The world-modelling task does not require reinforcement learning, and is instead solved via a supervised objective with the future state as the target.
Specifically, given observation W IR Hxx of the world, the model is tasked with generat- ing a W' E R Wxx. matrix of the world 8 timesteps in the future. W values corresponds to logit probabilities of the different elements, and loss is computed via cross-entropy between the true and predicted world. Tasks are represented by tuple of starting and ending observations.
Training examples for the world-modelling task are created via an parametrized procedural con- tent generation (PCG) algorithm. The algorithm synthesizes starting states by randomly selecting elements and drawing a series of lines, circles, and squares. Thus, the training distribution can be modified by specifying how many of each shape to draw, out of which elements, and how many total starting states should be generated. A set of hand-designed tests are provided as shown Figure I which each measures distinct property of Powderworld, c.g. simulate sand falling through water, fire burning a vine, or gas flowing upwards. To generate the targets, each starting state is simulated forwards for 8 timesteps, shown in Figure
The model is a convolutional U-net network (Ronneberger et al. 2015). operating over a world size of 64x64 and 14 distinct elements. The agent network consists of three U-net blocks with 32, 64, and 128 features respectively. Each U-net block contains two convolutional kemels with kernel size of three and ReLU activation, along with MaxPool layer in the encoder blocks. The model is trained with Adam for 5000 iterations with batch size of 256 and learning rate of 0.005. During training. replay buffer 1024#256 data points randomly sampled to form the training batch, and the oldest data points are rotated out for fresh examples generated via the Powderworld simulator.
4.1.1 CAN WORLD MODELS GENERALIZE TO UNSEEN TEST STATES?
starting experiment examines whether world models trained purely on simulated data can correctly generalize on
test states. The set of tests, shown in Figure 4 are out-of-distribution hand-designed worlds that do not appear in the training set. A world model must discover the core ruleset of environmental dynamics in order to successfully generalize.
Scaling laws for training large neural networks have shown that more data consistently improves performance (Kaplan al., 2020 Zhai al.) 2022). Figure 6 shows this observation to be true Powderworld as well; world models trained on increasing amounts of start states display higher performance test states. Each world model trained on the same number of training examples
5

---

Zero-Shot Generalization Test States
Adaptation to New Dynamics
0.50
850
Lines
0.15
Cirless
0.85
0.20
0.90
Blemets
0.20
1K States
0.20
Squanes
Bements
Test Laxs
10t Statess
Lines
Cir + Lin
Bements
0.35
100k Statis
16 Lines
0.15
Cir + Str
Blements
Lines
Elements
Cir, Lin. Say
0.4
Elements
0.05
0.05
0.05
0.2
0.00
0.00
0.0
# of Start States
# Lines
vs. Type of Shupes
of Elements
Only Circles
Only Squares
Circles+ Squares
Cir+ Sor Line
Lines
4 Lines
Lines
321 Lines
Start State State
Feture State
www.
Figure 6: World model generalization improves training distribution complexity is in- creased. Shown are the test performances of world models trained with data from varying numbers of start states, number of lines, and types of shapes. By learning from diverse data, world models can better generalize to unseen test states. Top-Right: World models trained on more elements can better fine-tune to novel elements. These results show that Powderworld provides rich enough simulation that world models learn robust representations capable of adaptation to new dynamics. Bottom: Examples of states generated with various PCG parameters.
and timesteps, the only difference is how this data is generated. The average test loss over three training runs are displayed.
Results show that the 10-state world model overfits and does not generalize to the test states. In contrast, the 100-state model achieves much higher test accuracy, and the trend continues as the number of training tasks improves. These results show that the Powderworld world-modelling task demonstrates similar scaling laws as real-world data.
4.1.2 How DO INCREASINGLY COMPLEX TRAINING TASKS AFFECT GENERALIZATION?
As training data expands include more varieties of starting states, does world model performance over set of test states improve? More complex training data may allow world models to learn more robust representations, but may also introduce variance which harms learning create degenerate training examples when many elements overlap.
Figure 6 displays how additional shapes are included within the training distribution, zero-shot test performance successfully increases. World models are trained on distributions of training states characterized by which shapes are present between lines, circles, and square. Lines are assigned random (X1.y1,, (x2,y and thickness. Circles and Squares are assigned random (X+_y1) along with a radius. Each shape filled in with randomly selected element. Between 0 and of each shape drawn. Interestingly, training tasks with less shape variation also display higher instability, as shown in the test loss spikes for Line-only, Cirele-only, and Square-only runs. Addi- tionally, world models operating test performance. This behavior may indicate that models trained over more diverse training data leam representations which more resistant perturbations.
training states with greater number of lines display higher
Results showcase how in Powderworld, as more diverse data is created from the same set core rules, world models increase in generalization capability.

---

Generic Task Framework.
Sand-Pushing
Destroying
Path-Building
Start
Actions
End
Goal
place elem. XY
tA (Place Wind Only)
Start
Start
#Ac-
Goal
Goal
Goal
Reward
Agent
Error beiween (End, Goal)
Figure 7: In Powderworld RL tasks, agents must iteratively place elements (including direc- tional wind) transform starting state into goal state. Within this framework, we present three RI tasks as shown above. Each task contains many challenges, as starting states are randomly generated for each episode. Agents are evaluated on test states that are unseen during training.
4.1.3 DOES ENVIRONMENT RICHNESS INFLUENCE TRANSFER TO NOVEL INTERACTIONS?
While perfect world model will always make correct predictions, there are guarantees such models can learn new dynamics. This experiment tests the adaptability of world models, by exam- ining if they can quickly fine-tune
elemental reactions.
Powderworld's s ruleset is also of importance, as models will only transfer to new elements if all ele- ments share fundamental similarities. Powderworld elements naturally share set of behaviors, e.g. gravity, reactions-on-comtal and velocity. Thus, this experiment measures whether Powderworld presents rich enough simulation that models generalize
rules within the environment.
To run the experiment, distinct world models are trained on distributions containing a limited set of elements. The I-element model sees only sand, the 2-element sees only sand and water, the 3-element sees sand, water, and wall, and so on. Worlds are generated via the same procedural generation algorithm, specifically up 5 lines are drawn. After training for the standard 5000 iterations, each world model is then fine-tuned for 100 iterations on training distribution containing three held-out elements: gas, stone, and acid. The world model loss is then measured on new environment containing only these three elements.
Figure (top-right) highlights how world models trained on increasing numbers of elements show greater performance when fine-tuned on set of unseen elements. These results indicate that world models trained on richer simulations also develop more robust representations, as these representa- tions can more easily be trained on additional information. Powderworld world models learn not only the core rules of the world, but also general features describing those rules, that can then be used to learn new rules.
5 REINFORCEMENT LEARNING TASKS
Reinforcement learning tasks can defined within Powderworld via a simple framework, as shown in Figure Agents allowed to iteratively place elements, and must transform a starting state into a goal state. The observation space contains the Powderworld world state W E R61x61x20, and the action space is multi-discrete combination of X,Y, Element, Va, Vy- Va and V,, only utilized if the agent is placing wind.
Tasks are defined by a function that generates a starting state, goal state, and any restrictions on element placement. Note that Powderworld tasks are specifically designed to be stochastically diverse and contain randomly generated starting states. Within this framework, many task varieties can be defined. This work considers:
Sand-Pushing, The Sand-Pushing environment is an RL environment where agent must move sand particles into goal slot. The agent is restricted to only placing wind, at a controllable velocity and position. By producing wind, agents interact with the velocity field, allowing them to push and move elements around. Wind affects the velocity field in 10x area around the specified position. Reward equals the number of sand elements

---

Sand-Pushing
Destroying
Path-Buile Bath-Building
2.9
3.75
15
Test
10
25.0
Test
Train Reward
3.50
Test Reward
3.25
2.7
16
20.0
3.00
Train
2.6
17.5
15
2.75
Test
1
1
4
8
Train Task Complexity
Train Task Complexity
Train Task Complexity
Figure 8: Increasing the complexity of RL training tasks helps generalization, up task- specific inflection point. Shown are the test rewards of RI. agents trained on tasks with increasing numbers of shapes (shown in log-scale). performance, as agents become unable to extract a sufficient reward signal. In Destroying, com- plexity consistently increases test performance. While increased complexity generally increases the difficulty of training tasks and reduces reward, in Path- aath-Building certain obstacles can be used to complete the goal, improving training reward.
Sand-Pushing, too much complexity will decrease test
within the goal slot, and episodes are run for 64 timesteps. The Sand-Pushing task presents sparse-reward sequential decision-making problem.
Destroying. In the Destroying task, agents are tasked with placing a limited number of elements to efficiently destroy the starting state. Agents are allowed place elements for five timesteps, after which the world is simulated forwards another 64 timesteps, and reward is calculated as the number of empty elements. A general strategy is to place fire flammable structures, and place acid on other elements to dissolve them away. The Destroying task presents a task where correctly parsing the given observation is crucial.
Path-Building The Path-Buildin task presents construction challenge in which agents must place or remove wall elements to route water into a goal container. An episode lasts 64 timesteps, and reward is calculated the number of water elements in the goal. Water is continuously produced from a source formation of Cloner+Wate elements. In the Path- Building challenge, agents must correctly place blocks such that water flows efficiently in the correct direction. Additionally, any obstacles present must be cleared or built around.
To learn to control this environment, a Stable Baselines 3 PPO agent (Raffin et al. 2021 Schulman al. 2017) is trained over 1,000,000 environment interactions. The agent model is comprised of two convolutional layers with feature size 32 and 64 and kernel size of three, followed by two fully- connected layers. A learning rate 0.0003 is used, along with a batchsize of 256. An off- off-the-shelf RL algorithm is intentionally chosen, so experiments can focus on the impact of training tasks.
Figure g highlights agents solving the various RL tasks. Training tasks generated using the same procedural generation algorithm as the world-modellnn modelling experiments. Task-specific structures are also placed, such the goal slots in Sand-Pushing and Path-Building, and initial sand/water elements.
To test generalization, agents are evaluated on test tasks that are out of distribution from train- ing. Specifically, test tasks are generated using procedural generation algorithm that only places squares (5 for Destroying and Sand-Pushing, 10 for Path-Building). contrast, the training tasks are generated using only lines and circles.
Figure g showcases how training task complexity affects generalization to test tasks. Displayed rewards are averaged from five independent training runs each. Agents trained tasks generated with increasing numbers of lines and circles (0, 1, 4 32, 64). These structures serve as obstacles, and training reward generally decreases complexity increases. One exception is in Path-Building, as certain element structures can be useful in routing water to the goal.
Different RL tasks display different response to training task complexity. In Sand-Pushing, it is helpful to increase complexity up to 8 shapes, but further complexity harms performance. This inflection point may correspond to the point where learning signal becomes too high-variance. RI.
8

---

Sand-Pushing. Create wind to move sand into goal.
Destroying. Place any element clear the world.
Path- Path-Building. Place clear walls move goal.
Figure 9: Agents solving the Sand-Pushing, Destroying, and Path-Building tasks. In the Sand- Pushing task, wind used push a block of sand elements between obstacles to reach the goal slot on the right. In Destroying, agents must place a limited number of elements to efficiently destroy the world. In Path-Building, agents must construct path for water flow from source goal container. Tasks are randomly generated via procedural algorithm.
highly dependent on early reward signal to explore and continue to improve, and training tasks that are too complex can cause agent performance to suffer.
In contrast, agents on the Destroying and Path-Building task reliably gain a benefit from increased training task complexity. On the Destroying task, increased diversity during training may help agents recognize where to place fire/acid in test states, For Path-Building, training tasks with more shapes may present more possible strategies for reaching the goal.
The difference in how complexity affects training in Powderworld world-modelling and reinforce- ment learning tasks highlights motivating platform for further investigation. While baseline RL methods may fail to scale with additional complexity and instead suffer due to variance, alternative learning techniques may better handle the learning problem and show higher generalization.
CONCLUSION
Generalizing to novel unseen tasks is one of the grand challenges of reinforcement learning. Con- sistent lessons in deep learning show that training data is of crucial importance, which in the case of RL is training tasks. To study how and when agents generalize, the research community will benefit from more expressive foundation environments supporting many tasks arising from the same core rules.
This work introduced Powderworld, expressive simulation environment that can generate both supervised and reinforcement learning task distributions. Powderworld's ruleset encourages mod- ular interactions and emergent phenomena, resulting world models which can accurately predict unseen states and even adapt to novel elemental behaviors. Experimental results show that increased task complexity helps the supervised world-modelling setting and in certain RI. scenarios. At times, complexity hampers the performance of a standard RL agent.
Powderworld built to encourage future research endeavors, providing a rich yet computationally efficient backbone for defining tasks and challenges. The provided experiments hope to showcase how Powderworld can be used as a platform for examining task complexity and agent generaliza- environment for studying open-ended agent learning,
tion. Future work may Powderworld unsupervised environment design techniques, or other directions. As such, all code for Powderworld released online in support of extensions.

---

ACKNOWLEDGMENTS
This work was supported by Packard Fellowship to P.I. Thanks Akarsh Kumar for assistance during discussions, paper feedback, and implementation on the RL tasks.
REFERENCES
Physics simulation game: Powder game. dust/
URL https://dan-ban /ptpa///.pp////p/////////e/// javagaanjaaaaee/////aa
The powder toy. URL/tttps://ooweettyy.co
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06689, 2019.
Max Bittker. Making sandspiel. URL https://maxbittkerr wwt.:/ccc//maaania sandspinddddle
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprini arXiv: 1rXiv:2108.0025:: 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 33:1877-1901, 2020.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048- 2056. PMLR, 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances neural information processing systems, 13:13099-1061 2020.
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.0885. 2022.
William H Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noburu Kuno, Stephanie Milani, Sharada Prasanna Mobanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. The minerl competition on sample efficient reinforcement learning using human priors. 2019.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018
Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint 17X00.2109.067880 2021.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1912.01633 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555-2565. PMLR, 2019b.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference Machine Learning, pp. 4940 4950. PMLR, 2021.
10

---

Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artifi- cial intelligence experimentation. In Ijeai, pp. 4246 4247. Citeseer, 2016.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Michat Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaškowski. Viz- doom: A doom-basee research platform for visual reinforcement learning. In 2016 IEEE conference on computational intelligence and games (CIG), pp. 1-8. IEEE, 2016.
Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. Pegrl: Procedural content gener- ation via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pp. 95-101, 2020.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:2111.09795 2021.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.000 02..0999.99,,,.. 2015.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl. Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint
2018.
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint arXiv:2203.01300 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer. d'Alché- Bucc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp- Curran Associates, Inc., 2019. URL wtw.hhttp:// http://papprs... neur wwt.eurii...cpppppp.....p.. atteorrchaa--mmeratiii imperatiie style weg.a-perfrrmacceeeeee--eeeeeeee dewpeeeeeddee---eeeeeeee
9015 wwwpppyttooaaaaaa an-orchh-n-imppeee
Library.
pdf
Diego Perez-Liehana, Spyridon Samothrakis, Julian Togelius, Tom Schaul, and Simon M Lucas. General video game ai: Competition, challenges and opportunities. Thirtieth AAAI conference artificial intelligence, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8831- PMLR, 2021.
11

---

Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, 2(8):428 436,, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. arXiv preprint 1rXiv: 2016.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holdes Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. arXiv preprint 10ii:2209.13002 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively mul- tiagent game environment for training and evaluating intelligent agents. arXiv preprint 19X1001903.00784. 2019.
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2 / /2808, 2021.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEFE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 1901.01733. 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning. pp. 1094-1100. PMLR, 2020.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
12

---

Empty
Wall a
Water
Wood
Fire
Plant
Lava
Acid
Dust soee
Clomer
Wind
Figure 10: The Powderworld Web GUI allows users to directly edit the state of a world by drawing on various elements. World states can generated at custom resolutions, and the simula- tion runs in real-time. Powderworld runs on a GPU server, with the web app acting only as display and controller.
A
APPENDIX
A.1 POWDERWORLD ENGINE
Described in this section is overview of the technical details of the Powderworld Engine. We provide these details 1) as a reference for future work built on top of Powderworld, and 2) as framework for building simulations and environments that run the GPU. Powderworld is designed to run as fast as possible on standard GPU environment, as many deep learning setups already support GPU access. Thus, Powderworld interacts with the GPU through Pytorch functions.
Data Representation. The state in Powderworld is represented as XxHxWxN size tensor. One benefit of rumning on the GPU that multiple simulations can be run parallel, therefore all func- tions in Powderworld are written to support *batches* of data. The other dimensions refer to height and width, along with an N-size vector representing the data stored in each location. This data is specifically a one-hot vector of the current element at the location, along with indices dictating gravity, density, flow state, and velocity.
Matrix Operations. To remain efficient and parallalizable, operations in Powderworld are not implemented as loops over XY space, but rather as series of matrix operations. For example, to simulate the world falling down one block, one can call the following function:
world[:] torch. roll (world, shifts=1, dims-2)
In the codebase, there are helper functions for shifting the world each of four directions: getBe- low, getAbove, getLeft, getRight. These functions are called frequently and used to build up more complex operations.
Additionally, since operations need occur over entire world matrix, to modify specific portions of the matrix we must construct a mask. We can do this by doing any kind of comparison over values in the world matrix, then setting the world matrix to a weighted sum of world (world"(I-mask) new newWorld # For example, to change all fire elements into water elements, the following pseudocode works:
// Transform fire into water. mask (world[:, dire_indext world[:] = world* (1-mask) water_vece (mask)
1)
Gravity. In Powderworld, each element has a Density and an IsGravity flag. These values are stored in the 1st and 2nd indicies of the world array. Gravity operates series of switches: if an element is above another element and the top element has greater density, and both elements are gravity-enabled, then the two elements swap. As baseline, empty space (Air) has density of
13

---

thus elements like Sand (density=2) will fall, and elements such Gas (density=O) will rise. The IsGravity flag is necessary to prevent elements from falling through stationary elements such as wall wood, which should remain static. Gravity handles only vertical swaps, and in combination with other behaviors creates the piling and flowing mechanics seen in sand and water.
One crucial component of the gravity procedure is that due to the nature of switching, two elements cannot attempt to switch into the same position. Swaps are performed simultaneously, and a swap involves setting the upper position to the lower element, and vice versa. If two elements swap into the same position, then that position will be written to twice, and the original element will duplicate itself above and below. Therefore, all swap operations must be sure never swap two elements into the same position. To solve this in the gravity case, we iterate gravity as a loop over the possible densities. If all densities were computed together, a vertical stack of densities [0, 1,2] would result in the elements with densities of 0 and 2 both attempting to move into the center position. By processing downward swaps for each density iteratively, an order established and the conflict does not occur (in the example provided, 0 would first swap with 1, and then O would swap with 2 in the following iteration.
// Run gravity. for currDensity in [0,1,2,3]:
density worldl;, density [indety_indx]] Delta between ABOVE and current density_delta get_above (density) density is If BELON has density_above_greatee
= (density_delta 0)
then density_below_les
is_densityy_below_less = htt.///owoos..eeeessstreereerree
is "s_dennity_ccuzrent = (density = curzDensity)
is de.density_aboee_crree
get get__abve (is dir.bboreriisddssttttrrrrre
is_gravity = (world gravity_index ==
is ss__ennte__emm_ballooorrrrr [s__cente__and_aaov__saat
get_belowe (te__eelowts__aavittt) & is_gravity
get_above (is (is_gravity) & is_gravity
world_above get get_above (world)
below below get www.beeewwwow.. (world)
world[:] = world =1 beecomoemeeeeeemeeeeeeeee below-does beelow-doe_bbecom__deee
+ world bel www.rddeeseecceeeeelleeeee below + world_above dooes aaaaooeedeeeeeccoooeeaaaaaaeeeee
Sand-Piling. Both the sand and dust elements display additional behavior when falling. These elements cannot support themselves upright, and if there is an empty space to their bottom-left or bottom-right then the element will fall into that location. In practice, this means that the sand elements form stable pyramids and hills.
To implement the sand-piling behavior, check if each element is sand-type (sand dust), then if either the bottom-left or bottom-right space is lower density, the two elements swap. To prevent ambiguity from left/rghtt falling and break symmetry, random value is sampled for each position dictating whether to check bottom-left first or bottom-right.
Water-Flowing. Water and other fluids (water, lava, acid) behave similarly sand, except that these elements can also flow left and right. This behavior is implemented in similar fashion to sand- sand-piling, except that the left and right adjustment elements are considered for swapping, instead of bottom-left and bottom-right.
In addition, an optimization implemented to increase the effective velocity of fluids. With the naive setup, each fluid can flow either left or right. However, this means that large clumps of fluids often have a hard time spreading out, as they rely on random osmisis in order to fully spread horizontally. To speed up the process, an index is reserved to keep track of the direction that given fluid element has previously flowed in. If fluid has previously flowed to the left, then the next timestep will first check if it can that a single particle of water will continue flowing left until it hits wall, rather than randomly dynamic fluid system.
to the left (rather than randomly selecting left/right). This ruleset means
move between left/right, creating
14

---

Ice and Water, The goal with ice and water create two elements with different phases (solid and liquid), which transition between one another depending on their number of neighbors. Ice has default chance of melting into water (2% timestep), and water has chance of freezing into ice it has three or more ice neighbors (5% a timestep).
To compute the number of neighbors for this behavior and more, simple convolutional kernel is employed. The kernel has a 3x3 kernel size and contains all ones, resulting in each position after running the convolution containing the number of a specific element that exist next to that position.
self. nelf.neighbor kerrgbbooreereel torch. ones ((1, 1, 3, 3), device=device) (world, "ice"),
water_ _aa_can___ lal_ccnn_eeeeeeee = (F (seff.d(se
self nelf.neighbor kernel, padding=1) >= 3)
does_turn_ice
helf.get_bert (world, "water") & vater_can_freezz & (ice_chance 0.05)
world[:] - interp (switch sossttchhoessstunnccce if fflfeelewwoddd if_true-selle eeem_ ('ice' i)
Fire-Burning. Fire is implement as an element which cannot survive its own. Fire always has a chance to burn away into air. However, fire that is next to a burnable element (wood, plant, dust) will not burn away, and instead has chance transform that element into fire. Thus, fire will travel along the paths of burnable elements, setting fire to anything close enough to touch. Fire also naturally moves upwards, thus fire can jump from one element another even across small air gap.
Note that each burnable element has distinct behavior when burned. Wood burns the slowest, as it has the lowest chance of burning when in contact with fire element. Plant burns faster, and dust burns the fastest and additionally creates large amounts of velocity when burned. This aims to simulate an explosive effect as the velocity will scatter nearby particles outwards.
Plant-growing. Plants spread in water, but in an incomplete fashion. The intent is to create a vine-like structure where plants absorb water and create more plant, leaving behind web of plant elements. Specifically, water elements that are near greater than four plants have a chance to turn into either water or air.
Lava. Lava a liquid that flows similarly to water. Lava that is exposed to air continuously creates fire at those positions, thus lava acts as constant source of fire that does not naturally dissipate. As lava obeys gravity and fluid flowing. lava can flow down structures and reach new locations. Lava that comes into contact with water forms stone element in that location.
Acid. Acid is a fluid which destroys other elements. Specifically, if acid is neighboring a non non-empty element, then there is 20% that the acid block will dissapear along with any neighboring elements. Normally, elements should not be able to interact with all its neighbors simultaneously (may cause conflicts), but since acid only destroys things, there is no issue.
The final element is a cloner, which keeps track of the first element that touches it, then continuously produces that element any adjacent empty positions. Cloner elements meant to be used as a source of mobile elements such as water or gas, and once assigned on contact can create structures such as waterfalls or spouts.
Cloner.
Velocity System. Powderworld elements interact with each other via elemental reactions as well as global velocity system. Each position in the world holds an x and y velocity, and blocks move if the magnitude of the velocity at that position is greater than a threshold. The moving behavior consists of a loop over the eight primary directions, and first checks if the velocity at each location is aligned in that direction. Then, assuming the velocity magnitude is great enough, the procedure checks if there an empty space in that direction, and if so, the element moves. All elements are effected by velocity except for walls.
Additionally, the velocity layer also goes through its own simulation. While other powder games use fluid dynamics to simulate velocity, this work instead opts to use simpler but quicker method. Specifically, velocities create additional velocity in the direction they point in. This is done during slightly averaged with their neighbors, and the entire
the same loop as above. Next, velocities velocity layer is scaled down by a factor of 0.95. Overall, the velocity simulation allows velocity values to travel forwards in the direction they point in, while slightly spreading out and decaying.
15

---

Test Loss vs. # of Training lasks
Start State
Tasks
100 Tasks
Tesks
11
0.25
100 Taskss
0.20
1k Tasks
0.6
10k Tasks
100k Tasks
Lots
True Future
T0K tasks
0.10
0.05
0.0
0.00
2000
Figure 11: World model generalization improves as the number starting states increased. Shown are the test performances of world models trained with data from 10 states, 100 states, 1000 states, etc. Models trained on less data show greater instability, as observed by the spikes test loss. Right: Comparison of sampled world model predictions on test state.
Test Loss vs. Training Task Complexity
Only Circles
Only Squares
Circles + Squares
Cire Sqr Line
11
615
Start State
Lines
0.5
0.20
6.6
0.15
+ Sar
0.3
Sqr
0.10
Lin, Sqr
Future Statt
0.2
0.05
2000
4000
Figure 12: Increasing environment complexity by including additional shapes improves trans- fer performance. World models trained tasks including lines, circles, and squares create diver- sity, enabling generalization to unseen tasks. Right: Sampled tasks generated with varying types of shapes.
Test Loss vs. of Lines Trained On
Lines
Lines
Lines
Lines
i
1 Lines
Start State
4 Lina
Lines
0:00
52 Lines
015
3.10
Futura State
0:00
0.05
0.00
0.00
2000
Figure 13: Training environments with lines results in stronger generalization. A higher number of lines increases the diversity of training states, but may also create destructive reactions.
Tast Finetune Elaments Trainnd
$11
0.85
Flements
Start State
0.50
Elements
Bements
255
0.8
3lements
Elements
Future Stato
05
6.7
25
100
Figure 14: World models trained on more elements showcase better performance when fine- tuned on novel clements. When transferring to environment containing three held-out elements (gas, stone, acid), models exposed to more elements during training perform better. These results show that Powderworld provides rich enough simulation that world models learn robust represen- tations capable of adaptation. The richer the simulation, the stronger these representations become.
16

---

Starting
Starting
Prediction
Prediction
Truth
Truth
Environment: Only Circles
Environment: Only Squares
Starting
Starting
Prediction
Prediction
Truth
Truth
Environment Only Lines
Environment Circles Squares
Starting
Starting
Prediction
Prediction
Truth
Truth
Environment: Circlos
Lines
Environment Circles Squares
Figure 15: Training tasks at various environment complexities. Environment complexity is de- fined the number of shapes included within the procedural generation algorithm. Each shape is generated up to five times, at random positions, sizes, and with a random element.
17

---

Starting
Starting
Prediction
Prediction
Truth
Truth
Environment: Line
Environment: Lines
Starting
Starting
Prediction
Prediction
Truth
Truth
Environment 4 Lines
Environment: Lines
Starting
Starting
Prediction
Prediction
Truth
Truth
Environment:
Environment:
Figure 16: Training tasks at various numbers of lines. Note that each line number represents the maximum possible number of lines, thus the 4-Line environment can generate 10.1,2.3,4] lines. Blank worlds appear when no lines are generated, lines are generated out of unstable elements (e.g. fire) that disappear over time.
18

---

Sand Falling
Stone Bridge
Ice and Plant
Burning Vine
on Wood
Water Frow
Dust on Lava
Gas Flow
Starting
*Truth*
Squares
Circles
K
Lin
Sqr + Cir
5qr+
Sqr Lin
Line
Lines
K
Lines
Figure 17: World model predictions over test tasks. This figure showcases the eight test tasks simulated for 16 timesteps into the future. World models trained to predict 8 timesteps forwards, thus results are shown with each world model applying two updates.
19

---

Falling
Bridge
Burning
Gas Flow
Starting
"Truth"
Lines
K
16 Lines
32 Lines
K
10 Tasks
100 Tasks
Tasks
10K Tasks
100K Tasks
1000K Tasks
K
Figure 18: Cont: World model predictions over test tasks.
20

---

Validation Loss vs. Training Task Complexity
Benchmark Loss vs. Tralning Task Comp axity
115
0.1
Lires
Liee
03
3.20
0.5
0.20
Lin
Gir
0.4
3.15
Ci + Sar
Denchmerk Lass
0.4
0.15
Cir Sar
0.3
Sig
0.3
3.10
CP. Lin, Siy
0.10
Cir, Lin. Sqr
02
0.2
0.05
2000
4000
2000 4000
Figure 19: World model training curves on environments with increasing complexity. The Validation loss represents the performance of the world models on tasks sampled from their training distribution (i.e. only circle, only squares, etc.) Note that these tasks are never actually seen during training. The loss represents performance over an arbitrarily-hhosen set of tasks, specifically, worlds with 5 lines.
Validation Loss vs, & of Lines Trained On
Benchmark Loss vs. + of Lines Trained On
1
1 Lines
Lines
0.20
Lines
0200
0:30
4 Lines
0.20
Lines
E Lines
Lines
0.15
Uins
Senthreee Lete
0:15
15
0020
32 Lines
0:20
32 Lines
Weldeien
015
0.10
0:15
0.10
0.00
0.05
0.05
0.05
0.00
0.00
0.00
0.00
Figure 20: World model training curves on environments with increasing number of lines. Note the mirrored correlations: world models trained on more complex environments show higher validation loss (as the tasks are harder) but lower benchmark loss.
Validiation Loss vs. of Training Tasks
Benchmerk Loss vs. of Training Tasks
0125
10 Tasks
10 Tasks
100 Tasks
100 Tasks
3.20
020
0.6
10k Tasks
0.6
10k Tasks
3115
100k Tasks
Benchmork Lazs
015
100k: Tacks
Valeoatee
0.4
3000k Tasks
3.10
0.10
02
0.2
2000
2000
Figure 21: World model training curves on increasing number of training tasks, More training tasks consistently improves both Validation and Benchmark performance.
Tonder Oiee Thentrpe-
Teaaee Eans Transeu-
Yernttr Severppdt
64
Figure 22: Transfer performance to novel elements, over fine-tune time. Showcased are world models trained on increasing numbers of elements, then fine-tuned on an environment with three novel elements (gas, stone, acid). While in the zero-shot setting there is little correlation in perfor- mance, fine-tuning reveals that models trained on larger numbers of elements can more efficiently adapt to the new environment.
21


Reviews:
[{"review_id": "okcUBFdlpt", "reviewer": "Reviewer_EMHS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new simulation environment (Powderworld) to facilitate the study of generalization across tasks which share rules. The environment is extensible and designed for speed and modern GPU hardware. Experiments demonstrate how Powderworld can be used to study generalization in world modeling and RL.\n", "summary_of_the_review": "The primary contribution is a novel tool consisting of a simulation environment. Its release might help advance the community better study an important problem in agent design. Difficult to predict the adoption of this tool but I don't see any reason to not add it to the toolbox. The experimental clarity of the 2 demonstrated uses of Powderworld (world modelinng and RL) could be improved.\n\n-----\n\nUPDATE: I thank the authors for their detailed feedback and engagement with the reviewers. The revised paper seems much improved. After reading the other reviews and the author response, I'm more inclined to accept this paper. It doesn't meet my bar for a strong accept though.\n\n", "strength_and_weaknesses": "Strengths\n  - Novel environment designed to facilitate study in key topic in agent design (generalization across tasks). The design makes it easy to extend the environment, vary task distributions and generate training data inexpensively.\n  - Efficient design and implementation of the simulator, leveraging modern GPU hardware.\n\nWeaknesses\n  - Some parts of the experimental setup could be more clearly described. An illustration of the trajectory generation process for training and inference might help improve clarity in this part of the paper.\n    - I was confused by the implemetation details and scaling law claims in Sec 4.1.1. \n      - Were each of the 5 models in Fig 5 trained on the same total amount of simulation data or not? Does the phrase \"training data\" in the line \"world models trained on increasing amounts of training data display higher performance on a set of test tasks\" refer to the number of tasks while keeping the total number of simulations constant across all 5 models? If yes, how exactly were these trajectories sampled? And does it mean that Fig 5 right is a sample observation (W_15) from the predicted logit probs (W').\n      - What exactly is a run and a trial? I found the statements \"each run trains on 10x more data as the previous\" and \"Runs are trained for three trials and the average test loss over training time is displayed\" confusing.\n    - A similar question arises in Sec 5. How exactly are the 2.5M datapoints for the RL agent generated for agents restricted to T training tasks?\n  - (More a question than a weakness) How well does powderworld support human agents? More concretely, how easy would it be to compare a RL agent with a human agent in Powderworld?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "msLnbb2LG2E", "reviewer": "Reviewer_pDhR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a new environment for supporting RL research, especially for more physics-oriented tasks.  In the environment, each \"pixel\" may be occupied by a certain type of material, and the material properties and environmental forces define how the update should be updated to the next state, and can be accomplished through very localized and efficient computation.  Special care was given such that the environments are simple to specify (and have a matrix representation), a diverse number of environments is easily configurable, and with the provide set of materials and rules, model generalizability to novel situations is easy to test.  The authors apply RL and world model baselines to an example set of environment worlds.\n\n\n\n\n", "summary_of_the_review": "The authors provide an environment which is:\n- fast\n- easily configurable\n- has a small set of fixed rules and properties controlling a rich space of observable interactions\n- mostly unique wrt other environments I'm familiar with\n\nThrough the paper discussion and experiments, they make a compelling case for using this environment if the task domain is relevant to ones research.  They outline a number of possible task scenarios that seem useful, but moreover, the dataset seems flexible enough to cover many unmentioned task scenarios.  I find it hard to find fault in the environment, and suggest not accepting the paper only if it is considered not publication-worthy in terms of its research content (though precedent on similar environment papers would make a good case for accepting it).\n\nUpdate: I'm updating the correctness score 1 -> 3.  It was incorrectly entered and so I hope it didn't cause too much confusion juxtaposed next to my otherwise positive review.", "strength_and_weaknesses": "It seems difficult to find an argument against this work and this environment.  In favor of this work, the environment seems very well thoughtout, and it occupies a niche that seems to be important and unfulfilled at the moment -- an environment that is easy for users to specify novel worlds and tasks, and in which the world interactions that occur at each step are complex and take place throughout the environment (thus maybe facilitating more explorations towards more real-world scenarios, while still maintaining many of the setup and computation we enjoy with toy environments).  Frankly, if I worked in an area where this environment would be more applicable, I would be strongly inclined to use it, and therefore I'm strongly inclined to accept it.  It looks great to me!  \n\nThe downsides of this work is that the experimental sections offer little in terms of useful experimental results.  The experiments exist only to sanity-check that the environment is suitable for the types of generalization experiments that are of the most interest to the authors, and to further reproduce tried-and-true beliefs about the relation between train environments, models, and generalization.  As such, assessing the merit of the paper as a publication is really (mostly) about considering the value of the software as a resource.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ms6pkT42CbU", "reviewer": "Reviewer_oDrU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Powderworld, a fast simulated environment capable of producing diverse tasks for either supervised learning or an RL agent. The paper demonstrates that Powderworld can be used for both learning world models and training RL agents in a sandpushing task. ", "summary_of_the_review": "At the surface the paper presents an interesting and unique environment that may facilitate future research. However, the two presented use cases are a world modelling task which is not directly relevant outside powderworld, and an RL task where the main result confirms what we already know from Procgen et al. Overall I do not think the paper contributes anything new at present. \n\nPost discussion:\n\nThe paper is completely revised so now provides some useful, fast settings that can be used for work in PCG and UED for RL. Analysis shows this can be a worthy testbed and may help researchers enter what is a growing and important field. ", "strength_and_weaknesses": "### Strengths\n- The paper is well-written and well-motivated, as training agents on large, diverse task distributions has become increasingly popular and important as we seek more general agents.\n- The environment seems very fast, given it runs on a GPU, which is great for iteration speed for research.\n\n### Weaknesses\n- The world modelling task is definitely interesting but it is hard to see how it is directly relevant outside of this environment. We would likely never have access to a Markovian state in such a controlled setting. The section appears to be motivated by works such as World Models and Dreamer, but in those cases 1) the models are learned directly from pixels without a Markovian state 2) there is an agent taking actions in the world. So this is a totally different paradigm. The fact that the model generalizes better with more data here is expected, as the authors note this has been the case in a variety of other settings already.\n- How can we be sure the hand designed tasks are unbiased? For all we know they could be somewhat arbitrary.\n- While the motivation in the intro is that this world is more general than others such as MiniGrid/Crafter/MiniHack, the only RL task presented is just sand pushing. How is this more diverse and useful than for example the tasks in Crafter/MiniHack which vary from navigation to tool use?\n- It looks like the experiments were all just one seed. When we know RL training is volatile, it seems like an oversight to have done this given the environment is meant to be fast. \n- One of the motivations in the intro is the potential use for UED, but there is no demonstration of this. It would be interesting to see if this environment offers something unique here vs. the alternatives. It may be beyond the scope to run this for a rebuttal but it would likely see an increased score.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5nvhWtNR8EZ", "reviewer": "Reviewer_fLgh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Powderworld, a platform to build environments on which to train world models and RL agents. The main purpose of this platform is to investigate generalization. \n", "summary_of_the_review": "This is a strong paper introducing a new platform for designing RL environments. I believe Powederworld can be very valuable to the community. \n", "strength_and_weaknesses": "### Strengths\n\nThe platform supports the design of environments with different levels of complexity and many degrees of freedom (i.e. many different elements can be added to the environment).\n\nIt can also run many instantiations (multiple worlds) in parallel on a GPU, which translates into runtime efficiency. \n\nEnvironments can be either manually or procedurally generated, which means that one can quickly generate thousands of different worlds to train on.\n\nThe environments are also modular in the sense that the different elements interact only locally with their neighbors. \n\nThe paper provides extensive experimentation, especially for the world model case.  The results are intuitive and generally agree with our intuitions of how adding more, tasks and/or complexity increases generalization.\n\n### Weaknesses\n\nAccording to the paper, the interactions between elements are computed via convolutional matrix multiplications. This means that the transition dynamics can be model **exactly** with a convolutional neural network (CNN). That is, theoretically, a CNN can be trained to zero loss, provided you can find the global optimum. An increase in complexity (i.e. adding more elements to the environment) can be resolved by adding more kernels to the CNN such that each kernel can potentially model each of the possible interactions. Can the authors comment on this? Or even better, provide results showing whether zero loss can be obtained for simple environments. Note that this won’t affect my score but I am curious to know if this is the case.\n\nSince the main purpose of this paper is training RL agents I am missing more experiments showing what can be done with this platform. So far I can only think of tasks like moving objects (elements) from one place to another or removing ceratin elements. Can the authors provide examples of other tasks?\n\nRelated to my previous question, have you considered adding agents (e.g. robots) that can interact with the elements?\n\nIn the last RL experiment, you mention that increasing the complexity of the environments decreases generalization performance. Have you tried to start with simple environments and gradually increase the complexity as the agent improves its policy?\n\nIn the first reference, there is a link to a site where there is a game called powder game. Is this the Powderworld platform?\n\nPerhaps this question is too technical but, how do you take care of the non-linear operations (e.g. gravity), how can those be performed on a GPU?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%