🏘️ ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Authors: Matt Deitke
Eli VanderBilt
Alvaro Herrasti
Luca Weihs
Kiana Ehsani
Jordi Salvador
Winson Han
Eric Kolve
Aniruddha Kembhavi
Roozbeh Mottaghi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Embodied AI
Large-Scale Environments
Procedural Generation

Abstract: Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding. This work presents a platform to enable similar success stories in Embodied AI. We propose ProcTHOR, a framework for procedural generation of Embodied AI environments. ProcTHOR enables us to sample arbitrarily large datasets of diverse, interactive, customizable, and performant virtual environments to train and evaluate embodied agents across navigation, interaction, and manipulation tasks. We demonstrate the power and potential of ProcTHOR via a sample of 10,000 generated houses and a simple neural model. Models trained using only RGB images on ProcTHOR, with no explicit mapping and no human task supervision produce state-of-the-art results across 6 embodied AI benchmarks for navigation, rearrangement, and arm manipulation, including the presently running Habitat 2022, AI2-THOR Rearrangement 2022, and RoboTHOR challenges. We also demonstrate strong 0-shot results on these benchmarks, via pre-training on ProcTHOR with no fine-tuning on the downstream benchmark, often beating previous state-of-the-art systems that access the downstream training data.

ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Matt Deitke Eli VanderBilt', Alvaro Herrastit, Luca Weihs Jordi Salvadori, Kiana Ehsani', Winson Han Eric Kolve Ali Farhadi Aniruddha Kembhavito, Roozbeh Mottaghi to PRIOR @ Allen Institute for AI, University of Washington, Seattle procthor www.aallelnooo. org
Abstract
Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding. This work presents a plat- form to enable similar success stories in Embodied AI. We propose PROCTHOR, framework for procedural generation of Embodied Al environments. PROCTHOR enables sample arbitrarily large datasets of diverse, interactive, customizable, and performant virtual environments to train and evaluate embodied agents across navigation, interaction, and manipulation tasks. We demonstrate the power and potential of PROCTHOR via sample of 10,000 generated houses and simple neural model. Models trained using only RGB images PROCTHOR, with explicit mapping and no human task supervision produce state-of-the-ar results across embodied AI benchmarks for navigation, rearrangement, and arm manipu- lation, including the presently running Habitat 2022, A12-THOR Rearrangement 2022, and RoboTHOR challenges. We also demonstrate strong O-shot results on these benchmarks, via pre-training on PROCTHOR with no fine-tuning on the downstream benchmark, often beating previous tate-of-the-ar th-the-rrt systems that access the downstream training data.
1 Introduction
Computer vision and natural language processing models have become increasingly powerful through the use of large-scale training data. Recent models such CLIP [45], DALL-E [47]. GPT-3 [7], and Flamingo [2] use massive amounts of task agnostic data to pre-train large neural architectures that perform remarkably well at downstream tasks, including zero and few-shot settings. In comparison, the Embodied AI (E-AI) research community predominantly trains agents in simulators with far fewer scenes [46, 29, 13]. Due to the complexity of tasks and the need for long planning horizons, the best performing E-AI models continue to overfit on the limited training scenes and thus generalize poorly to unseen environments.
In recent years, E-AI simulators have become increasingly more powerful with support for physics, manipulators, object states, deformable objects, fluids, and real-sim counterparts [29, 48, 49, 19, 60), but scaling them up to tens of thousands of scenes has remained challenging. Existing E-AI environments either designed manually [29, 19] obtained via 3D scans of real structures [48, 46). The former approach requires 3D artists to spend a significant amount of time designing 3D assets, arranging them in sensible configurations within large spaces, and carefully configuring the right textures and lighting these environments. The latter involves moving specialized cameras through many real-world environments and then stitching the resulting images together to form 3D reconstructions of the scenes. These approaches are not scalable, and expanding existing scene repositories multiple orders of magnitude not practical.
We present PROCTHOR, a framework built off of A12-THOR [29], to procedurally generate fully- interactive, physics-enabled environments for E-AI research. Given room specification (e.g.,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: We propose PROCTHOR, a framework to procedurally generate a large variety of diverse, interactable, and customizable houses.
house with 3 bedrooms, baths, and kitchen), PROCTHOR can produce large and diverse set of floorplans that meet these requirements (Fig. 1). A large asset library of 108 object types and 1633 fully interactable instances is used to automatically populate each floorplan, ensuring that object placements are physically plausible, natural, and realistic. One can also vary the intensity and color of lighting elements (both artificial lighting and simulated skyboxes) in each scene, to simulate variations in indoor lighting and the time of the day. Assets (such as furniture and fruit) and larger structures such as walls and doors can be assigned variety of colors and textures, sampled from sets of plausible colors and materials for each asset category. Together, the diversity of layouts, assets, placements, and lighting leads to an arbitrarily large set of environments allowing PROCTHOR to scale orders of magnitude beyond the number of scenes currently supported by present-day simulators. In addition, PROCTHOR supports dynamic material randomizations, whereby colors and materials of individual assets can be randomized each time an environment is loaded into memory for training. Importantly, in contrast to environments produced using 3D scans, scenes produced by PROCTHOR contain objects that both support variety of different object states (e.g. open, closed, broken, etc.) and are fully interactive so that they can be physically manipulated by agents with robotic arms. We also present A ARCHITECTHOR, a 3D artist-designed set of 10 high quality fully interactable houses, meant to be used as a test-only environment for research within household environments. In contrast to A12-iTHOR (single rooms) and RoboTHOR (lesser visual diversity) environments, ARCHITECTHOR contains larger, diverse, and realistic houses.
We demonstrate the ease and effectiveness of PROCTHOR by sampling an environment of 10,000 houses (named PROCTHOR- 10K), composed of diverse layouts ranging from small 1-room houses to larger 10-room houses. We train agents with very simple neural architectures (CNN+RNN) without a depth sensor, and instead only employing RGB channels, with no explicit mapping and no human task supervision on PROCTHOR-10S and produce otate-of-the-art (SoTA) models on several navigation and interaction benchmarks. As of 10am PT on June 14th, 2022 we obtain (1) Robo ObjectNav Challenge [4] - O-shot performance superior to the previous So'T which uses RoboTHOR training scenes - with fine-tuning we obtain an 8.8 point improvement in SPL over the previous SoTA: (2) Habitat ObjectNav Challenge 2022 [39]- - top of the leaderboard results with > point gain in SPL over the next best submission; (3) 1-phase Rearrangement Challenge 2022 [3] top of the leaderboard results with Prop Fixed Striet improving from 0.19 to 0. 0.245; (4) AI2-TTHOR ObjectNav - O-shot numbers which already outperform a previous model that trains on A12-ITHOR, with fine-tuning we achieve success rate of 77.5%: (5) ArmPointNav [16] - O-shot number that beats previous SoTA results when using RGB; and (6) Architec THOR ObjectNav large success rate improvement from 18.5% to 31.4%. Finally, an ablation analysis clearly shows the advantages of scaling up from 10 to 100 to IK and finally to 10K scenes and indicates that further improvements can be obtained by invoking PROCTHOR to produce even larger environments.
In summary, our contributions are (1) PROCTHOR, a framework that allows for the performant procedural generation of an unbounded number of diverse, fully-interactive, simulated environments, (2) ARCHITECTHOR, new, 3D artist-designed set of houses for E-AI E-AI evaluation, and (3) SoTA results across six E-AI benchmarks covering manipulation and navigation tasks, including strong 0-shot results. PROCTHOR will be open-sourced and the code used in this work will be released.
2 Related Work
Embodied AI platforms. Various Embodied AI platforms have been developed over the past several years [29, 48, 49, 60, 19, 58). These platforms target different design goals. A12-THOR [29] and its variants (ManipulaTHORR [16) and RoboTHOR [13)) are built in the Unity game engine and focus on agent-object interactions, object state changes, and accurate physies simulation. Unlike
2

---

A12-THOR, Habitat [48] provides scenes constructed from 3D scans of houses, however, objects and scenes are not interactable. A more recent version, Habitat 2.0 [51], introduces object interactions at the expense of being limited to one floorplan and syntheric scenes. iGibson [49] includes photo- realistic scenes, but with limited interactions such as pushing. iGibson 2.0 [32] extends iGibson by focusing on household tasks and object state changes in synthetic scenes and includes a virtual reality interface. ThreeDWorld [19] targets high-fidelity physics simulation such as liquid and deformable object simulation. VirtualHome [44] is designed for simulating human activities via programs. RLBench [27], RoboSuite [67] and Sapien [60] target fine-grained manipulation. The main advantage of PROCTHOR is that we can generate diverse set of interactive scenes procedurally, enabling studies of data augmentation and large-scale training in the context of Embodied AL.
Large-scale datasets. Large-scale datasets have resulted in major breakthroughs in different domains such as image classification [15, 30), vision and language [11, 52], 3D understanding [9, 61], autonomous driving [8, 50], and robotic object manipulation [43, 40J. However, there are not many interactive large-scale datasets for Embodied AI research. PROCTHOR includes interactive houses generated procedurally. Hence, there are an arbitrarily large number of scenes in the framework. The closest works to ours are [46, 42, 33]. HM3D [46] is a recent framework that includes 1,000 scenes generated using 3D scans of real environments. PROCTHOR has number of key distinctions: (I) unlike HM3D which includes static scenes, the scenes in PROCTHOR are interactive i.e., objects can move and change state, the lighting and texture of objects can change, and a physics engine determines the future states of the scenes; (2) it is challenging to scale up HM3D as it requires scanning house and cleaning up the data, while
can procedurally generate more houses; (3) HM3D can be used only for navigation tasks (as there is no physics simulation and object interaction), while PROCTHOR can be used for tasks other than navigation. OpenRooms [33] is similar HM3I) in terms of the source of the data (3D scans) and dataset size. However, OpenRooms is interactive. OpenRooms is also confined to the set of scanned houses, and it takes significant amount of time to annotate a new scene (e.g., labeling materials for one object takes I minute), while PROCTHOR does not suffer from these issues. Megaverse [42] is another large-scale Embodied AI platform that includes procedurally generated environments. Although it is impressive in terms of simulation speed, it includes only game-like environments with a simplified appearance. In contrast, PROCTHOR mimics real-world houses in terms of the complexity of appearance, physics, and object interactions.
Scene synthesis. Work on scene synthesis is typically broken down into generating floorplans [35, 36, 24, 57] and sampling object placement in rooms [17, 20, 65, 66, 12]. Our work aimed to generate diverse and semantically plausible houses using the best existing approaches building on existing works in areas that were insufficient for our use case. Our floorplan generation process is adapted from [35, 36], which takes in a high-level specification of the rooms in house and their connectivity constraints, and randomly generates floorplans satisfying these constraints. Our object placement is most similar to [66, 20, 65, 62, 10], where we iteratively place objects on floors, walls, and surfaces and use semantic asset groups to sample objects that co-occur (e.g. chairs next to tables). The modular generation process used in this work makes it easy swap and update any stage of our house generation pipeline with a better algorithm. In this work, we found the procedural generation approaches to be more reliable and flexible than the ones based on deep learning when adapting it to our custom object database and when generating more complex houses that were out of the distribution of static house datasets [18, 57, 34]. For more detailed comparison, including a discussion of some of the limitations of deep learning approaches, please refer to the Appendix.
3 PROCTHOR
PROCTHOR is a framework to procedurally generate B-AI environments. extends A12- AI2-THOR and, thereby, inherits AI2-THOR's 's large asset library, robotic agents, and accurate physics simulation. Just as in scenes painstakingly created by designers in A12-THOR, environments in PROCTHOR are fully interactive and support navigation, object manipulation, and multi-agent interaction.
Fig. 2 shows a high-level schematic of the procedure used by PROCTHOR to generate scene. Given a room specification (e.g. house with 1 bedroom sampling to, iteratively, generate a floor plan, create an external wall structure, sample lighting. and doors, then sample assets including large, small and wall objects, pick colors and textures, and determine appropriate placements for assets within the scene. We refer the reader to the appendix for details regarding our procedural generation and sampling mechanism, but highlight five key characteristics of PROCTHOR Diversity, Interactivity, Customizability, Scale, and Efficiency.
bathroom), we use multi-stage conditional
3

---

Sample Interior
Create Structure
Sample Doors
Sample Large Objects
Sample Surface Objects
Bedroom
Bathroom
Kitchen
Living
Sample Room Spec
Sample Floor Plan
Add Lights
Sample Structure Materials
Sample Objects
Figure 2: Procedurally generating house using PROCTHOR.
Diversity, PROCTHOR enables the creation of rich and diverse environments. Mirroring the success of pre-trie pre-training models with diverse data in the vision and NLP domains, demonstrate the utility of this diversity on several E-AI tasks. Scenes PROCTHOR exhibit diversity across several facets:
Figure 3: Floorplan diversity. Examples showing the diversity of the generated floorplans. Rooms the house are colored by Bedroom, Bathroom, Kitchen, and Living Room.
Diversity of floor plans. Given a room specification, we first employ iterative boundary cutting to obtain external scene layout (that range from simple rectangle complex polygon).
The recursive layout generation algorithm by Lopes et al. [35] is then used to divide the scene into the desired rooms. Finally, determine connectivity between rooms using set of user-defined constraints. These procedures result in natural room layouts (e.g.. bedrooms are often connected to adjoining bathrooms via door, bathrooms more often have single entrance, etc). As exemplified in Fig. 3, PROCTHOR generates hugely diverse floor plans using this procedure.
Figure 4: Object diversity. A subset of instances for four object categories.
Diversity of assets. PROCTHOR populates scenes with small and large assets from its database of 1633 household assets across 108 categories (examples in Fig. 4). While many assets are inherited from A12-THOR. we also introduce new assets such windows, doors, and countertops, hand- designed by 3D graphic designers. Asset instances are split into train/val/test subsets and are interactable, i.e. objects can be picked and placed within the scenes, some objects have multiple states (e.g. light can be or off) and several objects consists of parts with rigid body motions (e.g. door on a microwave).

---

Figure 5: Material augmentation. Different materials for objects and structural elements.
Diversity of materials. Walls have two kinds of materials or one of 122 wall textures such as brick and tile. also provide 55 floor materials. The ceiling material for the entire house is sampled from the set of wall materials. PROCTHOR also provides the ability to randomize materials of objects. Materials are only randomized within categories, which ensures objects still look and behave like the class they represent.
of 40 solid (and popular) colors
Figure 6: Object placement. Four examples of object placement within the same room layout.
Diversity of object placements. Asset categories have several soft annotations that help place them realistically within a house. These include room assignments (e.g. couch in living room but not bathroom) and location assignments (e.g. fridge along wall, TV not on the floor). We also develop the notion of Semantic Asset Group (SAG) groups of assets that typically co-occur (e.g. dining table with four chairs) and thus must be sampled and placed using dependent sampling. Given layout, individual assets and SAGs that on the floor are sampled and placed iteratively, ensuring that rooms continue to have adequate floor space for agents navigate and manipulate objects. Then wall objects such as windows and paintings get placed, and finally, surface objects (ones found on top of other assets) are placed (e.g. cups the kitchen counter). This sampling allows for large and diverse set of object choices and placements within any layout. Fig. 6 shows such variations.
Figure 7: Lighting variation. Morning, dusk, and night lighting for example scene.
Diversity of lighting. PROCTHOR supports single directional light (analogous to the sun) and several point lights (analogous to lightbulbs). Varying the color, intensity, and placement of these sources allows us simulate different artificial lighting, typically observed houses, and also at different times of the day. Lighting has significant effect the rendered images as seen in Fig. 7.
5

---

Figure 8: Interactivity. Object states can change (e.g., the laptop or the lamp in the left panel), and the agents can interact with objects and other agents (middle and right panels).
Interactivity. A key property of PROCTHOR is the ability interact with objects to change their location or state (Fig. 8). This capability is fundamental to many Embodied AI tasks. Datasets like HM3D [46] that created from static 3D scans do possess this capability. PROCTHOR supports agents with arms capable of manipulating objects and interacting with each other.
Figure 9: Customizability. PROCTHOR be used to construct custom types such as classrooms, libraries, and offices.
Customizability. PROCTHOR supports many room, asset, material, and lighting specifications. With a few simple lines of specification, one can easily generate customized environments of interest. Fig. 9 shows examples of such varied scenes (classroom, library, and office).
Scale and Efficiency. PROCTHOR currently uses 16 different scene specifications seed the scene generation process. These can result in over 100 billion layouts. PROCTHOR uses 18 different Semantic Asset groups and 1633 assets. These can result in roughly 20 million unique asset groups. Each of these assets can be placed in numerous locations. In addition, each house gets scaled and uses variety of lighting. This diversity of layouts, assets, materials, placements, and lighting enables the generation of arbilrarily large sets of houses either statically generated and stored as a dataset or dynamically generated at each iteration of training. Scenes are efficiently represented in a JSON specification and are loaded into A12-THOR at runtime, making the memory overhead of storing houses incredibly efficient. Scene generation fully automatic and fast and PROCTHOR provides high framerates for training E-AI models (see Sec. 4 for details).
PROCTHOR-10K
We demonstrate the power and potential of PROCTHOR using a sampled set of 10,000 fully in- teractive houses obtained by the procedural generation process described in Section which we label PROCTHOR-10K An additional set of 1,000 validation and 1,000 testing houses are available for evaluation. Asset splits across train/val/test are detailed in the Appendix. All houses are fully navigable, allowing agent to traverse through each room without any interaction. In terms of scale,
Objects
Housee Area Distmbutton
Datribbction closam Number
3 Rooms Houses
8,000
Room Honsee
7.10 Room Houses
5.000
Couas
2,000
3.4
201
20
2.3
3.0
100
250
30.
400
5100
Number Otgects Boots
Area of T.nzoo (1e7)
Numbber Raoma
Figure 10: PROCTHOR-10K statistics. Left: distribution of the number of objects in each room; Middle: distribution of the area of each house, bucketed into small, medium, and large houses; Right: bar plot showing the distribution over the number of rooms that make up each house.

---

Navigation FPS
Isolated Interaction FPS
Environment Query FPS
Compute
Small
Large
Small
Large
Small
Large
8 GPUs
8,599
3,208
6,488 1:20
2.8611107
180,205:119..m
433,587
1 GPU
1,427:174
6,280.140
1,265=71
5971337
160.622.12.846 157,567:12.689
Process
240.169
195.119
180.142
93.115
14,825
14,9161186
Table 1: Rendering speed. Benchmarking FPS for navigation (e.g. moving/rotating), interaction (e.g. pushing object), and querying the environment for data (e.g. checking the dimensions of the agent). We report FPS for Small and Large houses. See Appendix for details.
PROCTHOR-10K is one of the largest sets of interactive home environments for Embodied AT - as
comparison, A12-iTHOR [29] includes 120 scenes, RoboTHOR [13] has 89 scenes, iGibson [49] has 15 scenes, Habitat Matterport 3D [46] has 1,000 static (non-interactive) scenes, and Habitat 2.0 [51] has 105 scene layouts. Scaling beyond 10K houses is straightforward and inexpensive. This set of 10K houses was generated in hour a local workstation with 4 NVIDIA RTX A5000 GPUs. Fig. 11 shows examples of ego-centric and top-down views of houses present in PROCTHOR-I0K.
Figure 11: Example scenes in PROCTHOR-I0K with top-down and an egocentric view.
Scene statistics. Houses PROCTHOR-I0K are generated using 16 different room specifications. An example room spec is: A house with 1 bedruom connected to I bathroom, kitchen, and I living room and is visualized Fig. 2. Houses in this dataset have as few as room and as many as 10. Fig. 10 shows the distribution of areas (middle) and the number of rooms (right) of these generated houses. Our use of room specifications enables us to change the distribution the size and complexity of houses fairly easily. PROCTHOR-106 encompasses wider spectrum of scenes than A12-TTHÓR [29] and ROBOTHOR [13] (biased towards room-sized scenes) and Gibson [59] and HM3D [46] (biased towards large houses).
Rooms in each of these houses contain objects from 95 different categories including common household objects such fridges, countertops, beds, toilets, and house plants, and structure objects such as doorways and windows. Fig. 10 (left) shows distribution the number of objects per room per house, which shows that houses in PROCTHOR-10K are well populated. They also contain objects sampled via 18 different Semantic Asset groups Examples of Semantic asset groups (SAG) a Dining Table with 4 Chairs Bed with 2 Pillows. Given large asset library and SAGs, can create 19.3 million combinations of group instantiations.
Rendering speed. A crucial requirement for lcrge-scale training is high rendering speed since the training algorithms require millions iterations converge. Table shows these statisties. Experi- ments were run on a server with 8 NVIDIA Quadro RTX 8000 GPUs. For the I GPU experiments, we use 15 processes and for the GPU experiments, we use 120 processes, evenly distributed across the GPUs. PROCTHOR provides framerates comparable to iTHOR and RoboTHOR environments in spite of having larger houses (See Appendix for details), rendering it fast enough for training large models for hundreds of millions of steps in reasonable amount of time.
7

---

5 ArchitecTHOR
Figure 12: Top-down images of ARCHITECTHOR validation houses.
In order to test if models trained on ProcTHOR can generalize to real-world floorplans and object placements, test set of houses was needed. Neither iTHOR (single room scenes) nor RoboTHOR (dorm-sized maze-styled scenes) contain scenes that are representative of real-world homes. There- fore, we worked with professional 3D artists to create ArchitecTHOR, which contains 10 evaluation houses (5 val, 5 test) that mimick the style of real-world homes. Architec THOR val houses contain between 4-8 rooms, 121 + 26 objects per house, and typical floor size of 111 + 26 m By com- parison, PROCTHOR- 10K houses have a much higher variance, with between 1-10 rooms, 76 + 48 objects per house, and typical floor size of 96 + 74 m².
Experiments
Tasks. We now present results for models pre-trained on PROCTHOR-I0K on several navigation and manipulation benchmarks to demonstrate the benefits of large-scale training. We consider ObjectNav (navigation towards a specific object category) PROCTHOR, ARCHITECTHOR, RoboTHOR [13], HM3D [46], and A12-iTHOR [29]. We also consider two manipulation-basse
tasks: ArmPoint- Nav [16] and 1-phase Room Rearrangement [55]. In ArmPointNav, the agent moves an object using a robotic arm from source location to a destination location specified in the 3D coordinate frame. In Room Rearrangement, the goal is to move objects or change their state to reach a target scene state.
Models. Our models for all tasks consist of a CNN to encode visual information and a GRU to capture temporal information. We deliberately use simple architecture across all tasks to show the benefits of large-scale training. Our ObjectNav and Rearrangement models use the CLIP-based architectures of [28]. Our ArmPointNav model uses simpler visual encoder with 3 convolutional layers; we found this more effective than the CLIP encoder. All models are trained with the AllenAct [56] framework, see the Appendix for training details.
We present results in two settings: zero-shot and after fine-tuning on the training scenes provided by the downstream benchmark. Zero-shot experiments show us how well models trained on PROCTHOR generalize to new environments, whereas fine-tuning experiments tell us if repre- sentations learned from PROCTHOR can serve
Results.
good initialization for quick tuning. For all experiments, we use only RGB images (no depth and other modalities is used).
Zero-shot is particularly challenging since other environments have different appearance statistics, layouts, and object distributions compared to PROCTHOR. ARCHITECTHOR and A12-iTHOR [29] are high-fidelity drtist-designed scenes with high-quality shadows and lighting. HM3D is constructed from 3D scans of houses which can differ quite bit from synthetic environments. RoboTHOR [13] houses use wall panels and floors with very specific textures.
Zero-shot transfer results. Models trained only on PROCTHOR and evaluated 0-shot outperform previous SoTA models on 3 benchmarks (see Table 2). These strong results suggest that models generalize to not only unseen objects and scenes, but also new appearance and layout statistics.
Fine-tuning results. Further fune-tuning of the model using each benchmark's 's training data, achieves of-te-of-the-arr results on all benchmarks (refer to fine-tune rows of Table 2). Notably, our model is ranked first on three public leaderboards as of 10am PT, June 14th 2022: Habitat 2022 ObjectNav challenge, AI2-THOR Rearrangement 2022 challenge, and RoboTHOR ObjectNav challenge. should be noted that our model achieves these results using a very simple architecture and only RGB images. Other techniques typically use more complex architectures that include mapping or visual odometry modules and use additional perception sensors such as depth images.
8

---

Task
Benchmark
Method
Metries
Success
SPL
ObjectNav
RoboTHOR Challenge
EmbCLIP [28]
47.0%
0.200
ProRTHOR 0-shot
55.0%
0.237
ProCTHOR fine-tune
65.2%
0.288
Success
SPL
MINLCC
52.0%
0,280
Habitat Challenge (2022) HM3D-Sen Sem3D--emantics
FusionNav (AIRI)"
54.0%
0.270
ObjectNav
ProcTHOR 0-shot
9.00%
0.055
ProcTHOR fine-tune
53.0%
0.270
ProCTHOR Large" 0-shot
13.2%
0.077
ProcTHOR I.rgee fine-tune
54.4%
0.318
Success
SPL
ObjectNav
A12-THOR
EmbCLIP [28]
68 4%
0,516
ProcTHOR O-shot
75.7%
0.644
ProRTHOR fine-tune
77.5%
0.621
Success
SPL
ObjectNav
ARCHITECTHOR
EmhCLIP [28]
18.5%
0.118
ProRTHOR
31.4%
0.195
Success
Fixed Strict
Rearrangement
A12-THOR Challenge 1-phase (2022)
EmbCLIP [28]
7.10%
0.190
ProeTHOR 0-shot
3.80%
0.156
ProCTHOR fine-tune
7.40%
0.245
Success
PickUp SR
ArmPointNav
ManipulaTHOR
iTHOR- Seoo--Simpleoov [16]
29.2%
73.4
ProCTHOR 0-shot
37.9%
74.8
Table 2: Results for models trained ProcTHOR and evaluated O-shot and with fine-tuning
several E-AI benchmarks. For each benchmark we also compare to the relevant baselines (previous SoTA leaderboard submissions where applicable). "EmbCLIP [28] trained ROBOTHOR, EEbCLIP [28] trained on A12-iTHOR, "submission on the Habitat 2022 ObjectNav leaderboard [39]. For HM3D we present results when pretraining using the EmbCLIP architecture (which uses CLIP- pretrained ResNet50) as well as with a "Large" model which uses larger CLIP backbone CNN as well as a wider RNN, see supplement for details. "uses the model from [16] but retrains on the complete iTHOR data with RGB inputs. O-shot results, whereby models are pre-trained on PROCTHOR-10K and do not use any training data from the benchmark that they are evaluated on.
Scale ablation. To evaluate the effect of scale we train the models on 10, 100, 1K, and 10k houses. Here, we do not any material augmentations. As shown in Table the performance improves as we use more houses for training, demonstrating the benefits of large-scale data for E-Al tasks.
ARCHETECTHOR Test
ROROTHOR (0-Shot)
HM3D
A12-iTHOR (0-Shol)
Valid (0-Shol)
HOUSES
SPL
SR
SPL
SR
SPL
SR
SPL
SR
10 Houses
0.077
11.3%
0.040
8.53%
0.007
1.60%
0.249
28.7%
100
Houses
0.102
18.6%
0.076
20.9%
0,050
10.4%
0.352
42.0%
1,000 Houses
0.122
17.2%
0.157
33.1%
0.027
4.65%
0.456
53.0%
10,000 Houses
0.185
27.0%
0.210
44.5%
0.060
9.70%
0.554
64.9%
Table 3: Ablation study to evaluate the effect of the number of training houses. Each model is trained 80% success during training. Test performance increases with the number of training houses.
Conclusion
We propose PROCTHOR to procedurally generate arbitrarily large sets of interactive, physics- enabled houses Embodied Al research. We pre-train simple models on 10k generated houses and show SOTA results across 6 embodied tasks with strong O-shot results.

---

Acknowledgements
We would like to thank the teams behind the open-source packages used in this project, includ-
ing AI2-THOR [29], AllenAct [56], PRIOR [14], Habitat [48], Datasets [31], NumPy [23],
PyTorch [41], Pandas [38], Wandb [5], Shapely [21], Hydra [63]. SeiPy [53], UMAP [37], Net-
workX [22], EvalAI [64], TensorFlow [1], OpenAl Gym [6], Seaborn [54], PySAT [26], and Mat-
plotlib [25].
References
[1] Martín Abadi, Paul Barham, Jianmin Chen, Chen, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaogiang Zhang. Tensorflow: A system for large-scale machine learning. In OSDI,
2016. 10
[2] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong. Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski. Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. arXiv, 2022.
[3] Allen Institute for Al. Rearrangement Challenge 2022.
allenai. wwallnniiorr//eeeee.e....o. org/ iithr_
rearrangement_ 1phase__00000 __222.
[4] Allen Institute for AL. RoboTHOR ObjectNav Challenge.
https://githin comm man/all allenai/
robothor- chwllenge. 2
[5] Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
10
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv, 2016. 10
[7] Tom B. Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan. Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu. Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[8] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Frin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving.
In CVPR, 2020. 3
[9] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, 7imo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information- Rich 3D Model Repository. arXiv, 2015. 3
[10] Angel X. Chang, Manolis Savva, and Christopher D. Manning. Learning spatial knowledge for text to 3d seene generation. In EMNLP, 2014.
[11] Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021, 3
[12] Aditya Chattopadhyay, Xi Zhang. David Paul Wipf. Rene Vidal, and Himanshu Arora. Structured graph variational autoencoders for indoor furniture layout generation. arXiv preprint N.KKv::2244048666 2022. 3
[13] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi. Eric Kolve, Roozbeh Mottaghi. Jordi Salvador, Dustin Sehwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. Robothor: An open simulation-to-real embodied ai platform. In CVPR, 2020. 2, 7,8
[14] Matt Deitke, Anirddhaa Kembhavi, and Luca Weihs. PRIOR: A Python Package for Seamless Data Distribution AT Workflows, 2022.
[15] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. CVPR, 2009.
[16] Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. ManipulaTHOR: A Framework for Visual Object Manipulation. In CVPR, 2021.
2, 8,
[17] Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example-based synthesis of 3d object arrangements. ACM Transactions on Graphics (TOG), 31,6):111,,200 2012.
[18] Huan Fu, Bowen Cai, Lin Gao, Ling-Xie Xiao Zhang. Jiaming Wang, Cao Li, Qixun Zeng. Chengyue Sun, Rongfei Jia. Binqiang Zhao, et al. fronto 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10933- 10942, 2021. 3
10

---

[19] Chuang Gan, Jeremy Schwartz, Seth Alter. Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius. Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Damian Mrowea, Michael Lingelbach. Aidan Curtis, Kevin T. Feigelis, Daniel Bear, Dan Gutfreund, David Cox, James J. DiCarlo, Josh H. McDermott, Joshua B. Tenenbaum, and Daniel L. K. Yamins. Threedworld: A platform for interactive mudti- physical simulation. In NewrIPS (dataset track), 2021. 1, 2,
[20] Tobias Germer and Martin Schwarz. Procedural arrangement of furniture for real-time walkthroughs. In Computer Graphics Forum, volume 28, pages 2068-2078. Wiley Online Library, 2009.
[21]
Sean Gillies et al. Shapely: manipulation and analysis of geometric objects. 2007. 10
[22] Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab, 2008. 10
[23] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor. Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer. Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Kevin Sheppard, Tyler Reddy, Warren Weekesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant, Array programming with numpy. Nature, 2020. 10 [24] Ruizhen Hu, Zeyu Huang, Yuhan Tang, Oliver Matias van Kaick, Hao Zhang, and Hui Huang. Graph2plan:
Learning floorplan generation from layout graphs. ACM Trans. on Graphics, 2020. 3
[25] John D Hunter. Matplodib: 2d graphies environment. Computing science & engineering, 2007. 10
[26] Alexey Ignatiev, Antonio Morgado, and Joao Marques-Silva. PySAT: A Python toolkit for prototyping with SAT oracles. In SAT, pages 428-437, 2018. 10
[27] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning
benchmark & learning environment. IEEE Robotics and Automation Letters, 2020. 3
[28] Apoory Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective:
Clip embeddings for embodied ai. In CVPR, 2021. 8,
[29] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv,
2017. 2, 7, 8, 10
[30]
Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.
The open images dataset v4. IJCV, 2020. 3
[31] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishck Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario vSavsko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao. Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid. Sylvain Gugger, Clement Delangue, Th' eo Matussiere, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, Franccois Lagunas, Alexander M. Rush, and Thomas Wolf. Datasets: A community library for natural language processing. arXiv, 2021. 10
[32] Chengshu Li, Fei Xia, Roberto Mart' in Por-arrtnn in, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. CoRL, 2021. 3
[33] Zhengqin Li, Ting Yu, Shen Sang, Sarah Wang, Mengeheng Song. Yuhan Liu, Yu- YinY Ych, Rui Zhu, Nitesh B. Gundavarapu, Jia Shi, Sai Bi, Hong-Xing Yu, Zexiang Xu, Kalyan Sunkavalli, Milos Hasan, Ravi Ramamoorthi, and Manmohan Chandraker. Openrooms: An open framework for photorealistic indoor scene datasets. In CVPR, 2021. 3
[34] Lid LIFULL Co. Lifull home's 's dataset, 2015. Informatics Research Data Repository, National Institute of Informatics. 3
[35] Ricardo Lopes, Tim Tutenel, Ruben M Smelik, Klaas Jan Kraker, and Rafacl Bidarra. A constrained
growth method for procedural floor plan generation. In Game-ON, 2010. 3, 4
[36] Fernando Marson and Soraia Raupp Musse. Automatic real-time generation of floor plans based on squarified treemaps algorithm. International Journal of Computer Games Technology, 2010. 3
[37] Leland McInnes, John Healy, Nathanicl Saul, and Lukas Grossherger. Umap: Uniform manifold approxi- mation and projection. The Journal of Open Source Software, 2018. 10
[38] Wes McKinney et al. pandas: foundational python library for data analysis and statistics. Python for high performance and scientific computing, 2011. 10
[39] Meta AL. Habitat ObjectNav Challenge 2022. https: /tits://aiiabitat... orgg chts//aaalle//e//cceeeaaeaa 2022. 2.9 9
[40] Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang. Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations. In NeurlPS (dataset track), 2021. 3
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. 10
11

---

[42]
Aleksei Petrenko, Erik Wijmans, Brennan Shacklett, and Vladlen Koltun. Megaverse: Simulating embodied agents at one million experiences per second. In ICML, 2021. 3
[43] Lerrel Pinto and Abhinav Gupta. Supersizing sell-supervision: Learning to grasp from 50k tries and 700
robot hours. In ICRA, 2016. 3
[44] Xavier Puig, Kevin
K Kyunghwan Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating houschold activities via programs. In CVPR, 2018. 3
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In /CML, 2021.
[46] Santhosh K. Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John Tumnrr, Eric Undersander. Wojciech Galuba, Audrew Westbury, Angel Xuan Chang. Manolis Savva, Yili Zhao, and Dhruv Batra. -latepotmatterrortt 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv, 2021. 1, 3, 6, 7,
[47] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and lya Sutskever. Zero-shot text-to-image generation. ICML, 2021. 1
[48] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub. Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In ICCV, 2019. 1, 3, 10
[49] Bokui Shen, Fci Xia, Chengshu Li, Roberto Mart Mi-Mirt'in, in, Linxi (Jim) Fan, Guanzhi Wang, S. Buch, Claudia. Pérez D' Arpino, Sanjana Srivastava, Lyne P. Tehapmi, Micael Edmond Tchapmi, Kent Vainio, Li Fei-Fei, and Silvio Savarese. igibson, a simulation environment for interactive tasks in large realistic scenes. IROS, 2021. 1, 2, 3,7
[50] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, al. Scalability in perception for autonomous driving:
Waymo open dataset. In CVPR, 2020.
[51] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel Xuan Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In NewrlPS, 2021. 3,7
[52] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl S. Ni, Douglas N. Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Comm. of the ACM, 2016. [53] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski. Pearu Peterson, Warren Weckesser, Jonathan Bright, al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods. 2020. 10
[54] Michael L Waskom. Seaborn: statistical data visualization. Journal of Open Source Software, 2021. 10 Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In CVPR, 2021. 8
[55]
[56] Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng. Roozbeh Mottaghi, and Aniruddha Kembhavi. AllenAct: A framework for embodied AT research. arXiv, 2020. 8, 10
[57] Wenming Wu, Xiao-Ming Fu, Rui Tang, Yuhan Wang, Yu-Hao Qi, and Ligang Liu. Data-driven interior plan generation for residential buildings. ACM Trans. on Graphics, 2019.
[58] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv, 2018. 2
[59] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env:
Real-world perception for embodied agents. In CVPR, 2018. 7
[60] Fanbo Xiang. Yuzhe Qin, Kaichun Mo. Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang. Leonidas Guibas, and Hao Su. SAPIEN: A SimulAted bast- Interactive ENvironment. [61] Yu Xiang. Wonhui Kim, Wei Chen, Jingwei Ji, Christopher Bongsoo Choy, Hao Su, Roozbeh Mottaghi, Leonidas J. Guibas, and Silvio Savarese. Objectnet3d: A large scale database for 3d object recognition. In ECCV, 2016. 3
CVPR, 2020. 1, 2,
[62] Kun Xu, Kang Chen, Hongbo Fu, Wei-Lun Sun, and Shi-Min Hu. Sketch2sceme: Sketch-hased co-retrieval and co-placement of 3d models. ACM Transactions on Graphics (TOG), 32((4) 2013. 3
[63]
Omry Yadan. Hydra- - a framework for elegantly configuring complex applications. Github, 2019. 10
[64] Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyayy Taranjeet Singh, Akash Jain, Shiv Baran Singh, Stefan Lee, and Dhruv Batra. Evalai: Towards better evaluation systems for ai agents.
arXiv, 2019. 10
[65] Lap Fai Yu. Sai Kit Yeung, Chi Keung Tang, Demetri Terzopoulos. Tony F Chan, and Stanley J Osher. Make it home: automatic optimization of furniture arrangement. ACM Transactions Graphics (TOG)- Proceedings of ACM SIGGRAPH 2011, v. 30,(4), July 2011, article no. 86, 30(4), 2011. 3
[66] Shao- Kui Zhang, Wei- Yu Xie, and Song-Hai Hai Zhang. Geometry-base layout generation with hyper- relations among objects. Graphical Models, 116:101104, 2021. 3
12

---

[67] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin robosuite: A modular simulation framework and benchmark for robot learning. arXiv, 2020. 3
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please refer the Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please refer to the Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] We have provided code in the supplementary material and provided anonymous links in the rebuttal.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer to the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We observed training to be robust to different random seeds. See the Robustness section of the appendix for details.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Please refer the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Please refer to the Appendix.
(c) Did you include any new assets either in the supplemental material as URL? [Yes]
The environment including all the assets is included.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "NihVzPgZSSY", "reviewer": "Reviewer_QxMz", "summary": "The work introduces ProcTHOR, a procedural indoor environment generator. Each environment generated by ProcTHOR is a plausible environment with a physics engine, which can be used for training agents to solve various indoor tasks. The authors also show that pretraining on a sample of 10k homes from this generator, either in a zero-shot setting or after fine-tuning, the trained agent outperforms top contributions on several leaderboards.", "questions": "- **Q.1)** I'm not sure the paper sufficiently explains what you mean by \"fully interactive\". I understand that objects can be modeled as rigid bodies, but do your robot agents actually do friction-based grasping, or are objects tethered to the gripper when the \"grasp\" action is executed with the gripper reasonably close to the object? And for shelves/fridges/things that can be opened, do they only have a binary open/closed state, or can a robot open them by an inch?\n- **Q.2)** If all objects are rigid bodies, how do you assign mass, friction, and elasticity? Are these also procedural or can they be changed?\n- **Q.3)** What percentage of objects have these states (open/closed, etc)?\n- **Q.4)** In the appendix, you mentioned how many frames of training are required but how does that translate to wall clock time, i.e. how long did you train a policy for on ProcTHOR-10k until convergence?\n- **Q.5)** It's a bit crazy to me to say that you couldn't run additional experiments to verify performance and you took the first result because they're so expensive. In RL, it's a well-known phenomenon that different seeds can lead to vastly different performance or even different implementations of the same algorithm. Could you please elaborate, or if you had time since submission, report the mean/std of your results and how many seeds were used?\n- **Q.6)** Did you mention the 16 different scene specifications somewhere? Why these?\n- **Q.7)** Same for the 18 different Semantic Asset groups.\n- **Q.8)** You mention customizability and \"a few simple lines of specifications\" but what does that actually look like?\n- **Q.9)** Thanks for providing the performance table in Tab.1. That's always been my main gripe with AI2THOR. But how does the process distribution work? I.e. how do you run 15 processes on each of the 8 GPUs? ", "limitations": "~~Limitations aren't mentioned in the main body of the paper. I thought that was supposed to be included last year or 2 years ago.~~\n\nIn addition to the limitations listed in the appendix, I'd add that the lighting model is still relatively simple and might not correspond to lighting conditions in real houses but this effect may be \"domain-randomized\" away. Also, robot navigation is vastly simplified; real-world environments might have different friction surfaces, carpets with bumps, stairs connecting different rooms on the same floor, etc.\nAlso, real environments might have more decoration and trinkets lying around, as well as soft objects that can appear in many different configurations (a hoody thrown over a chair or splayed out on the sofa)\n\nEDIT: apparently limitations can go into the appendix and the authors did just that. So this is fine.", "rating": 9, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "TL;DR my main concerns are (a) reproducibility/code release and (b) why this wasn't submitted to the datasets/benchmarks track.\n\n### Strengths:\n- **S.1)** The paper is really well written and structured. It was a blast to read. All the illustrations and plots are meaningful and illustrate the work.\n- **S.2)** The contribution of ProcTHOR itself is great and really needed by the community. \n- **S.3)** The RL results look great. The fact that with a bit of fine-tuning (and sometimes without), this outperforms many baselines is impressive.\n\n### Weaknesses:\n- **W.1)** This really should've been submitted to the dataset & benchmarks track and I don't understand why the authors didn't do that. Because now I have to evaluate this for its methodological contribution and there isn't any. To my best knowledge, the ProcTHOR generator was pieced together with various artist assets, heuristics, and a lot of human annotation. That isn't a method that translates to any other domain and the authors also make no claim that their ProcTHOR can easily be adapted to car assembly factories for example. The experiments that are shown at the end of the paper, that I positively highlighted in (S.3), aren't methodologically new. The authors even go so far as to say they deliberately used a simple agent network to highlight the benefits of the dataset. Ultimately, the decision is not on me if this fits in into the main track of the conference, so I'll refer this to the AC.\n- **W.2)** Part of reviewing a paper for me is checking if the contributions claimed by the authors live up to the methods and results explained in the paper. Sadly, ArchitecTHOR (great name btw) doesn't do that. It's not even mentioned in the main body of the paper. Why is this listed as contribution if the main paper doesn't even discuss it? Sure, there are some pictures in the appendix but the main paper has to stand on its own - the appendix is for clarification and additional detail. (a) Why did you create ArchitecTHOR? (b) What did the designers focus on in designing these spaces? (c) What wasn't there yet in AI2THOR that needed to be added here? (d) What are the statistics of the spaces in terms of floor size, rooms, number of objects? (e) How does that compare to envs generated by ProcTHOR? (f) When should I use one over the other for training or is A-THOR only for evaluation? If these questions are answered in the main body of the paper, I'm happy to recognize ArchitecTHOR as dataset contribution and increase my score.\n- **W.3)** The authors make the vague promise that at some point, code will be released publicly. This is a dataset paper and I have no way to reproduce the hundreds if not thousands of person-hours that went into this. If this is not reproducible and not publicly available, it's of no use to the scientific community. If this was a paper introducing a new method, this would be different because, from the mathematical description of the method, I'd be able to at least reproduce some of it. Since this is not the case, I have to make a deduction in my rating. I have to insist that you release the dataset (generator) if you want us to accept a dataset paper.\n- **W.4)** In (S.1) I praised the writing and structure and I stand by that, but this was made at the sacrifice of some essential details. Sadly most of the information on HOW ProcTHOR was made was relegated to the appendix. I understand that this is due to space constraints but it kind of diminishes the value of the main paper with respect to the appendix. \n- **W.5)** Validity of claims: In the related works section, you mention HM3D, OpenRooms, and Megaverse and your main criticism of these is that they're either too small or too game-like but you don't verify that claim as far as I can tell. A way to verify this would be to use these environments for pretraining, instead of ProcTHOR-10k, and check the 0-shot and fine-tuning performance, at least in object-nav tasks (because yes, some benchmarks require interaction, and that's not offered in these but that's beside the point). This way, you can verify that (a) the size/diversity of ProcTHOR-10k is necessary and (b) the visual fidelity of ProcTHOR is necessary for this demonstrated transfer performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N050GQ-RfCu", "reviewer": "Reviewer_L9s9", "summary": "This work presents ProcTHOR, a procedural generation schema to generate synthetic house scenes subject to user-specified constraints. ProcTHOR has been applied to generate a set of 10,000 diverse house environment (fully equipped with sampled objects, materials, and other physics/rendering attributes). Training on these procedurally generated environments achieves state-of-the-art results over a range of embodied-AI tasks that solely rely on RGB images.", "questions": "I would like to see **W1**, **W2**, and **W3** discussed in the rebuttal phase", "limitations": "I find the discussion satisfactory", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=========\n\n**S1** Embodied-AI is a very active area of research in the robot learning and computer vision communities. The tasks addressed in this paper are directly based on a number of benchmarks proposed over the last 5 years. I thus expect this work to be relevant to several members at Neurips.\n\n**S2** This paper is extremely well-written. It is easy to follow the core contributions of this work, and the range of experiments (both in terms of environments and tasks) is somewhat broad. I also skimmed through some of the technical details about the dataset and I appreciated all of the thought and work that went into preparing the manuscript and supplemntary materials. Overall, this is thoroughly well-executed work.\n\n---\n\nWeaknesses\n==========\n\nWhile the work itself is well-executed, I have concerns over the technical aspects of this paper, which need further discussion.\n\n**W1** *Core contribution a dataset?* The paper, in its current form, comes off as a dataset contribution: the core takeaway for me was that it is possible to procedurally generate as many scenes as desired; and that the combination of the procedural generation scheme and the scale it achieves results in state-of-the-art performance over a number of embodied-AI tasks. While this is a demonstration that scale (and to an extent domain randomization) results in good performance, the key contribution enabling this is the ProcTHOR generation technique, which is more of a dataset (generator). I feel that the \"datasets and benchmarks\" track at Neurips would have been a much more suitable fit for this paper. I would like to see this aspect discussed further in the author-response phase. Particularly because, the neurips guidelines (here)[https://neurips.cc/Conferences/2022/CallForDatasetsBenchmarks] and (here)[https://neurips.cc/Conferences/2022/ReviewerGuidelines] focus on \"algorithmic advances, analysis, and applications\"; and I have a hard time reconciling with either of these categories in terms of technical contributions of this submission.\n\n**W2** *Focus on RGB-only agents*: The current submissions only evaluate RGB-only agents across environments. While this is arguably a harder task than the scenario where RGB-D data is available, several researchers interested in using the ProcTHOR environment are also likely interested in leveraging RGB-D sensors. It is therefore important to analyze performance on various other sensor configurations, and evaluate the gains ascribed by ProcTHOR when varying sensing modalities.\n\n**W3** *Procedural generation*: Building on **W1**, if the procedural generation scheme itself were to be considered a core contribution, it would have helped to have evaluation with respect to other competing procedural generation schemes (such as those cited in Sec. 2), or with respect to the various design choices within the ProcTHOR environment itself.\n\n---\n\n(Another aspect that could be relevant for future exploration is to optimize aspects of the procedural generator itself, to maximize performance on a set of downstream tasks. A somewhat recent example of this is observed in \"Meta-Sim: Learning to Generate Synthetic Datasets\").", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0nsuTMch1lE", "reviewer": "Reviewer_AqAB", "summary": "The paper presents ProcThor, a framework to sample virtual and interactive 3D environments from an underlying distribution of room and object layouts. In the current work, 10000 3D environments of varying sizes, # rooms, and object distributions are sampled and enable simulation for object search and manipulation tasks. The experiments demonstrate that training policies on the synthetically generated environments and the finetuning it on other datasets like RoboThor, AI2Thor, and Habitat-Matterport3D lead to state-of-the-art performances. Furthermore, the ablation experiments reveal that the transfer task performance continues to improve as more and more virtual 3D environments are sampled for training. ", "questions": "- Is the scene generation process novel? Could the authors do a detailed comparison of different steps to existing literature? This is essential for understanding ProcTHOR and improving it in the future.\n- How are room specs obtained (and are they realistic)? Does having only 16 specs limit the diversity?\n- How do rendering speeds compare to other frameworks like AI2Thor, iGibson, Gibson, Habitat, Habitat-2.0, etc?\n- What is being transferred when the visual appearance is significantly different (like HM3D-Semantic ObjectNav)?\n- Do the scaling ablations hold true when models are finetuned? Does the lack of consistent scaling for HM3D-Semantic ObjectNav reflect poorly on the ability to use ProcThor to benefit real-world robotics?\n", "limitations": "The authors have adequately addressed this.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# ----------------------- Post-rebuttal update --------------------\nThe authors' responses during the rebuttal addressed majority of my concerns and I'm retaining my rating at 8. I request that the authors update the paper with the reviewer feedback where appropriate. \n\nI'd also like to see the HM3D-objectnav scaling experiments under fine-tuning settings in the final version of the paper (or even in supp). I feel like that would partially answer the question of whether synthetically generating training environments can enable meaningful performance-scaling in real-world like scenes (with very different visual characteristics).  \n\n\n# ----------------------- Pre-rebuttal review --------------------\n# Strengths\n- The paper is well-written and easy to understand. The supplementary gives detailed information about generating scenes and helps clarify any ambiguity.\n- The framework developed is very extensive and is a massive feat of engineering (as detailed in supplementary). The authors have shared the code for their experiments and promised to open source it. I expect this to be very impactful for the embodied AI literature and beyond. \n- L208 - The efficiency of generating scenes is also impressive. 10k scenes were generated in 1 hour with 4 GPUs. The scenes are also efficient to store in a small JSON file per scene.\n- The experiments are well designed to cover 3 tasks from multiple online leaderboards. Both the zero-shot and finetuning experiments are impressive, especially considering that the agent only relies on RGB sensing (and not depth or panoramic sensing like some prior work). \n\n# Weaknesses\n- The novelty of the scene generation process is a bit unclear. My impression is that regardless of the novelty of each step of the pipeline, this framework on its own is a massive feat of engineering. Nevertheless, it would be useful to do a careful related works comparison to different steps of the pipeline. For example, the Semantic Asset Groups are similar to hyper-relations from [1].\n- In supplementary Sec. B2, the authors state the use of only 16 room specs for generating the 10000 scenes. This raises a question of diversity for generation. It's also not clear how room specs were obtained in the first place. \n- Table 1 - the rendering speeds are good, but there is no comparison to existing simulation platforms like iGibson, Gibson, Habitat, Habitat-2.0, etc. I feel it is important to benchmark these under a consistent hardware setup. It is okay even if ProcThor is slower than any of these individual frameworks since none of them support procedural generation of interactive scenes (and on such a large scale). \n- Table 2 - in the transfer results to HM3D-Semantic ObjectNav, the zero-shot performance is very low, but the fine-tuned performance is state-of-the-art. I'm not sure I understand what is being transferred here (visual representations? Navigation ability? Object-based scene priors? Ability to stop?)\n- Table 3 - the ablation study is good, but it leaves two questions unanswered. The results are all zero-shot, so it is not clear if the trends hold true when models are finetuned. Also, the HM3D zero-shot performance isn't consistently increasing. This raises the question of whether this large-scale procedural generation will actually help for real-world robotics.\n\n[1] Zhang, Shao-Kui, Wei-Yu Xie, and Song-Hai Zhang. \"Geometry-based layout generation with hyper-relations among objects.\" Graphical Models 116 (2021): 101104.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept