Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting
Authors: Yian Wang
Xiaowen Qiu
Jiageng Liu
Zhehuan Chen
Jiting Cai
Yufei Wang
Tsun-Hsuan Wang
Zhou Xian
Chuang Gan
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Scene Generation
Primary area: generative_models

Abstract: Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. However, generating diverse embodied environments with realistic detail and considerable complexity remains a significant challenge. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing $\textit{Architect}$, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. While there are still challenges that the camera parameters and scale of depth are still absent in the generated image, we address those problems by ''controlling'' the diffusion model by $\textit{hierarchical inpainting}$. Specifically, having access to ground-truth depth and camera parameters in simulation, we first render a photo-realistic image of only the background. Then, we inpaint the foreground in this image, passing the geometric cues to the inpainting model in the background, which informs the camera parameters.
This process effectively controls the camera parameters and depth scale for the generated image, facilitating the back-projection from 2D image to 3D point clouds. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate the placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments. Experimental results demonstrate that $\textit{Architect}$ outperforms existing methods in producing realistic and complex environments, making it highly suitable for Embodied AI and robotics applications.

ARCHITECT: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting
Yian Wang" Umass Amherst y rianwangdumass.co edu
Xiaowen Qiu' Umass Amherst xiaowengiuumaass.com edu
Jiageng Liu" Umass Amherst j jiagengliu@umass. edu
Zhehuan Chen
Jiting Cai
Yufei Wang
Umass Amherst zhehuanchen&umaas edu
Shanghai Jiao Tong University
Carnegie Mellon University yufeiw28andreww cmu. edut
caijitingesjtu edu. cn
Tsun-Hsuan Wang MIT tsunw@mit.eed
Zhou Xian
Chuang Gan
Carnegie Mellon University zhoux zhouxian@com imw.cmunncm......m edu
Umass Amherst
chuanggan&umass edu
Abstract
Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. However, generating diverse embodied environments with realistic detail and considerable complexity remains signifi- cant challenge. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, hindered by limitations such excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object config- uration than LLMs, we address these challenges by introducing ARCHITECT, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-be based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. While there are still challenges that the camera parameters and scale of depth are still absent in the generated image, we address those problems by "controlling" the diffusion model by hierarchical inpainting. Specifically, having access to ground-trute depth and camera parameters in simulation, we first render photo-realistic image of only back-grounds it. Then, we inpaint the foreground in this image, passing the geometric cues in the back-ground to the inpainting model, which informs the camera parameters. This process effectively controls the camera parameters and depth scale for the generated image, facilitating the back- projeccconn from 2D image 3D point clouds. Our pipeline is further ex- tended to hierarchical and iterative inpainting process to continuously generate scene. This iterative
placement of large furniture and small objects to enrich structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments. Experimental results demonstrate that ARCHITECT outperforms existing methods in producing realistic and complex environments, making highly suitable for Embodied AI and robotics applications.
Equaal Contribution
3 Project page: https: settps///iis.e www.ce cs. umass. edu/ ARCHITECT
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Bathroow
Bedroom
Moat
Corcals
Apartment
Grocery
bedroom
Study
Dining Living
Vegenables & Fruits
Connter
We present ARCHITECT, a generative framework to create diverse, realistic, and complex Embodied AI scenes. Leveraging 2D diffusion models, ARCHITECT generates scenarios in an open-vocabulary manner. Here, we showease two cases in detail: an apartment and grocery store.
Figure 1:
Collecting or generating large-scale training data has recently emerged as a promising direction for advancing Robotics and Embodied AI research. A major focus in recent works pursuing this direction advocates for data generation in simulated environments [Wang et al., 2023b,a, Ha et al., 2023, Dalal al., 2023], simulation offers cost-effective approach to data collection that scales naturally with computational resources; this thrust holds the potential for producing realistic physics and rendering data, and meanwhile grants access valuable ground-truth state information for speeding up policy learning. Among the types of data needed for training Embodied Al agents, diverse and realistic environments with the possibility of interacting with surrounding entities crucial. However, obtaining vivid interactive scene and environment data remains challenging. Recent studies have attempted to tackle this problem by developing generative models for environment creation via various approaches, including procedural generation with predefined rules [Deitke et al., 2022], diffusion-based scene generation [TTang et al., 2023a, Yang al., 2024b, Feng et al., 2024], and large language model (LLM) guided scene population and design [Wang et al., 2023b, Yang et al., 2024c, Wen et al., 2023].
Despite these recent efforts, generating diverse, realistic, and complex Embodied Al environments still remains a challenging problem due the inherent drawbacks and assumptions made in the pipeline designs of existing methods. For example, manually designed environment datasets [Ramakrishnan al., 2021, Weihs al., 2021, Li et al., 2023a, Fu et al., 2020a,b] require excessive human effort and are hence inherently difficult to scale. Procedural generation methods [Deitke et al., 2022, Khalifa al., 2020, Earle al., 2021, Zhao al., 2021] rely on predefined rules, which limited their ability to learn from and resemble real-world distributions, and struggle to generate open-vocas vocabbaayy scenes. Large language model (LLM) guided scene generation process [Wang al., 2023b, Yang et al., 2024c, Wen et al., 2023, Lin et al., 2023, Aguina-Kann et al., 2024, Feng et al., 2024] also presents its own limitations, as LLMs operate in language space and have limited 3D understanding and spatial reasoning capabilities. Moreover, existing LLM-based scene generation methods still rely on certain simplifications and hand-designed rules, such as primarily focusing on placements of large furniture pieces on the floor or against walls, and only considering simple inter-object relationships such as random placement of small items on top of large background furniture. Therefore, these methods struggle to generate more complex and cluttered object arrangements that are often encountered in

---

daily life, such as "an organized dining table", "an office desk drawer cluttered with objects", or "a shelf of toys". which typically require nuanced object placement and context-aware positioning.
As a result, there still exists a gap in current literature for generating interactive 3D scene with detailed and complex configurations that closely resemble real-world distributions. To this end, we propose ARCHITECT, a generative framework for creating realistic and interactable 3D scenes via diffusion-based 2D inpainting [Podell et al., 2023]. Our pipeline leverages controllable and hierarchical generation in 2D image space. Compared to LLMs which operate in language space, pre- reettrined image-based generative models are able to better capture scene and object configurations from massive image data readily available, both at the scene level and in fine- tne-grained inter-object spatial information. Pre-trained depth estimation models [Ke et al., 2024, Bhat al., 2023, Yang et al., 2024a] can then be used to lift the generated 2D static image to 3D environments. However, images created from 2D generative models do not provide accurate camera parameters, which are crucial for reconstructing accurate 3D environments. In addition, the predicted depth images also present scale ambiguity. To address these challenges, we propose to "control" the 2D generative models with 3D constraints via hierachical inpainting. First, we render a photo-realistic image in simulated empty scene with only a static background, where we have access to ground-truth depth and camera parameters. We then use this image template for inpainting the foreground using 2D diffusion models. During this process, the generation respects the camera parameters informed by the geometric cues in the background image and ensures that the inpainted components are both semantically and spatially consistent with existing components in the input image. By generating images this way, effectively control the camera parameters and depth scale for the generated image, which allows us to project it back to 3D point clouds. Subsequently, utilizing visual recognition models [Kirillov et al., 2023, Liu et al., 2023, Ren et al., 2024], we segment the 2D image obtain the semantics and geometric information of each generated object. These objects are then instantiated in the actual simulated environments, by either retrieving from large-scale asset databases [Deitke et al., 2023a, Mo et al., 2019] or generating using image-to-3D generative models [Xu et al., 2024].
While the pipeline described above is able to generate 3D scene configurations described from single camera view, our goal is to generate complete scenes observable from multiple views. In addition, we aim to generate scenes with real-worl, complexity, where objects of different scales together form a holistic environment (e.g. ideally we want to also generate small items placed on a shelf or in a drawer). Therefore, we further extend the pipeline to perform iterative and hierarchical inpainting, during which we continuously render new image patches of different locations of the scene to further enhance the complexity when needed. Specifically, given text description of target scene, we (i) generate the floor plan following previous works [Yang et al., 2024c, Wen et al., 2023], (ii) add background assets such as walls and floors into a simulated environment, (iii) render images of this empty scene and perform the aforementioned, proposed iterative inpainting process for scene-level generation from multiple camera views, (iv) hierarchically, apply inpainting again at a finer level to place small objects in various semantically plausible locations in the interior space, and (v) finally resulting in a complex 3D scene. Note that such iterative process results in a flexible generative pipeline that can handle different levels of inputs: text descriptions, floor plans, or even pre-arranged scenes.
Our pipeline is able to generate complex scenes that are fully interactable, with detailed asset placement and configurations at multiple scales, as shown in Figure 1. Note that since we make use of powerful prior knowledge encoded in 2D pre-trained generative models, we are able to generate open-vocabulary scenes in a zero-shot manner, for not only diverse room types in home settings, but also non-home environments such as grocery stores. Our experiments show that our framework outperforms prior scene creation approaches in generating interactable scenes that are more complex and realistic. We summarize our main contributions as follows:
We introduce ARCHITECT, zero-shot generative pipeline that creates diverse, complex, and realistic 3D interactive scenes to advance Embodied AI agents and Robotics research.
We propose to leverage 2D prior from vision generative models to facilitate the 3D interactive scene generation process, and make such process controllable by initializing from simulation-rendered image for hierarchical inpainting, ensuring consistent spatial features and controllable camera parameters and depth scale, allowing accurate 2D to 3D lifting.
The experimental results show that our method outperforms previous approaches in generating more complex and realistic interactive 3D scenes, both quantitatively and qualitatively.
3

---

Methods
No Train
No Human Effort
Interactive
Organized Small Objects
Open Vocab
Behavior- lk
ProcTHOR
Holodeck
AnyHome
RoboGen
PhyScene
DiffuScene
LayoutGPT
Text2Room
Ours
Table We compare our work with previos works that also aims to generate large scale 3D scenes in 5 aspect. Here, No Train means no need for training data of indoor layouts.
Our code will be made publicly available.
2 Related Works
Indoor Scene Generation A large body of works have focused on automatic indoor scene generation. Some works generate only the static mesh of the scene [Höllein et al., 2023, Schult et al., 2023]; in contrast, ours generates interactive scenes that can be used for downstream embodied Al and robotics tasks. One line research generates interactive indoor scenes via procedure generation with manually defined rules [Deitke et al., 2022]. The quality and diversity of the generated scenes highly depend on the predefined rules, which demands huge human efforts. Another group of works train generative model (e.g., transformers or diffusion models) on in-door scene datasets [Fu et al., 2021] and use the trained models for scene generation [Yang et al., 2024b, Feng et al., 2024, Tang al,,, 2023a, Paschalidou et al., 2021], and the quality and diversity of the scenes are bounded by the training dataset. Ours differ from these two lines of work as we do not use any manually defined rules nor pre-collected datasets, which might constrain the diversity of the generated scenes. Instead, we achieve higher diversity in the generated scenes by leveraging 2D image generative models that are trained with abundant internet data, which cover a wider distribution of scenes than those generated by manually defined rules or fixed dataset. Recently, several works employ a Large Language Model (LLM) for indoor scene generation [Wang al., 2023b, Yang et al., 2024c, Wen et al.. 2023], such as floor plan, layout, and object placements. Since the 3D spatial reasoning abilities of LLMs are still limited, the quality and diversity of the generated scenes are still bounded. By combining 2D image generative models, simulation rendering and controlled image impainting, our method achieves more coherent 3D layouts and higher scene diversity. We make a comparison between us and previous works in Table 2.
Text-fo-Image Image Diffusion models Text-to-image models based on diffusion model, such as DALL-E2 [Ramesh et al., 2021] I] and LDM [Rombach et al., 2021] ]. have become dominant in text-to-image generation. These text-to-image diffusion models have been trained on billions of images, giving them strong visual and 3D priors in addition to their image generation capabilities. Through SDS (Score Distillation) loss proposed by DreamFusion [Poole al., 2022], the 3D prior can be leveraged for a variety of downstream tasks like 3D object generation [Poole et al., 2022, Tang et al., 2023b, Wang et al., 2023c, Liang et al., 2023]. In our observations, in addition to visual and 3D priors, these text-to-image diffusion models also have strong priors for the layout of furniture and objects in room. Therefore, we designed a pipeline to exploit these layout priors from inpainting diffusion models to generate realistic indoor scenes.
3 Method
We automatically generate complex and realistic embodied environment scenes through a hierarchical and iterative process of rendering, inpainting, and visual reasoning. Specifically, as shown Figure 2, starting from an empty room, our pipeline first iteratively generates the layout of large furniture items (Figure 2 Large Furniture). Subsequently, we place smaller objects inside or upon these large furniture pieces, as depicted in Figure 2 Small Objects. In detail, our process entails the following

---

Input
Iteration
Empty Scene
Large Furniture
Enitializing
Inpainting
Visual Perception
Placing
Exsply Scare
Iqpaaring Mask
JLM
Ineaared Layoes
Deyti Estenation
Cless Chozsding
Placeract Lapoul
Cabs Vilbng
Mask
Tepeisted Leyout
Dapia Lorrattion
Obgent Crsanling
Environment
Small Objects
Output
Demonstration of our pipeline that generate complex interactive environment starting from empty scenes, including Initializing, Inpainting, Visual Perception and Placing modules.
Figure 2:
steps: (1) Initializing: We begin by selecting a view in the scene and rendering an image using Pyrender [Matl, 2019] and LuisaRender [Zheng al., 2022], and generating inpainting mask; (2) Inpainting: Given a text prompt generated by LLMs, along with the rendered image and masks from step 1, we inpaint the image according the text prompts, leveraging Latent-Diffusion [Rombach et al., 20210:2:: (3) Visual Perception: Upon recognizing the inpainted image, we generate 3D bounding boxes for objects using GPT4v, Grounded-SAM, and Marigold [OpenAI, 2023, Kirillov et al., 2023, Liu et al., 2023, Ren et al., 2024, Ke et al., 2024], as well as the rendering parameters from step I; (4) Placing: We place objects into the scene according to the 3D bounding boxes and return step to continue generating new objects in the next iteration.
3.1 Initializing Module
Overview In this module, initialize the viewpoint for the iteration, rendering image and obtaining the inpainting mask and ground-truth depth for the following steps.
Large Furniture Given an empty room, we heuristically choose the first view that spans from one corner to the opposite corner, maximizing the visible space of the room to render an image, as in Figure 2 Empty Scene. Setting light source in the middle top of the room, based method to render high-quality image and a raster-basse based method to obtain the ground -ruth depth and object segmentation. Next, generate inpaint mask for the image, which is centered within the frame, as shown in Figure Inpainting Mask. objects are already present in the scene, utilize the object segmentation mask obtained from the rasterizer to filter out those pixels from the impaint mask, ensuring that any furniture or objects placed within the room will not disappear from the newly inpainted images.
ray-tracing
Small Objects As illustrated Figure Small Objects, to place small objects on large furniture, we first heuristically choose front-top view front view depending on whether we are placing on top of a large object (e.g., table) or inside an object (c.g.. a shelf). To obtain the inpainting mask, we place cube within atop the bounding box of the larger furniture object, with size slightly smaller than that of the furniture object itself, as shown in Figure Cube Filling. cases placing small objects top of larger ones, we position the cube top of the larger object's bounding box, with fixed height and the other two dimensions slightly smaller than those of the larger object. Similar to large furniture, remove the pixels of existing small objects from the inpainting mask.
3.2 Hierarchical Inpainting Module
Overview this module, we utilize the image and inpainting mask provided from the previous step to first generate text prompts using LLMs. Subsequently, these prompts for image inpainting.
5

---

The inpainting process for both large furniture and small objects is depicted in Figure 2 Inpainting. To ensure smoother inpainting results, we apply techniques such as erosion and Gaussian blur the mask before commencing the inpainting process. This preparation allows for more effective filling of the contents within the mask. To enhance the diversity and ensure the generation of reasonable objects, we prompt LLMs to automatically generate suitable text prompts and negative prompts for the inpainting model. For example, when we input the configuration of a partially generated living room that includes TV set into an LLM, the LLM will place the word "TV" in the negative prompt, reasoning that a living room normally has only one TV set.
During this step, we generate multiple inpainted images. If the number of recognized objects in generated image falls below a predefined criterion, we filter out this image and generate new ones.
3.3 Visual Perception Module
Overview This module takes the inpainted image, ground-truth depth, and camera parameters to recognize and segment objects, estimate their depth, back-project them into 3D, and finally output 3D bounding boxes for each object.
For object recognition, we initially utilize GPT4v to detect all objects present in the image. The identified object names then serve as tags for Grounding- Dino, which performs object detection and provides the output bounding boxes. Following this, we use the SAM to obtain instance-level segmentation masks based on these bounding boxes, as depicted in Figure 2 Object Grounding.
After that, we estimate the relative depth of the generated image and rescale the predicted depth using reference depth. Specifically, giving reference pixel set P, { ja).ju)).. (fn 1, in fn-1)); }.
referenced depth map D, in HeW xf where W and // are the resolution of the image, estimated depth map De in R XH, we rescale the estimated depth map to Drescaled the following formulas:
min, = min (D(tede) max. 1E(U,n-1]
(D(
max, = max 1E(0,n-1|
max
1E[0,--1]
(D(i)))) scclle max, min,
min min nE|0,n-1]
max. mine
Drescated De
scale
scale Σ 1E
BHE [0
We employ different strategies when selecting reference pixels P,. For placing large furniture, we utilize all the non-masked parts of the image reference pixels to ensure general consistency with the room's 's floors and walls. For small objects, we focus on the non-masker areas of large furniture as reference pixels, aiming for consistency specifically with the inpainted object. This approach is adopted because the depth information outside the inpainted objects exhibits discontinuities that are challenging to predict accurately. For instance, shown in Figure 2 Depth Estimation, predicting the depth of the wall behind the shelf is difficult and could introduce noise if considered.
Once the depth estimation is acquired, use the camera parameters from the rendering process to pock-project the depth into a 3D point cloud. Utilizing the 2D masks provided by SAM, we extract the point cloud for each object instance. To eliminate any outliers, we apply DhaCAN]Khan et al., 2014] clustering to each segmented object, which allows us to derive axis-aligned bounding boxes for each object.
3.4 Placing Module
Overview Equipped with the 3D bounding boxes of objects, this module places the objects into the simulation and returns to the Initializing phase to commence the next iteration.
Large Furniture To incorporate large furniture into scene utilizing 3D bounding boxes, we initially retrieve each piece of furniture according to a text description generated by GPT4v. After extracting a list of instances from the datasets, select them based on feature similarity and the proportionality of their scale in three dimensions. The scale of each item is adjudicated by large language models using common sense knowledge.
Owing to the possibility that the retrieved furniture may not precisely conform to the 3D bounding boxes and minor errors in depth estimation, directly placing furniture at the center of the bounding

---

box can lead to issues such as collisions or complications arising from partial view observations. To mitigate these issues, we adopt an alternative approach by generating constraints derived from the 3D bounding boxes. We employ search methods akin to those used in Holodeck [Yang et al., 2024c] to determine optimal placement. Unlike Holodeck, which utilizes L.MSs to generate all constraints, we derive ours directly from the generated images and 3D bounding boxes. This search process enables us to avoid collision conflicts while simultaneously ensuring alignment with the generated image. Further details about these constraints can be found in Appendix A.
Small Objects To position small objects within a scene as defined by 3D bounding boxes, we first generate 3D instances based on the semantic content of each object and then place them at the specified positions within the 3D bounding boxes. Subsequently, we adjust the orientation and scale of the small objects to match the orientation and size of the bounding boxes. Notably, due to the partial nature of point clouds, it is impractical to uniformly apply the scale of the bounding box across all three dimensions. Instead, we focus on utilizing the scale along the dimensions perpendicular to the viewing direction. The placement process for small objects is depicted in Figure
4 Experiments
Dataset We retrieve objects from Objaverse. Deitke et al. [2023b,a] Objaverse dataset that contains massive annotated 3D objects. It includes various objects, including manually designed objects, everyday items, historical and antique items, etc. In the process of generating indoor scene objects, we retrieve suitable furniture from the Objaverse dataset and place them in the scene.
We also retrieve articulated objects from PartnetMobility [Xiang et al., 2020]. PartnetMobility contains 2346 3D articulated models from 46 categories, with articulation annotations.
Implementation We use Marigold [Ke al., 2024] as the depth estimation model. We use Grounded- Begment-Anything as our segementation model. We use the SD-XL [Podell et al., 2023] inpainting model provided by diffusers as the image inpainting diffusion model. We use LuisaRender [Zheng et al., 2022] as our renderer. For text-to-3D generation, we first use MVdream [Shi et al., 2023] to generate a image and then feed the image to InstantMesh [Xu et al., 2024] to generate the 3D asset. All experiments, including qualitative evaluation, quantitative evaluation and robotics task are all conducted on an A100 GPU.
Baseline We compare ARCHITECT with state-of-the-ard indoor scene generation approaches: (1) Holodeck [Yang et al., 2024c], leveraging common sense from LLMs generate floor plans and place objects: (2) Tex12Room [Höllein et al., 2023], utilizing the knowledge from image diffusion models and depth estimation models generate the entire mesh of a scene: (3) DiffuScene [Tang et al., 2023a], learning a diffusion model to generate the layout of 3D objects in a scene. In addition to the methods mentioned above, AnyHome [Wen et al., 2023] is another baseline we would like to compare with. However, the method is not yet fully open- open-sourced, so we will leave it for future work.
Metric To evaluate the semantic correctness of the generated scenes, we the following metrics that calculate the similarity between a rendered image and a given caption: (1) CLIPScore [Hessel et al., 2021], computing the correlation between image feature of the rendered image extracted by CLIP image encoder and text feature of the caption extracted by CLIP text encoder; (2) BLIPScore, using the image-text-matching head of BLIPv2 [Li et al., 2023b] to compute alignment between the rendered image and caption; (3) VQAScore [Lin et al., 2024], feeding the rendered image and caption into a VQA model, returning the probability that the answer to the question "Does the image show caption" is "Yes" as the score; (4) GPT4o Ranking, asking GPT4o to rank rendered images of generated scenes, and calculate the average ranking as the score. We also conduct a user study to evaluate various aspects of the generated scenes. We ask users to rate the following four indicators from to Visual Quality (VQ), Semantic Correctness (SC), Layout Correctness (I.C) and Overall Preference (OP).
4.1 Scene Generation
Qualitative Evaluation We compare rooms generated by our model with those generated by other methods. Figure 3 shows the results for both household and non-household scenes. The comparisons with Tom and ScffusScene are provided in Appendix B, as these methods generate either non-interactive scenes or lack diversity.
7

---

Hospital Room
Hair Salon
Video Store
Casino
Holoolock
Children Room
Shoe Store
Living Room
Dining Room
Vhauss Mormano titratt
Holodack
Figure 3: compare ARCHITECT with other methods in both household scenes(living room and dining room) and other non houschol scenes. We only compared the household scene generated by Diffuscene due to its limitations in Figure and compared with Text2Room in Figure
Compared Holodeck, our scenes are more realistic, leveraging the capabilities 2D diffusion models. For example, in the hair salon scenario, Holodeck fails to generate semantically correct scenes because the spatial constraints and objects are entirely generated by LLMs. Additionally, our work demonstrates the ability to generate more complex and detailed placements of small objects. For instance, the shoe store filled with paired shoes, toys on the children's room shelf, and the organized placement of items on the dining room table all exceed the generative abilities of Holodeck. This comparison highlights the effectiveness of our iterative and hierarchical inpainting process, showing that the use of 2D generative models indeed brings more spatial priors compared to LLMs.
Quantitative Evaluation We compare ARCHITECT to other state-of-hhe-ae indoor scene generation results using 2D image scores and user studies. The results are shown in Table 2 and Table 4. ARcHITEcT outperforms others in CLIP score, BLIP score, and GPT-4o ranking, while achieving relatively high VQA score (only slightly lower than Texi2Room). It demonstrate that our gen- erated scene are generally better aligned with the room caption (with better semantic and layout coherence). In GPT-4o's explanations of it's ranking, we found some common points of our previous analysis. Rooms generated by Text2Room are often criticized for having artifacts and distortion and result are often ranked lower, rooms generated by Holodeck often described simply

---

Text-Image Scores
User Study
Method
CLIPT
BLIPT
VQA*
GPT40L
VQT
SCT
LCT
OPT
Diffuscene
0.6785
0.4310
0.7561
3.76
3.52
3.37
3.50
Text2Room
0.6491
0.1223
0.8149
2.64
2.79
3.62
3.04
3.12
Holodeck
0.6502
0.3463
0.5696
1.91
3.34
3.14
3.11
3.07
Ours 0.7173
0.5859
0.8073
1.36
3.87
3.76
3.65
3.71
Table 2:
Quantitative Comparison. We evaluate 2D image metrics, including CLIP Score, BLIP Score, VQA Score and GPT4o ranking. We also conducted study, reporting visual quality(VQ), semantic correctness(SC), layout coherence(LC)) and overall preference(OP). GPT-4 ranking involves ranking and therefore does not include Diffuscene which only generate limited number of household scenes.
arranged; rooms generated by ARCHITECT are more favored by GPT-4o evaluator. In user study, ARcHITEct outscored other methods in all four aspects. worth noting that the visual quality score of Text2Room is significantly lower than other methods, which is likely due to the artifacts and distortions in Tex:2Room-genen -ennerttd scenes.
4.2 Embaadie/Roooote task
Task: move the glass from the dining table the hitchen table
Task: put the bottles the beverage shelf to the shopping cart
Figure 4: Two robot manipulation tasks generated in our scene setting.
Figure 5: Left: the robot organizes the room by pushing the chair under the table and pushing the keyboard inside the table. Right: the robot opens the fridge door, grasps the mango and puts it into the fridge, opens the kitchen-dining room door, grasps the soda can and puts it on the dining room table, and finally closes the fridge.
Inspired by previous work RoboGen [Wang al., 2023b], we making efforts to collect large-scale data for long-term embodied or robotics tasks in our generated scene.
A significant challenge we face is that the inclusion of all the detailed small objects in the simulation significantly slows down the speed of the embodied environment. To address this, after generating the scene, task and task decomposition, we use L.MSs to select relevant objects for each substep, while all other objects are designated as background objects and will not be physically simulated during this substep.

---

Given our house-level scene generation pipeline. we can now extend RoboGen to generate action trajectories for skills that require long-distance navigation and more complex tasks. Specifically, by inputting the floor plan, large furniture, and small objects into GPT-4, it first generates a task related to the existing objects in the scene. Then, as in RoboGen, it decomposes the task and filters out irrelevant objects to the background. Finally, we leverage action primitives and training supervision generated by LLMs to obtain trajectory actions to solve the task. We demonstrate two example generated tasks in Figure 4, and two other task with corresponding collected trajectories in Figure 5. The comparison of diversity of is shown in Table 3 by the self-BLEU score of task discription.
4.3 Object Generation
Fruit Bowl
Phone
Baby's Crib
DWID
DVD
Dragon Toy
Wooden Table
Figure 6: Examples of generated small objects and large furniture.
To address one of our limitations, the dependence on a large furniture database, we apply pipeline to generate high-quality large furniture. It optimizes differentiable tetrahedron mesh with SDS loss [Guo et al., 2024], using the normal-depth diffusion model and albedo diffusion model provided by RichDreamer [Qiu et al., 2024] as the main supervision signal. This pipeline is capable of generating high-quality object meshes from text guidance, specifically large furniture in our case. Some results are shown the right part Figure right part. We also show some qualitative resutls about small object generation in Figure 6 left part, which is another crucial factor of the quality of generated scenes.
Conclusion and Future Work
In this paper, we propose ARCHITECT, generative framework capable of creating diverse, realistic, and complex Embodied AI environments. Leveraging pre-trained 2D image inpainting diffusion models that better capture scene and object configurations compared LLMs, ARCHITECT iteratively extracts diverse and realistic layouts from image inpainting results. We also propose to control this inpainting process by processing geometric cues the background of rendered image. This process effectively controls the camera parameters and depth scale for the generated image, allowing us to project back into 3D point clouds. The scenes generated by ARCHITECT provide realistic and complex environments for downstream Embodied AI and robotics applications. In qualitative and quantitative comparisons, ARchITEct outperformed baseline methods both realism and diversity. We believe ARCHITECT is important step towards creating large-scale interactive 3D environments.
Limitation and Future work Currently, ARCHITECT retrieves furniture and large objects from datasets. This means that the diversity of furniture in our results is inherently limited by the dataset. In the future, we will explore generative methods to create more high-quality and articulated objects to further enhance the diversity of the generated scenes.
10

---

Acknowledgement
We thank the anonymous reviewers for their helpful suggestions. This work is funded in part by grants from Microsoft Accelarate Foundational Models Research Initiative, Cisco, and NSF IIS NS-224033.
References
Rio Aguina-Kang, Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Ganeshan, R Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, and Daniel Ritchie. Open-universe indoor scene generation using Ilm program synthesis and uncurated object databases. arXiv preprint arXiv:2403.00675, 2024.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023.
Murtaza Dalal, Ajay Mandlekar, Caclan Garrett, Ankur Handa, Ruslan Salakhutdinov, and Di- eter Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint 2.Xiv:2305./630, 2023.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982-555 5994, 2022.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.05666 2023a.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi. and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-13153, 2023b.
Sam Earle, Maria Edwards, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Learning controllable content generators. In 2021 IEEE Conference on Games (CoG), pages 1-9, IEEE, 2021.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Huan Fu, Bowen Cai, Lin Gao, Lingxiao Zhang, Cao Li, Qixun Zeng, Chengyue Sun, Yiyun Fei, Yu Zheng, Ying Li, Yi Liu, Peng Liu, Lin Ma, Le Weng, Xiaohang Hu, Xin Ma. Qian Qian, Rongfei Jia, Binqiang Zhao, and Hao Zhang. 3d-front: 3d furnished rooms with layouts and semantics. 2020a.
arXiv preprint 20X:::2011.091120
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. fd-future: 3d furniture shape with texture. arXiv preprint 2009022009.06633, 2020b.
Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10933-10942, 2021.
Minghao Guo, Bohan Wang, Kaiming He, and Wojeiech Matusik. Tetsphere splatting: Representing high-quality geometry with lagrangian volumetric meshes. arXiv preprint arXiv:2405.20288 2024.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference Robot Learning, pages 3766-3777. PMLR, 2023.

---

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. CoRR, 008 2021. URL https://arxiv org/abs/ 22/04/0//114 2104.08718.
Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Pruceedings of the IEEE/CVF International Conference Computer Vision, pages 7909-7920, 2023.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- beadesioo--baeed image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. Pegrl: Procedural content generation via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pages 95-101, 2020.
Kamran Khan, Saif Ur Rehman, Kamran Aziz, Simon Fong, and Sababady Sarasvady. Dbscan: Past, present and future. In The fifth international conference on the applications of digital information and web technologies (ICADIWT 2014), pages 232-238. IEEE, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. urXiv:2304.026443 2023.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- lk: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80-93. PMLR, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: pre-training with frozen image encoders and large language models, 2023b.
Bootstrapping language-image
Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching, 2023.
Yiqi Lin, Hao Wu, Ruichen Wang, Haonan Lu, Xiaodong Lin, Hui Xiong, and Lin Wang. Towards language-guided interactive 3d generation: Lims layout interpreter with generative feedback. arXiv preprint 2305 2023.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291, 2024.
Shilong Liu, Zhaoyang Zeng Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2003.05499 2023.
M. Matl. PyRender. https: //tt::////tttt cww htt.///u...oommmmmaap////enoeeee 2019.
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 909-918, 2019.
R OpenAl. Gpt-4v (ision) system card. Citekey: gpivision, 2023.
Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 34:12013-12026, 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
12

---

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion, 2022.
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9914,9225, 2024.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat- matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint 0050:2209900238, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πlya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102. 2021. URL https://arxiv.orgg 22002. 12092.
Tianhe Ren, Shilong Liu, Ailing Zeng. Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng. Hao Zhang, Feng Li, Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Jonas Schult, Sam Tsai, Lukas Höllein, Bichen Wu, Jialiang Wang, Chih-Yar Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, et al. Controlroom3d: Room generation using semantic proxy rooms. arXiv preprint 2rXiv:2312.05200 2023.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308. 16512, 2023.
Jiapeng Tang. Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Scene graph demoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303. 200.3033...2.... 14207, 2023a.
Jiaxiang Tang, Jiawei Ren, Hang Zhou. Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 2023b.
Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. In Arxiv, 2023a.
Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang. Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01455 2023b.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation, 2023c.
Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5922-5931, 2021.
Zehao Wen, Zichen Liu, Srinath Sridhar, and Rao Fu. Anyhome: Open-vocabulary generation of structured and textured 3d homes. arXiv preprint 2rXiv:2312.06644: 2023.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated bast- interactive environment. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint ar 074101240. 2024.
13

---

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401.10891, 2024a.
Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. arXiv preprint arXivv2404.09465, 2024b.
Yue Yang. Fan-Yur Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, al. Holodeck: Language guided generation of 3d embodied ai environments. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024), volume 30, pages 20-25. IBEE/CVE, 2024c.
Yizhou Zhao, Kaixiang Lin, Zhiwei Jia, Qiaozi Gao, Govind Thattai, Jesse Thomason, and Gaurav Sukhatme. Luminous: Indoor scene generation for embodied ai challenges. arXiv preprint arXiv:2111.05527. 2021.
Shaokun Zheng. Zhiqian Zhou, Xin Chen, Difei Yan, Chuyan Zhang, Yuefeng Geng, Yan Gu, and Kun Xu. Luisarender: high-performance rendering framework with layered and unified interfaces on stream architectures. ACM Transactions on Graphics (TOG), 41(6);1-19, 2022.
14

---

Appendix
A Implementation Details
A.1 Constraint and Search
Furniture can be devided into floor objects and wall objects. In detail, for the floor objects, we rely on the following type of constraints for furniture: Global, Location, Distance, Relation, Alignment and Rolation.
1M Global Constraint */
Edge: the edge of the room, close the wall. Middle: not close to the edge of the room.
Corner: the corner of the room. Horizontal/Vertical the global direction.
1* Distance Constraint */
Near, object: near to the other object. Far, object: far away from the other object.
1M Relation Constraint */
front of, object: front of another object. Behind, object: behind of another object object.
Left of, object: the left of another object.
Right of, object: to the right of another object.
1* Alignment Constraint */ Center aligned, object: aligned with another object
1 Soft Location Constraint */
Location, (x, y): predicted bounding-box location.
** Rotation Constraint */
Face to, object: face to the center of another object.
Different from Holodeck that is using LLMs to generate all the constraints, all above constraints generated by sorting floor objects by size and traversing them based the position of their bounding boxes except for the rotation constraints. Specifically, for each floor object's bounding box, constraints of each type boundaries and those of other floor object bounding boxes. In particular, rotation constraints cannot be solely determined by the bounding box, based on common sense (e.g., chairs facing a table). After obtaining the complete constraints, a DFS algorithm is utilized explore possible placements for each item. Placements that do meet the hard constraints are filtered out, and the highest scoring placements are selected based on the soft constraint scores. Here, hard constraints refer mandatory constraints, such global constraints and position constraints. Soft constraints refer to cumulative scores, where the highest scoring options prioritized, such location constraints. In practice, we applied greedy pruning mechanism the DFS algorithm, exploring only 3 nodes with highest score at each time. The score Sp for each placement calculated follows:
assigned based the distance and directional relationship between its
LLM is consulted to obtain the rotation constraint
R
Sp = Wtoc
Σ A.
Wi + Weur
A + C + W rotation
1.)
For each current object, W represents the weight of constraints; w represents the weight of each object; A represents the deviation from the reference, which hopes to be close to the reference and away from other items already placed; C represents constants to keep result positive; 1 is indicator function that equals if the rotation satisfies the constraint; represents the rotation of the item.
15

---

As for wall objects, most of its constraint comes from floor. Wall objects constraints are as follows:
1 Global Constraint */
A bove, object: close the wall, above specific floor object.
1* Soft Location Constraint */
Location, (x, y): predicted bounding- box location.
1 Position Constraint */ Height: The height of the object.
The placement of wall objects is relatively simpler because it does not require specifying orienta- tion; by default, they face away from the wall. Additionally, the likelihood of conflicts on the wall is lower, and using soft location constraint suffices for effective arrangement. And the searching process of wall objects is just almost the same as floor objects.
A.2 Large Furniture Retrieving
Following Holodeck, for each piece of large furniture, we first retrieve multiple candidates from the dataset using text descriptions of the assets. Then, we select one asset from the retrieved candidates based on scale similarity, which is calculated as the difference between the scale of 3D bounding box of object point cloud and the 3D bounding box of object mesh. Additionally, we integrated image similarity using the cosine similarity of CLIP features in the selection process in our latest pipeline. Here, scale similarity and image similarity are used only in the candidate selection process rather than the retrieval process, since there could be significant occlusions in the image (e.g., a chair behind a table) that could greatly influence the accuracy of retrieving.
A.3 Small Objects Generation and Selection
We use a text-to-3D pipeline (text-to-image and image-to-3D) to generate 3D assets for small objects. To make the scene more reasonable and resemble the inpainted image, we generate multiple candidates for each type of object and use the cosine similarity of DINO features to select from the candidates. We also experimented with an image-to-33 pipeline, starting from the object image segmented from the inpainted image. However, the resolution of the segmented image is low, resulting in sub-optimal 3D shapes and textures.
A.4 View Selection
For large furniture placement, heuristically select up to three views (right-back corner to left-front corner, front middle to back middle, and left-back corner to right-front corner) that can cover the whole room area for inpainting. Assuming the room ranges from (0, 0) to (x, y), the three views would be looking from (I, y, 1.8) to (0, O, 0.5), from (#, 0, 1.8) to (5 y, 0.5), and from (0, y, 1.8) to (ar, 0.5). We stop inpainting from new views when the occupancy of the room is larger than 0.7 or it has been inpainted from all three views.
Additionally, use 84-degree FOV for our camera during rendering, standard parameter for real-world cameras. Consequently, for a square room, this setup results in approximately 95 percent of the room being visible from single comer-to-cornet view.
For small object placement, we first ask LLMs to determine which objects can accommodate small objects on or in them, and then inpaint each of them with heuristic relative views. For objects like tables or desks on which we are placing items, we use top-down view. For shelves or cabinets in which we are placing objects, we use front view. The distance of the camera from the object is adjusted according the scale of the object and the camera's FOV, ensuring the full object is visible during inpainting.
16

---

B More Experiments
B.1 More Comparison Cases
As shown in Figure 7 and 8, compared to Text2Room, our method generates more photorealistic scenes. However, since Text2Room directly projects RGB pixels into 3D space using depth maps, artifacts are unavoidable, the geometry of the generated 3D mesh is distorted, and it can't It serve as an embodied environment since it's not interactive. These issues are addressed retrieval-based methods. Compared to Diffuscene in Figure 7, our method generates more reasonable and detailed scenes. Our results exhibit more detail and precision. Additionally, the dataset lacks information on the placement of small objects, limiting the diversity of small object placement in Diffuscene-gennt generated scenes.
Ours
Text2room
Diffuscene
Living Room
Dining Room
Figure 7: Comparison of living room and dining room scene generated by ARCHITECT, Text2room, Diffscene.
B.2 Comparison with Baselines
The comparison with PhyScene is shown Table where we present the comparison results for generated living rooms (only the weight of living room generation released).
In our experiments, aim to provide general comparison with three types of related works: works that generate only the static mesh, works that are trained on existing datasets and works that generate ooea-vocy scenes using foundation models.
It's challenging to make a completely fair comparison between our methods and the Texr2Room method since they serve different purposes. While Text2Room generates the entire mesh without retrieving objects, none of its assets are interactive and may achieve higher photorealism by directly generating meshes from 2D images.
B.3 Controllability and Editing
In short, our method combines diffusion-barn pipeline with an LLM--bssed method, which still possess the ability of controlling and editing. The impainting-to-layon pipeline functions
17

---

Children Reom
Casino
Hospital Room
Hair Salon
Goulle
Figure 8: Comparison of four scenes generated by ARCHITECT, Text2room.
can considered as an API function callable by the LLM. Our approach aims to generate scene configurations seeded from diffusion models, with scene editing othogonal feature enabled by LLMs. Specifically, the scene configuration generated by our pipeline can be represented by each object's name, position, scale, bounding box, orientation, and asset UID, which can be easily converted to text representations. This allows us to feed this information directly into LLMs to further control or edit the scene layout. Corresponding experiments could be found in Appendix C.
B.4 Comparison of Architect and LLMs in Small Object Placement
It's challenging for LLMs to directly solve arrangement problems. First, for small object placement on shelves, LLMs lack information about supporting surfaces, making it impossible for them to solve this issue. Second, for placement tables, while we might know the supporting surfaces given the bounding boxes, L.MSs struggle with object orientations, often resulting less complex scenes or scenes with severe collisions. We show comparison of small objects generated by methods and LLMs the middle part of Figure and Table 3.
B.5 Similarity of Generated Scenes and Inpainted Images
We've ve also evaluated the image similarity between inpainted images and images of generated scenes against empty scenes, shown in Table 3. The results indicate that, although not exactly the same, the generated scenes are to some degree faithful to the image generation results.
B.6 Consistency of Inpainting
The appearance the masked area is consistent with other areas both stylistically and geometrically. We also apply a commonly used technique, softening the boundary of inpainting masks, to improve consistency. A comparison the results before and after using softened inpainting masks is shown in the left part of Figure 9,
18

---

Large Furniture
Small Objects
Similarity (%)
Similarity (%)
Empty vs. Inpaint
45.33 85.04
77.
Inpaint vs. Placed
83.83
LLM Placeeent VQScore
74.93
Our Placement VQScore f
80.81
RoboGen Self-BLEU
0.284
Ours Self-BLEU
Table 3: Quantitative experimental Results.
Method
CLIP
BLIP f
VQScore f
PhyScene
71.42
46.51
88.72
Holodeck
69.37
53.23
84.06
Text2Room
64.91
12.22
90.73
Diffuscene
65.32
49.95
87.28
ARCHITECT (Ours)
72.96
63.62
94.58
Table 4: Experimental result of comparison with PhyScene.
C
Scene Editing
To demonstrate that our pipeline is compatible with editing and complex text control, implemented additional APIs to add, remove, and rescale objects, enabling LLMs to edit the scene.
Initial results for scene editing shown in the right part of Figure 9, We issued commands LLMs such as replace the books on the shelf with vases, replace the bookshelf with a cabinet, and make the bookshelf smaller. The LLMs achieved the correct results by calling the provided APIs.
Blur No Blur
ARCHITECT vs LLM
Text Control and Edit
Figure 9: Demonstration of comparison between different mask, different small objects placement and effect of text control. Red dashed box indicates inconsistency when using non-blur mask.
D User Study Details
We conducted comprehensive human evaluations assess the quality of ARCHITEct scenes, with a total of 115 undergraduate students and graduate students participating in the user studies. All participants were volunteers without compensation.
We first provided participants with two minutes read the instructions, the specific content which was follows:
19

---

Thank you for your participation! This questionnaire used for the experimental part scientific research articles, which requires human evaluation. We will keep the information of the participants confidential.
The estimated total time about 10 minutes.
There are some pictures in the questionnaire from different utils. Just observe and score all of these pictures. The higher the score, the better the quality. An example is shown in the following image:
1. Plause rele the lollowing scere image:
name:
children room
2
3
visual quality
0
100
0
0000
0
semantic ntic correctness
0
0
0000
000
layout coherence
0
10
overall preference
0
Figure 10: The example questionnaire for participants.
Then, we randomly assigned each volunteer 23 scenes and asked them score the scenes from one to five, considering visual quality, semantic correctness, layout coherence, and overall preference, Each volunteer received only 23 scenes ensure they could complete their responses in approximately 10 minutes.
We collected ratings from all participants and caleulated the average scores for these four metrics. These average scores were used to evaluate our model, Holodeck, Text2Room and DiffuScene allowing compare the scene generation performance of model with the two baseline models.
The average response time was 525 seconds, with the longest response time being 1113 seconds and the shortest being 150 seconds. Responses with duration of less than 230 seconds were filtered out.
20

---

E Prompts
E.1 Object Recognition Prompt
Detect all objects the picture, generate a description each object, and decide whether it is floor-object or wall-object.
Here are the definitions of object types:
floor object: object that is placed on floor or in direct contact with the floor.
wall object: object that is placed on wall and not in contact with the floor.
Here is sample answer:
table: A big yellow table floor-object
chair: gray armchair floor-object
tv: black wall-mounted television wall-object
Requirements:
1. Description should be too long.
2. You should only give the result and no unnecessary words.
3. Don't describe the positional relationship between objects.
4. Classification can only be "floor-object** *wal--objectt*
5. Please pay attention to only large furniture like sofa, table, lamp, shelf, and ignore small objects like bottles or books.
The prompt above is fed into GPT-4V along with an image generated by a 2D inpainting model. The prompt asks GPT-4V recognize all objects the inpainting image, briefly describe them, and then classify them as either objects on the floor or objects on the wall. The generated object names will be used prompt Grounded-SAN The generated descriptions will be used for retrieving objects for text-to-3D generation.
E.2 Inpainting Prompt and Generation Prompt
Given the objects in the current scene, please list which objects have already reached their potential limits, and the objects are still lacking.
Your answer should be in the following format: reached limit: object A, object B, lacking: object C, object D.
The objects in the current scene are: /* list of objects with quantities, eg: sofa, 1 coffee table, TV**
Remember, do answer anything not asked. The lacking objects should ideally contain objects that are not in the scene. The lacking objects you list should be precise, do not give things like "other furniture".
The prompt above asks GPT-4V provide negative prompts and positive prompts in addition the room caption for the inpainting model. ROOM-CAPTION will be substituted with the actual room caption. The lacking objects will be added the positive prompt, and the objects that have reached their limit will be added to the negative prompt.
F
Societal Impacts
F1
Positive Impacts
Advancements in Robotics and AI: ARCHITECT enhances the development of versatile robots capable of assisting in various tasks.
21

---

Educational Tools:
Generated 3D environments can be used for immersive learning experi- ences, aiding in the understanding of spatial relationships and complex systems.
Accessibility: Improved Al environments can lead the development of assistive technolo- gies for individuals with disabilities, enhancing their quality of life.
E.2 Negative Impacts
Job Displacement: Advanced Al and robotics could potentially displace jobs in certain sectors, necessitating consideration of economic and societal impacts.
Ensuring training data and algorithms are representative and fair is crucial to avoid perpetuating existing biases.
Bias and Fairness:
Misuse of Technology: Inferring internal geometric structures of 3D objects could be misused for unauthorized reproduction surveillance, leading ethical and legal concerns.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions, zero-shot generative pipeline that creates diverse, complex, and realistic 3D interactive scenes and the paper's scope are accurately reflected by main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discuss the limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification: The paper does not provide theoretical results, it provides practical 3D scene generation pipelines instead. Thus it does provide full set of assumptions and proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides clear and specific description of the pipeline that places large furniture and small objects. All the information needed to reproduce the main experi- mental results are provides, thus, the paper is easy be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
24

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code will be made publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (attps://nips... cc/
onoooicy))yy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cc/ wtc/public//ee httllii//gides///e/eee SCae//Coommmiiiccc iundbbiisse ieiinonooiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The ARCHITECT method that the paper proposed dose not need for training data of indoor layouts, which specified in main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide experiments compute resources Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https: http::////e/u..eeeeeeeeee wttpp/ewww.....uuu.eeeeeeee cctpublic/Ettiieeceeeececeo ctt.oiiieetiissuueellll...
didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the positive and negative social impacts thoroughly in Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses such risks, we are using all existing pre-trained models and datasets rather than releasing new ones.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets, used in the paper are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new ARCHITEcT method is well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper include the full text of instructions given to participants and screen- shots appendix. The participants are volunteers with no compensation.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: There potential risks for participants and the IRB approvals were obtained.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "hOdXvlq1nY", "reviewer": "Reviewer_YpUS", "summary": "The paper introduces ARCHITECT, a generative framework designed to create complex and realistic 3D environments for Robotics and Embodied AI research. Unlike traditional methods that rely on manual design, predefined rules, or large language models (LLMs), ARCHITECT utilizes pre-trained 2D image generative models to capture scene and object configurations more effectively. To address the problem of camera parameters and depth estimation, ARCHITECT utilizes a hierarchical inpainting pipeline.\nSpecifical contributions are: 1. The paper proposes to leverage 2D prior from vision generative models for the 3D interactive scene generation process to generate complex and cluttered object arrangements; 2. The paper introduces a controlled inpainting process that utilizes a controllable rendering process and integrates geometric cues from the background to guide the generation of the foreground, allowing for better control over camera parameters and depth perception.", "questions": "1. Can the inpainting model ensure that the appearance of the masked area is consistent with other areas?\n2. Could you provide some results about generating using image-to-3D generative models? I found that the paper mentioned this approach in the introduction section but did not provide subsequent experiments.\n3. How do you perform object retrieval? Is it like the HOLODECK method?\n4. Can the algorithm based on LLMs solve the problem of small object arrangement by modifying the example in the prompt?", "limitations": "The authors have addressed the limitations. However, the limitations of inpainting models still need to be further elucidated.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Compared to previous work that overly relies on the ability of LLMs, this work proposes the use of inpainting models to draw room layouts, making the generated furniture layouts more diverse. This can inspire future work to better utilize different pre-trained multimodal models to assist in generation, control, and other task.\n2. This work proposes a solution for the arrangement of small objects, which helps generate more realistic scenes and is more conducive to subsequent interaction planning.", "weaknesses": "1. For methods based on rules and LLMs, we can directly edit the positions, types, and other conditions of objects to control the output. However, when using a pre-trained inpainting model to plan object layouts, it becomes difficult to make such detailed edits, which seems to result in a loss of some controllability.\n2. Although the goal of the paper is to generate scenes using multiple viewpoints, the paper only provides the method of selecting the first camera viewpoints. I hope the authors can further clarify the specific rules for selecting multiple viewpoints.\n3. Heuristically choosing the view that spans from one corner to the opposite corner results in only half of the room being occupied by objects, which would make the room appear unrealistic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer YpUS \n\n*Thank you for your insightful and constructive comments! We have added additional experiments and modified our paper according to your comments.*\n\n**1. Controllability and Editing**\n\nIn short, our method combines a diffusion-based pipeline with an LLM-based method, which still possess the ability of controlling and editing. The inpainting-to-layout pipeline functions can be considered as an API function callable by the LLM. Our approach aims to generate scene configurations seeded from diffusion models, with scene editing as an othogonal feature enabled by LLMs.\n\nSpecifically, the scene configuration generated by our pipeline can be represented by each object's name, position, scale, bounding box, orientation, and asset uid, which can be easily converted to text representations. This allows us to feed this information directly into LLMs to further control or edit the scene layout.\nThe corresponding experiments could be found at General Response 2B.\n\n**2. View Selection**\n\nAs illustrated in General Response section 3C, we heuristically select up to three views to ensure that most areas of the room are inpainted and appropriately placed.\n\n**3. Only Half Room Inpainted With a Single View**\n\nAddressing this concern, we select up to three different views to cover the entire room area for inpainting. Additionally, we use an 84-degree FOV for our camera during rendering, a standard parameter for real-world cameras. Consequently, for a square room, this setup results in approximately 95 percent of the room being visible from a single corner-to-corner view.\n\n### Questions\n\n> **Q1: Can the inpainting model ensure that the appearance of the masked area is consistent with other areas?**\n\nYes, the appearance of the masked area is consistent with other areas both stylistically and geometrically. We also apply a commonly used technique, softening the boundary of inpainting masks, to improve consistency. A comparison of the results before and after using softened inpainting masks is shown in the left part of Rebuttal Figure 3.\n\n> **Q2: Could you provide some results about generating using image-to-3D generative models? I found that the paper mentioned this approach in the introduction section but did not provide subsequent experiments.**\n\nWe are using the previous work InstantMesh[1] to generate 3D assets. This is not our main contribution and is not specifically highlighted in our paper. Some of the results can be seen in the images within our paper, where all the small objects are generated using InstantMesh. Additionally, we present example generated 3D assets in Rebuttal Figure 2, which also includes new results from our large object generation pipeline.\n\n> **Q3: How do you perform object retrieval? Is it like the HOLODECK method?**\n\nAs illustrated in General Response 3A and 3B, our method is basically the same as the Holodeck[2] retrieving method.\n\n> **Q4: Can the algorithm based on LLMs solve the problem of small object arrangement by modifying the example in the prompt?**\n\nIt's challenging for LLMs to directly solve arrangement problems. First, for small object placement on shelves, LLMs lack information about supporting surfaces, making it impossible for them to solve this issue. Second, for placement on tables, while we might know the supporting surfaces given the bounding boxes, LLMs struggle with object orientations, often resulting in less complex scenes or scenes with severe collisions.\nWe show a comparison of small objects generated by our methods and LLMs in the middle part of Rebuttal Figure 3 and in Table 1.\n\n[1] Xu, Jiale, et al. \"Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models.\" arXiv preprint arXiv:2404.07191 (2024).\n\n[2] Yang, Yue, et al. \"Holodeck: Language guided generation of 3d embodied ai environments.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n*We wish that our response has addressed your concerns. If you have any more questions, please feel free to let us know during the rebuttal window.*\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1fO6OzNbEP", "reviewer": "Reviewer_YmnU", "summary": "This paper proposes a hierarchical diffusion-based 2D inpainting method for creating interactive 3D scenes. By leveraging the generative prior of 2D diffusion models, the proposed method could generate more realistic and diverse object layouts compared with Holodeck [Yang et al., 2024c] that depends on LLMs which lack 3D spatial reasoning ability. Specifically, it renders a 3D scene (initialized with empty scene) and utilizes its GT depth and camera parameters for the back-projection from 2D inpainted components to 3D point clouds. Based on this layout, it determines how to place 3D objects into the interactive 3D scene, where large objects are retrieved from databases and small objects are generated with text-to-3D and image-to-3D generative models.", "questions": "(i) The proposed method generates small objects using a text-to-3D generation method (MVdream) and an image-to-3D generation method (InstantMesh), but it retrieves large furniture pieces instead of such generation. Is this because the large furniture pieces determine how small objects are placed and thus should be neat? What is the reason for the retrieval?\n\n(ii) According to L194-195, the proposed method filters out images if the number of recognized objects falls below a pre-defined criterion. What is the criterion? Is there also an upper bound? \n\n(iii) According to L224-228, the proposed method finally selects large furniture pieces based on feature similarity and proportionality of their scale. How did the authors compute feature similarity and balance between the similarity and proportionality?", "limitations": "Yes, in page 9 and 18.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has the following strengths:\n\n(i) The proposed hierarchical diffusion-based 2D inpainting method is reasonable and seems to be effective for generating diverse and realistic object layouts. With the help of the prior encoded in 2D diffusion models, it shows better 3D spatial reasoning compared with Holodeck that heavily depends on LLMs. As shown in Figures 3 and 4, it could also synthesize and organize small objects, which is interesting.\n\n(ii) This paper comprehensively evaluates the proposed method in terms of CLIPScore, BLIPScore, VQAScore and GPT4o Ranking, and also conducts a user study. As shown in Table 2, it achieves state-of-the-art results. The quantitative results are supported by qualitative comparisons shown in Figures 5 and 6.\n\n(iii) This paper is well-organized and easy to understand. Most descriptions are clear.", "weaknesses": "This paper has the following weaknesses:\n\n(i) The diversity of inpainting results might be affected by the bias of LLMs, sinch inpainting prompts are generated by LLMs. According to L191-193, it says “TV” will be placed into the negative prompt since a living room normally has only one TV set. Such bias might enhance the realism but limit the diversity of inpainting.\n\n(ii) The proposed method essentially enables generating open-vocabulary scenes, but it currently retrieves large furniture pieces from databases. \n\n(iii) Several implementation details are not fully described. For example, it does not elaborate the predefined criterion used in the filtering (L194-195) and how to finally select large furniture (L224-228).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer YmnU\n*We appreciate the positive and constructive comments from you! We have modified our paper according to your comments.*\n\n**1. Bias from LLMs**\n\nThere is always a trade-off between diversity and realism. It is true that sometimes chaotic data can be valuable in enhancing an agent's capability to deal with such scenarios. We'll make this LLM bias optional in our future version. Thank you for mentioning this!\n\n**2. Retrieving from Dataset**\n\nThis is a limitation of our work, as mentioned in the Limitation and Future Work section of our paper. However, we can address this limitation by augmenting our method with a 3D generation pipeline for large furniture, as shown in General Response 2C.\n\n**3. Implementation Details**\n\nWe've provided the implementation details in General Response 3. \nFor the predefined criterion used in the filtering, we filter out the inpainting images with fewer than three new objects. If 30 images are filtered out, we choose the one with the most objects from them.\n\n### Questions\n\n> **Q1: The proposed method generates small objects using a text-to-3D generation method (MVdream) and an image-to-3D generation method (InstantMesh), but it retrieves large furniture pieces instead of such generation. Is this because the large furniture pieces determine how small objects are placed and thus should be neat? What is the reason for the retrieval?**\n\nWe retrieve large furniture instead of generating it mainly because the quality of retrieved large assets is generally better than that of generated large assets. Conversely, for small objects, the quality of generated objects is generally higher and more diverse than retrieved ones. To address this limitation, we have developed a pipeline for generating higher quality large furniture, as shown in General Response 2C.\n\nAnother reason for retrieving large furniture is our aim to make assets more interactive. We sometimes add articulated furniture to the scene, for which there is currently no suitable generation pipeline.\n\n> **Q2: According to L194-195, the proposed method filters out images if the number of recognized objects falls below a pre-defined criterion. What is the criterion? Is there also an upper bound?**\n\nThe lower bound is 3 and there's no upper bound.\n\n> **Q3: According to L224-228, the proposed method finally selects large furniture pieces based on feature similarity and proportionality of their scale. How did the authors compute feature similarity and balance between the similarity and proportionality?**\n\nThis is included in General Response point 3A. Essentially, we use feature similarity to retrieve multiple assets and then use proportionality to select from them.\n\n*We hope the additional explanations have convinced you of the merits of our work. Please do not hesitate to contact us if you have other concerns.*\n\n*We really appreciate your time! Thank you!*\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P21MevuGwZ", "reviewer": "Reviewer_yxqm", "summary": "This paper considers creating large 3d indoor scenes (e.g. an apartment or grocery store) by an hierarchical generation procedure. The key idea is to use diffusion models for inpainting to guide where to place the objects in the scene. This is done for both large objects (e.g. table or couch) and also small objects relative to the larger ones (e.g. books or a coffee cup on a coffee table). The objects are retried from both large scale object datasets (Objaverse) and image-to-3d models. The paper is compared to several existing methods for scene generation, and found to perform well, generating scenes that adhere to the text prompt. The scenes generated by the method could be useful e.g. for robotics and for training embodied agents, although such experiments are not presented.", "questions": "See weaknesses", "limitations": "This is adequately addressed", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a methodology to use images created by generative models (inpainting version of stable diffusion xl) to find relative object placements. This is in contrast to existing work often relying on heuristics, generative models for scene configurations, or object relations generated by text generated by LLMs. \n- Qualitatively, the generated scenes are detailed, with multiple objects that look reasonably placed and with fine-grained details. Also quantitatively, experiments show e.g. higher CLIP and BLIP similarity between the generated scenes and text prompts than several works that are compared to.\n- The paper presents an hierarchical approach to placing objects where it starts with an empty scene, and then large furniture is placed, and finally small objects are placed on the large objects, guided by the inpaintings.", "weaknesses": "- The 2d inpainting is used only partially. For the large objects, it is only used as a means of finding object placements in the form of 3d bounding boxes, and then the content within the bounding box (inpainted image, point cloud etc) is more or less discarded and an object is retrieved from Objaverse instead and placed in the 3d bounding box. This is not the case for small objects where the inpainted images are fed to an image-to-3d model, and it is unclear why this distinction between large and small objects was made.\n- Several of the closest related works, with two examples being Holodeck [Yang 2024c] and procTHOR [Deitke 2022], evaluate how agents for e.g. object navigation perform after training on their proposed datasets, and evaluate how it generalizes to other datasets. Such policy learning experiments are missing from the current paper.\n- Qualitatively, it clearly looks synthetic and not photo-realistic both for the scene layout with the floor and walls, and also for the large objects, which are retrieved from partially synthetic datasets. This is also the case for the closest related work (e.g. Holodeck), so it is not a major issue because of this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer yxqm\n*We appreciate the positive and insightful comments from you! We adress your concerns in details below.*\n\n**1. Partially Usage of 2D Inpainting**\n\nGenerally speaking, the occlusion when inpainting large furniture is much more severe than when inpainting small objects. Therefore, we treat them differently. But in our latest pipeline, we incorporate image similarity for object selection, as mentioned in General Response point 3A. We also want to clarify that the inpainted image is not fed into the image-to-3D pipeline, as mentioned in General Response point 3B.\n\nMore details are included in General Response points 3A and 3B.\n\n**2. Policy Learning Experiments**\n\nTo the best of our knowledge, only Holodeck[1] and procTHOR[2] demonstrate policy learning from their collected data, as they are based on the AI2-THOR platform, which provides comprehensive APIs and benchmarks for navigation and other semantic embodied tasks. In contrast, other related works [3, 4, 5, 6, 7, 8] do not focus on such tasks.\n\nOur work significantly advances small object placement, a task particularly relevant to robotic manipulation. Consequently, our research aligns more closely with previous works like RoboGen[9] and Gensim[10], which emphasize the acquisition of manipulation skills. Therefore, our experiments for embodied tasks are centered around manipulation skill acquisition rather than training agents on the proposed dataset.\n\nWe've illustrated more details of these experiments in General Response 2A.\n\n**3. Synthetic Style**\n\nWe believe that the reason some scenes still appear synthetic is mainly due to two factors: 1) the retrieved assets are not realistic enough, and 2) the rendering configuration (surface materials and lighting conditions) is not carefully adjusted.\n\nWe can address these issues by retrieving assets from artist-designed datasets and improving the rendering configuration. As shown in Rebuttal Figures 1 and 3, with a better rendering configuration, our rendering results have become more photorealistic than before.\n\n[1] Yang, Yue, et al. \"Holodeck: Language guided generation of 3d embodied ai environments.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Deitke, Matt, et al. \"🏘️ ProcTHOR: Large-Scale Embodied AI Using Procedural Generation.\" Advances in Neural Information Processing Systems 35 (2022): 5982-5994.\n\n[3] Wen, Zehao, et al. \"Anyhome: Open-vocabulary generation of structured and textured 3d homes.\" arXiv preprint arXiv:2312.06644 (2023).\n\n[4] Yang, Yandan, et al. \"Physcene: Physically interactable 3d scene synthesis for embodied ai.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[5] Tang, Jiapeng, et al. \"Diffuscene: Denoising diffusion models for generative indoor scene synthesis.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[6] Höllein, Lukas, et al. \"Text2room: Extracting textured 3d meshes from 2d text-to-image models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[7] Feng, Weixi, et al. \"Layoutgpt: Compositional visual planning and generation with large language models.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[8] Raistrick, Alexander, et al. \"Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[9] Wang, Yufei, et al. \"Robogen: Towards unleashing infinite data for automated robot learning via generative simulation.\" arXiv preprint arXiv:2311.01455 (2023).\n\n[10] Wang, Lirui, et al. \"Gensim: Generating robotic simulation tasks via large language models.\" arXiv preprint arXiv:2310.01361 (2023).\n\n\n*We hope the additional explanations have convinced you of the merits of our work.*\n*We appreciate your time! Thank you so much!*\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2DrA3TF0DC", "reviewer": "Reviewer_Te9g", "summary": "This paper aims to generate interactive 3D scenes suitable for downstream tasks like robotics. The paper first generates an empty scene and then utilizes pre-trained 2D inpainting models to fill in the ‘foreground’ and apply visual recognition and depth estimation models to ‘lift’ the 2D objects to 3D space via retrieval or off-the-shelf image-to-3D models. Finally, to generate complete and complex scenes observable from multiple views, this paper further applies iterative and hierarchical inpainting. In summary, the framework can generate complex and interactive scenes with detailed asset placement and shows superior results compared with previous works in complexity and realism.", "questions": "Apart from the weakness section, I still have the following questions.\n\n1. The idea of leveraging pre-trained image diffusion models to provide a layout prior to the scene is interesting. However, how well can image diffusion models provide scene layout prior can be discussed more thoroughly. For example, the diffusion model seems to be applied from a certain perspective, lacking the awareness of the holistic scene. Will it generate semantic implausible scenes like two beds in one scene?\n2. The 3D constraint seems to be a 3D bounding box generated by GPT4v, how well can it guarantee that objects will not be floating above the surface or will there be other physical constraints since a physically plausible scene is important for embodied AI tasks?\n3. Is the small objects decomposable? For example, in the Living Room & Dining Room case in Fig.3, there exists a plate with multiple objects on it. Are these objects decomposable in the pipeline of GPT4v--Grounding DINO--2D to 3D? If not, will this fine-grained synthesis results hinder downstream tasks like grasping?", "limitations": "As discussed in the paper, they use off-the-shelf generative models or retrieved models. The diversity of retrieval results is limited and the quality and articulation structure of generative models is also limited.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper successfully mixes a bunch of off-the-shelf base models including GPT4v, Grounding-DINO, SAM, MDE(monocular depth estimation), and inpainting models as well as a bunch of techniques like depth alignment to generate 2D content and then lift them into 3D.\n2. The paper pays attention to the generation of fine-grained small objects like books on the shelf and plates on the table, which is important for embodied AI research like object manipulation tasks.\n3. The proposed method is clear and simple, which should be easy to follow. And the generated scenes seem reasonable and friendly for embodied AI tasks.\n4. The limitations and future work are properly discussed. It points out that the current method retrieves furniture and large objects from datasets, limiting the diversity.", "weaknesses": "1. Faithfulness to image generation results? The paper seems to be using retrieval models or text-to-3D models after acquiring coarse 3D information about an object, which may be unfaithful to the appearance cue provided by the diffusion model.\n    \n2. Unfair comparison. The comparison with Text2Room does not seem very fair as this paper retrieves furniture items from carefully designed datasets while Text2Room generates furniture items from scratch. On the other hand, as the paper focuses on embodied AI tasks, PhyScene[1] can serve as a potential baseline that considers many physical plausible constraints and it would be interesting to see the comparison results.\n    \n3. Limited technical contribution. Though the paper mixes lots of off-the-shelf models, the main idea is to acquire layout from pre-trained image diffusion models, whose capability can be justified in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Te9g\n*We appreciate your positive and insightful comments! Below, we address your concerns in detail.*\n\n**1. Faithfulness to image generation results.**\n\nSince our ultimate goal is to generate diverse and realistic interactive scenes, the inpainted images serve as guidance to place all the assets reasonably rather than as a strict target to align with.\n\nFurthermore, we have applied several methods to improve faithfulness, such as retrieving assets based on image similarity. The details are provided in General Response 3.\n\nWe've also evaluated the image similarity between inpainted images and images of generated scenes against empty scenes, as shown in Rebuttal Table 1. The results indicate that, although not exactly the same, the generated scenes are to some degree faithful to the image generation results.\n\n**2. Comparison with baselines.**\n\nThe comparison with PhyScene is shown in Rebuttal Table 2, where we present the comparison results for generated living rooms (only the weight of living room generation is released).\n\nIn our experiments, we aim to provide a general comparison with three types of related works: works that generate only the static mesh [1], works that are trained on existing datasets [2] and works that generate open-vocabulary scenes using foundation models [3].\n\nIt's challenging to make a completely fair comparison between our methods and the Text2Room method since they serve different purposes. While Text2Room generates the entire mesh without retrieving objects, none of its assets are interactive and it may achieve higher photorealism by directly generating meshes from 2D images.\n\n**3. Technical Contribution.**\n\nWe are the first to utilize 2D diffusion models for interactive scene generation. We have also implemented a pipeline that accurately lifts 2D inpainted images to 3D space, leveraging known camera parameters and partial depth information during the rendering process.\n\nThe problem of accurately acquiring the 3D information for a single generated 2D image challenging. Some previous works such as [4] try to address this problem but can't get accurate results due to the lack of information of depth scale and camera informations. We are the first to address this problem by rendering in simulation and inpainting, where we naturally get the camera informations and the scale of depth.\n\n### Questions\n\n> **Q1: The diffusion model seems to be applied from a certain perspective, lacking the awareness of the holistic scene. Will it generate semantic implausible scenes like two beds in one scene?**\n\nAn inverse problem is mentioned by reviewer YmnU. \nThis is addressed in our work by:\n1. Ensuring that later views can observe some generated assets from prior views and remain consistent during inpainting.\n2. Using a large language model to guide the inpainting process, as mentioned in paper lines 189-193. This helps prevent inconsistencies.\n\n> **Q2: The 3D constraint seems to be a 3D bounding box generated by GPT4v, how well can it guarantee that objects will not be floating above the surface or will there be other physical constraints since a physically plausible scene is important for embodied AI tasks?**\n\nThe 3D constraint is based on 3D bounding boxes from point clouds, generated through segmentation and depth estimation. We directly generate the constraints based on the distance and directional relationships between these bounding boxes; more details can be found in Appendix A of the paper.\n\nSimilar to Holodeck[3], during the process of solving these constraints, we always enforce collision-free placement between furniture pieces, ensuring all furniture is either placed on the floor or hung on the wall.\n\nFor small objects, we use the 3D positions of the bounding boxes. Given that the objects are all placed in reasonable locations (without severe penetration or floating), we can simply run forward physics simulations for several steps to resolve these issues.\n\n> **Q3: Is the small objects decomposable? For example, in the Living Room & Dining Room case in Fig.3, there exists a plate with multiple objects on it. Are these objects decomposable in the pipeline of GPT4v--Grounding DINO--2D to 3D? If not, will this fine-grained synthesis results hinder downstream tasks like grasping?**\n\nIt's not always decomposable, given that Grounding DINO might fail to segment certain objects. For example, the fruit bowl in Figure 3 is recognized and generated as one 3D mesh. This could be mitigated by first generating an empty bowl and then performing hierarchical inpainting inside it.\n\nThis limitation does not affect our downstream pipeline since we are automatically generating tasks based on the ground-truth semantics of each object in the scene. The fruit bowl will be treated as a single asset and thus we won't generate tasks like picking up the fruit from the fruit bowl.\n\n[1] Höllein, Lukas, et al. \"Text2room: Extracting textured 3d meshes from 2d text-to-image models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Tang, Jiapeng, et al. \"Diffuscene: Denoising diffusion models for generative indoor scene synthesis.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[3] Yang, Yue, et al. \"Holodeck: Language guided generation of 3d embodied ai environments.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Chen, Boyuan, et al. \"Spatialvlm: Endowing vision-language models with spatial reasoning capabilities.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n*We sincerely appreciate your comments. Please feel free to let us know if you have further questions.*\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kix4kvkcFg", "reviewer": "Reviewer_4D8M", "summary": "The paper proposed a pipeline for 3D scenes generation, which leveraging 2D prior from  diffusion-based image and depth generation. The main proposed point is the hierarchical inpainting, a inpainting mask is generated from a simulated 3D environment, providing a good condition to control the 2D image generation.", "questions": "1. The operation of the depth map is not clear, what is the meaning of *estimated depth map and referenced depth map,  what is the goal of the scaling? \n2.   How many iterations is needed? and what is the ending condition?\n3. the showed results are conditioned simple texts, does the method supports more complex text descriptions?", "limitations": "I don't see any negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper demonstrates a possible approach towards 3D scenes generation by leveraging the prior from 2D diffusion models. \n2. The proposed method can support the small objects, which is an advantage compared to other methods.", "weaknesses": "1. The effectiveness of the proposed hierarchical inpainting is not well verified, without the control of *3D constraints, how is the quality of the generated scenes?*\n2. In 4.2, the paper presents a quite simple example for embodied tasks, this is an important application scenario for 3D scene generation, more results are required to verify the effectiveness of the proposed methods can be used for embodied tasks.\n3. typo  in line 16: hierachical → hierarchical", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 4D8M\n*Thank you for your insightful and constructive comments! We discuss some of your questions and concerns below.*\n\n**1. Without the control of 3D constraints, how is the quality of the generated scenes**\n\nTo clarify, **the 3D constraint is generated by the hierarchical inpainting** (refer to lines 232 to 245 in the paper). Specifically, after acquiring estimated 3D bounding boxes, if we directly place furniture according to them, it would be less accurate since the point cloud is partial. As a result, we first generate constraints based on those bounding boxes, such as *A is around (x, y, z)* or *A is on the left of B*. We then solve these constraints while enforcing collision-free placement between objects to obtain the final result that best satisfies the constraints.\n\n\n**2. More results for embodied tasks**\n\nWe show more experimental results in General Response 2A.\n\n**3. Typo**\n\nThank you for mentioning! We will fix that.\n\n### Questions\n\n> **Q1: The operation of the depth map is not clear, what is the meaning of estimated depth map and referenced depth map, what is the goal of the scaling?**\n\nIn the pipeline, we first render an image and then inpaint within this image.\n\nFor the rendered image, we have the ground truth depth for each pixel since it is rendered in simulation, which serves as the reference depth map.\n\nFor the inpainted image, we estimate per-pixel depth using Marigold, resulting in the estimated depth map.\n\nSince Marigold[1] generates only a normalized depth map (and other monocular depth estimation methods might also be confused by the scale of depth), we align the scale of the estimated depth with the reference depth. This alignment is based on the fact that the depth values should match in the uninpainted pixels.\n\n> **Q2: How many iterations is needed? and what is the ending condition?**\n\nAs mentioned in General Response 3C, we select up to three views for large object placement, and one view to place small objects for each selected large object. The process stops after these views have been used. The iterations needed for placing large furniture in one room are up to 3, and the iterations needed for placing small objects are conditioned on the number of large furniture pieces present.\n\n> **Q3: The showed results are conditioned simple texts, does the method supports more complex text descriptions?**\n\nIt does support more complex text descriptions, benefiting from the hierarchical structure. While diffusion models alone can't handle complex prompts, we use an LLM as central control to interpret complex descriptions and distribute tasks to the diffusion models. For example, given a text description like *a living room with a shelf full of books and a desk with a laptop on it*, the LLM will include the shelf and desk in the positive prompt of the diffusion model to ensure their presence. It will also adjust the prompt to *a shelf full of books* to place books on the shelf and include the laptop as a positive prompt when placing objects on the desk.\n\nA similar idea is demonstrated in General Response 2B.\n\n[1] Ke, Bingxin, et al. \"Repurposing diffusion-based image generators for monocular depth estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n*We wish that our response has addressed your concerns, and turns your assessment to the positive side. If you have any more questions, please feel free to let us know during the rebuttal window.*\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response to All Reviewers
*We express our gratitude to all the reviewers for their perceptive comments and helpful suggestions aimed at enhancing the quality of our work.*

**1. Our Contributions**

We are pleased that the reviewers have generally acknowledged our contributions:

* We properly leverage 2D priors from vision generative models to generate 3D interactive scenes.
* We implement a hierarchical inpainting pipeline that can 'control' the inpainting process, allowing for accurate 2D to 3D lifting.
* Our method demonstrates a strong ability to generate reasonable 3D interactive scenes compared to previous works, especially in the detailed placement of small objects.

**2. New Experiments**

In this rebuttal, we add several more supporting experiments to address reviewers’ concerns, and we select three of them to list in the General Response. 

* **[A] Embodied Skill Acquiring.** We further tested our methods based on previous work, RoboGen. Specifically, we aimed to automatically generate and solve manipulation tasks using scenes generated by our pipeline. Taking the scene configuration as input, we used GPT-4 to generate tasks, decompose them, and solve them by calling primitive functions we provided. We can now generate more diverse and longer-term tasks, such as *put the mango in the fridge and move the soda from the fridge to the dining table*, which involves multiple rooms. The comparison of diversity is shown in Rebuttal Table 1 by the self-BLEU score of task discription, and some example tasks are shown in Rebuttal Figure 1.

* **[B] Scene Editing.** To demonstrate that our pipeline is compatible with scene editing and complex text control, we implemented additional APIs to add, remove, and rescale objects, enabling LLMs to edit the scene. Initial results for scene editing are shown in the right part of Rebuttal Figure 3. We issued commands to LLMs such as *replace the books on the shelf with vases*, *replace the bookshelf with a cabinet*, and *make the bookshelf smaller*. The LLMs achieved the correct results by calling the provided APIs.

* **[C] Large Furniture Generation.** To address one of our limitations, the dependence on a large furniture database, we apply a pipeline to generate high-quality large furniture. It optimizes a differentiable tetrahedron mesh [1] with SDS loss, using the normal-depth diffusion model and albedo diffusion model provided by RichDreamer[2] as the main supervision signal. This pipeline is capable of generating high-quality object meshes from text guidance, specifically large furniture in our case. Some results are shown in the right part of Rebuttal Figure 2.


**3. Implementation Details**

We observe that most of questions arise from the implementation details [YpUS, YmnU, yxqm, 4D8M]. Here, we give some detailed description of how the pipeline works.

* **[A] Large Furniture Retrieving.** Following Holodeck, for each piece of large furniture, we first retrieve multiple candidates from the dataset using text descriptions of the assets. Then, we select one asset from the retrieved candidates based on scale similarity, which is calculated as the L1 difference between the scale of 3D bounding box of object point cloud and the 3D bounding box of object mesh. Additionally, we integrated image similarity using the cosine similarity of CLIP features in the selection process in our latest pipeline. Here, scale similarity and image similarity are used only in the candidate selection process rather than the retrieval process, since there could be significant occlusions in the image (e.g., a chair behind a table) that could greatly influence the accuracy of retrieving.

* **[B] Small Objects Generation and Selection.** As also mentioned in our paper (line 255), we use a text-to-3D pipeline (text-to-image and image-to-3D) to generate 3D assets for small objects. To make the scene more reasonable and resemble the inpainted image, we generate multiple candidates for each type of object and use the cosine similarity of DINO features to select from the candidates. We also experimented with an image-to-3D pipeline, starting from the object image segmented from the inpainted image. However, the resolution of the segmented image is low, resulting in sub-optimal 3D shapes and textures.

* **[C] View Selection.** For large furniture placement, we heuristically select up to three views (right-back corner to left-front corner, front middle to back middle, and left-back corner to right-front corner) that can cover the whole room area for inpainting. Assuming the room ranges from $(0, 0)$ to $(x, y)$, the three views would be looking from $(x, y, 1.8)$ to $(0, 0, 0.5)$, from $(\frac{x}{2}, 0, 1.8)$ to $(\frac{x}{2}, y, 0.5)$, and from $(0, y, 1.8)$ to $(x, 0, 0.5)$. We stop inpainting from new views when the occupancy of the room is larger than 0.7 or it has been inpainted from all three views.
For small object placement, we first ask LLMs to determine which objects can accommodate small objects on or in them, and then inpaint each of them with heuristic relative views. For objects like tables or desks on which we are placing items, we use a top-down view. For shelves or cabinets in which we are placing objects, we use a front view. The distance of the camera from the object is adjusted according to the scale of the object and the camera's FOV, ensuring the full object is visible during inpainting.

We hope our responses below convincingly address all reviewers’ concerns. We thank all reviewers’ time and efforts again!

[1] Guo, Minghao, et al. "TetSphere Splatting: Representing High-Quality Geometry with Lagrangian Volumetric Meshes." arXiv preprint arXiv:2405.20283 (2024).

[2] Qiu, Lingteng, et al. "Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Meta-review: The submission initially received mixed reviews; the authors did a great job during the rebuttal, after which all reviewers became positive about the submission.  The AC agrees with the recommendations.  The authors should incorporate the rebuttal into the camera ready.
Decision: Accept (poster)