Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting

Authors: yian  wang
xiaowen  qiu
jiageng  liu
zhehuan  chen
jiting  cai
yufei  wang
tsun-hsuan  wang
zhou  xian
chuang  gan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:01:36

Abstract: Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. However, generating diverse embodied environments with realistic detail and considerable complexity remains a significant challenge. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing $\textit{Architect}$, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. While there are still challenges that the camera parameters and scale of depth are still absent in the generated image, we address those problems by ''controlling'' the diffusion model by $\textit{hierarchical inpainting}$. Specifically, having access to ground-truth depth and camera parameters in simulation, we first render a photo-realistic image of only the background. Then, we inpaint the foreground in this image, passing the geometric cues to the inpainting model in the background, which informs the camera parameters.
This process effectively controls the camera parameters and depth scale for the generated image, facilitating the back-projection from 2D image to 3D point clouds. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate the placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments. Experimental results demonstrate that $\textit{Architect}$ outperforms existing methods in producing realistic and complex environments, making it highly suitable for Embodied AI and robotics applications.

Main paper: 
 introduction: 
 Figure 1 : We present ARCHITECT, a generative framework to create diverse, realistic, and complex Embodied AI scenes. Leveraging 2D diffusion models, ARCHITECT generates scenarios in an open-vocabulary manner. Here, we showcase two cases in detail: an apartment and a grocery store.Collecting or generating large-scale training data has recently emerged as a promising direction for advancing Robotics and Embodied AI research. A major focus in recent works pursuing this direction advocates for data generation in simulated environments [Wang et al., 2023b ,a, Ha et al., 2023 , Dalal et al., 2023] , as simulation offers a cost-effective approach to data collection that scales naturally with computational resources; this thrust holds the potential for producing realistic physics and rendering data, and meanwhile grants access to valuable ground-truth state information for speeding up policy learning. Among the types of data needed for training Embodied AI agents, diverse and realistic environments with the possibility of interacting with surrounding entities is crucial. However, obtaining vivid interactive scene and environment data remains challenging. Recent studies have attempted to tackle this problem by developing generative models for environment creation via various approaches, including procedural generation with predefined rules [Deitke et al., 2022] , diffusion-based scene generation [Tang et al., 2023a , Yang et al., 2024b , Feng et al., 2024] , and large language model (LLM) guided scene population and design [Wang et al., 2023b , Yang et al., 2024c , Wen et al., 2023] .Despite these recent efforts, generating diverse, realistic, and complex Embodied AI environments still remains a challenging problem due to the inherent drawbacks and assumptions made in the pipeline designs of existing methods. For example, manually designed environment datasets [Ramakrishnan et al., 2021 , Weihs et al., 2021 , Li et al., 2023a , Fu et al., 2020a,b] require excessive human effort and are hence inherently difficult to scale. Procedural generation methods [Deitke et al., 2022 , Khalifa et al., 2020 , Earle et al., 2021 , Zhao et al., 2021] rely on predefined rules, which are limited in their ability to learn from and resemble real-world distributions, and struggle to generate open-vocabulary scenes. Large language model (LLM) guided scene generation process [Wang et al., 2023b , Yang et al., 2024c , Wen et al., 2023 , Lin et al., 2023 , Aguina-Kang et al., 2024 , Feng et al., 2024 ] also presents its own limitations, as LLMs operate in language space and have limited 3D understanding and spatial reasoning capabilities. Moreover, existing LLM-based scene generation methods still rely on certain simplifications and hand-designed rules, such as primarily focusing on placements of large furniture pieces on the floor or against walls, and only considering simple inter-object relationships such as random placement of small items on top of large background furniture. Therefore, these methods struggle to generate more complex and cluttered object arrangements that are often encountered in daily life, such as "an organized dining table", "an office desk drawer cluttered with objects", or "a shelf of toys", which typically require nuanced object placement and context-aware positioning.As a result, there still exists a gap in current literature for generating interactive 3D scene with detailed and complex configurations that closely resemble real-world distributions. To this end, we propose ARCHITECT, a generative framework for creating realistic and interactable 3D scenes via diffusion-based 2D inpainting [Podell et al., 2023] . Our pipeline leverages controllable and hierarchical generation in 2D image space. Compared to LLMs which operate in language space, pre-trained image-based generative models are able to better capture scene and object configurations from massive image data readily available, both at the scene level and in fine-grained inter-object spatial information. Pre-trained depth estimation models [Ke et al., 2024 , Bhat et al., 2023 , Yang et al., 2024a] can then be used to lift the generated 2D static image to 3D environments. However, images created from 2D generative models do not provide accurate camera parameters, which are crucial for reconstructing accurate 3D environments. In addition, the predicted depth images also present scale ambiguity. To address these challenges, we propose to "control" the 2D generative models with 3D constraints via hierachical inpainting. First, we render a photo-realistic image in a simulated empty scene with only a static background, where we have access to ground-truth depth and camera parameters. We then use this image as a template for inpainting the foreground using 2D diffusion models. During this process, the generation respects the camera parameters informed by the geometric cues in the background image and ensures that the inpainted components are both semantically and spatially consistent with existing components in the input image. By generating images this way, we effectively control the camera parameters and depth scale for the generated image, which allows us to project it back to 3D point clouds. Subsequently, utilizing visual recognition models [Kirillov et al., 2023 , Liu et al., 2023 , Ren et al., 2024] , we segment the 2D image to obtain the semantics and geometric information of each generated object. These objects are then instantiated in the actual simulated environments, by either retrieving from large-scale asset databases [Deitke et al., 2023a , Mo et al., 2019] or generating using image-to-3D generative models [Xu et al., 2024] .While the pipeline described above is able to generate 3D scene configurations described from a single camera view, our goal is to generate complete scenes observable from multiple views. In addition, we aim to generate scenes with real-world complexity, where objects of different scales together form a holistic environment (e.g. ideally we want to also generate small items placed on a shelf or in a drawer). Therefore, we further extend the pipeline to perform iterative and hierarchical inpainting, during which we continuously render new image patches of different locations of the scene to further enhance the complexity when needed. Specifically, given a text description of a target scene, we (i) generate the floor plan following previous works [Yang et al., 2024c , Wen et al., 2023] , (ii) add background assets such as walls and floors into a simulated environment, (iii) render images of this empty scene and perform the aforementioned, proposed iterative inpainting process for scene-level generation from multiple camera views, (iv) hierarchically, apply inpainting again at a finer level to place small objects in various semantically plausible locations in the interior space, and (v) finally resulting in a complex 3D scene. Note that such iterative process results in a flexible generative pipeline that can handle different levels of inputs: text descriptions, floor plans, or even pre-arranged scenes.Our pipeline is able to generate complex scenes that are fully interactable, with detailed asset placement and configurations at multiple scales, as shown in Figure 1 . Note that since we make use of powerful prior knowledge encoded in 2D pre-trained generative models, we are able to generate open-vocabulary scenes in a zero-shot manner, for not only diverse room types in home settings, but also non-home environments such as grocery stores. Our experiments show that our framework outperforms prior scene creation approaches in generating interactable scenes that are more complex and realistic. We summarize our main contributions as follows:• We introduce ARCHITECT, a zero-shot generative pipeline that creates diverse, complex, and realistic 3D interactive scenes to advance Embodied AI agents and Robotics research.• We propose to leverage 2D prior from vision generative models to facilitate the 3D interactive scene generation process, and make such process controllable by initializing from simulation-rendered image for hierarchical inpainting, ensuring consistent spatial features and controllable camera parameters and depth scale, allowing accurate 2D to 3D lifting.• The experimental results show that our method outperforms previous approaches in generating more complex and realistic interactive 3D scenes, both quantitatively and qualitatively.No Train No Human Effort Interactive Organized Small Objects Open Vocab Behavior-1k Our code will be made publicly available.✓ ✓ ProcTHOR ✓ ✓ Holodeck ✓ ✓ ✓ ✓ AnyHome ✓ ✓ ✓ RoboGen ✓ ✓ ✓ ✓ PhyScene ✓ ✓ DiffuScene ✓ LayoutGPT ✓ Text2Room ✓ ✓ ✓ Ours ✓ ✓ ✓ ✓ ✓Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The paper's contributions, a zero-shot generative pipeline that creates diverse, complex, and realistic 3D interactive scenes and the paper's scope are accurately reflected by main claims in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Indoor Scene Generation A large body of works have focused on automatic indoor scene generation. Some works generate only the static mesh of the scene [Höllein et al., 2023 , Schult et al., 2023] ; in contrast, ours generates interactive scenes that can be used for downstream embodied AI and robotics tasks. One line of research generates interactive indoor scenes via procedure generation with manually defined rules [Deitke et al., 2022] . The quality and diversity of the generated scenes highly depend on the predefined rules, which demands huge human efforts. Another group of works train a generative model (e.g., transformers or diffusion models) on in-door scene datasets [Fu et al., 2021] and use the trained models for scene generation [Yang et al., 2024b , Feng et al., 2024 , Tang et al., 2023a , Paschalidou et al., 2021] , and the quality and diversity of the scenes are bounded by the training dataset. Ours differ from these two lines of work as we do not use any manually defined rules nor pre-collected datasets, which might constrain the diversity of the generated scenes. Instead, we achieve higher diversity in the generated scenes by leveraging 2D image generative models that are trained with abundant internet data, which cover a wider distribution of scenes than those generated by manually defined rules or a fixed dataset. Recently, several works employ a Large Language Model (LLM) for indoor scene generation [Wang et al., 2023b , Yang et al., 2024c , Wen et al., 2023] , such as floor plan, layout, and object placements. Since the 3D spatial reasoning abilities of LLMs are still limited, the quality and diversity of the generated scenes are still bounded. By combining 2D image generative models, simulation rendering and controlled image impainting, our method achieves more coherent 3D layouts and higher scene diversity. We make a comparison between us and previous works in Table 2 .Text-to-Image Diffusion models Text-to-image models based on diffusion model, such as DALL-E2 [Ramesh et al., 2021] and LDM [Rombach et al., 2021] , have become dominant in text-to-image generation. These text-to-image diffusion models have been trained on billions of images, giving them strong visual and 3D priors in addition to their image generation capabilities. Through SDS (Score Distillation) loss proposed by DreamFusion [Poole et al., 2022] , the 3D prior can be leveraged for a variety of downstream tasks like 3D object generation [Poole et al., 2022 , Tang et al., 2023b , Wang et al., 2023c , Liang et al., 2023] . In our observations, in addition to visual and 3D priors, these text-to-image diffusion models also have strong priors for the layout of furniture and objects in a room. Therefore, we designed a pipeline to exploit these layout priors from inpainting diffusion models to generate realistic indoor scenes.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The paper discuss the limitations in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We automatically generate complex and realistic embodied environment scenes through a hierarchical and iterative process of rendering, inpainting, and visual reasoning. Specifically, as shown in Figure 2 , starting from an empty room, our pipeline first iteratively generates the layout of large furniture items (Figure 2 Large Furniture). Subsequently, we place smaller objects inside or upon these large furniture pieces, as depicted in Figure 2 Small Objects. In detail, our process entails the following steps: (1) Initializing: We begin by selecting a view in the scene and rendering an image using Pyrender [Matl, 2019] and LuisaRender [Zheng et al., 2022] , and generating an inpainting mask; (2) Inpainting: Given a text prompt generated by LLMs, along with the rendered image and masks from step 1, we inpaint the image according to the text prompts, leveraging Latent-Diffusion [Rombach et al., 2021] ; (3) Visual Perception: Upon recognizing the inpainted image, we generate 3D bounding boxes for objects using GPT4v, Grounded-SAM, and Marigold [OpenAI, 2023 , Kirillov et al., 2023 , Liu et al., 2023 , Ren et al., 2024 , Ke et al., 2024] , as well as the rendering parameters from step 1; (4) Placing: We place objects into the scene according to the 3D bounding boxes and return to step 1 to continue generating new objects in the next iteration.Overview In this module, we initialize the viewpoint for the iteration, rendering an image and obtaining the inpainting mask and ground-truth depth for the following steps.Large Furniture Given an empty room, we heuristically choose the first view that spans from one corner to the opposite corner, maximizing the visible space of the room to render an image, as in Figure 2 Empty Scene. Setting a light source in the middle top of the room, we use a ray-tracing based method to render a high-quality image and a raster-based method to obtain the ground-truth depth and object segmentation. Next, we generate an inpaint mask for the image, which is centered within the frame, as shown in Figure 2 Inpainting Mask. If objects are already present in the scene, we utilize the object segmentation mask obtained from the rasterizer to filter out those pixels from the inpaint mask, ensuring that any furniture or objects placed within the room will not disappear from the newly inpainted images.Small Objects As illustrated in Figure 2 Small Objects, to place small objects on large furniture, we first heuristically choose a front-top view or a front view depending on whether we are placing on top of a large object (e.g., a table) or inside an object (e.g., a shelf). To obtain the inpainting mask, we place a cube within or atop the bounding box of the larger furniture object, with a size slightly smaller than that of the furniture object itself, as shown in Figure 2 Cube Filling. In cases placing small objects on top of larger ones, we position the cube on top of the larger object's bounding box, with a fixed height and the other two dimensions slightly smaller than those of the larger object. Similar to large furniture, we remove the pixels of existing small objects from the inpainting mask.Overview In this module, we utilize the image and inpainting mask provided from the previous step to first generate text prompts using LLMs. Subsequently, we use these prompts for image inpainting.The inpainting process for both large furniture and small objects is depicted in Figure 2 Inpainting. To ensure smoother inpainting results, we apply techniques such as erosion and Gaussian blur to the mask before commencing the inpainting process. This preparation allows for more effective filling of the contents within the mask. To enhance the diversity and ensure the generation of reasonable objects, we prompt LLMs to automatically generate suitable text prompts and negative prompts for the inpainting model. For example, when we input the configuration of a partially generated living room that includes a TV set into an LLM, the LLM will place the word "TV" in the negative prompt, reasoning that a living room normally has only one TV set.During this step, we generate multiple inpainted images. If the number of recognized objects in a generated image falls below a predefined criterion, we filter out this image and generate new ones.Overview This module takes the inpainted image, ground-truth depth, and camera parameters to recognize and segment objects, estimate their depth, back-project them into 3D, and finally output 3D bounding boxes for each object.For object recognition, we initially utilize GPT4v to detect all objects present in the image. The identified object names then serve as tags for Grounding-Dino, which performs object detection and provides the output bounding boxes. Following this, we use the SAM to obtain instance-level segmentation masks based on these bounding boxes, as depicted in Figure 2 Object Grounding.After that, we estimate the relative depth of the generated image and rescale the predicted depth using reference depth. Specifically, giving n reference pixel set P r = {(i 0 , j 0 ), ..., (i n-1 , j n-1 )}, referenced depth map D r in R W ×H where W and H are the resolution of the image, estimated depth map D e in R W ×H , we rescale the estimated depth map to D rescaled in the following formulas:max r = max t∈[0,n-1] D (it,jt) r , min r = min t∈[0,n-1] D (it,jt) r , max e = max t∈[0,n-1] D (it,jt) e min e = min t∈[0,n-1] D (it,jt) e , scale = max r -min r max e -min e D rescaled = D e • scale - 1 n t∈[0,n-1] D (it,jt) e • scale + 1 n t∈[0,n-1] D (it,jt) rWe employ different strategies when selecting reference pixels P r . For placing large furniture, we utilize all the non-masked parts of the image as reference pixels to ensure general consistency with the room's floors and walls. For small objects, we focus on the non-masked areas of large furniture as reference pixels, aiming for consistency specifically with the inpainted object. This approach is adopted because the depth information outside the inpainted objects exhibits discontinuities that are challenging to predict accurately. For instance, as shown in Figure 2 Depth Estimation, predicting the depth of the wall behind the shelf is difficult and could introduce noise if considered.Once the depth estimation is acquired, we use the camera parameters from the rendering process to back-project the depth into a 3D point cloud. Utilizing the 2D masks provided by SAM, we extract the point cloud for each object instance. To eliminate any outliers, we apply DBSCAN [Khan et al., 2014] clustering to each segmented object, which allows us to derive axis-aligned bounding boxes for each object.Overview Equipped with the 3D bounding boxes of objects, this module places the objects into the simulation and returns to the Initializing phase to commence the next iteration.Large Furniture To incorporate large furniture into a scene utilizing 3D bounding boxes, we initially retrieve each piece of furniture according to a text description generated by GPT4v. After extracting a list of instances from the datasets, we select them based on feature similarity and the proportionality of their scale in three dimensions. The scale of each item is adjudicated by large language models using common sense knowledge.Owing to the possibility that the retrieved furniture may not precisely conform to the 3D bounding boxes and minor errors in depth estimation, directly placing furniture at the center of the bounding box can lead to issues such as collisions or complications arising from partial view observations. To mitigate these issues, we adopt an alternative approach by generating constraints derived from the 3D bounding boxes. We employ search methods akin to those used in Holodeck [Yang et al., 2024c] to determine optimal placement. Unlike Holodeck, which utilizes LLMs to generate all constraints, we derive ours directly from the generated images and 3D bounding boxes. This search process enables us to avoid collision conflicts while simultaneously ensuring alignment with the generated image. Further details about these constraints can be found in Appendix A.Small Objects To position small objects within a scene as defined by 3D bounding boxes, we first generate 3D instances based on the semantic content of each object and then place them at the specified positions within the 3D bounding boxes. Subsequently, we adjust the orientation and scale of the small objects to match the orientation and size of the bounding boxes. Notably, due to the partial nature of point clouds, it is impractical to uniformly apply the scale of the bounding box across all three dimensions. Instead, we focus on utilizing the scale along the dimensions perpendicular to the viewing direction. The placement process for small objects is depicted in Figure 2 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experiments: 
 Dataset We retrieve objects from Objaverse. Deitke et al. [2023b,a] Objaverse is a dataset that contains massive annotated 3D objects. It includes various objects, including manually designed objects, everyday items, historical and antique items, etc. In the process of generating indoor scene objects, we retrieve suitable furniture from the Objaverse dataset and place them in the scene. We also retrieve articulated objects from PartnetMobility [Xiang et al., 2020] . PartnetMobility contains 2346 3D articulated models from 46 categories, with articulation annotations.Implementation We use Marigold [Ke et al., 2024] as the depth estimation model. We use Grounded-Segment-Anything as our segementation model. We use the SD-XL [Podell et al., 2023] inpainting model provided by diffusers as the image inpainting diffusion model. We use LuisaRender [Zheng et al., 2022] as our renderer. For text-to-3D generation, we first use MVdream [Shi et al., 2023] to generate a image and then feed the image to InstantMesh [Xu et al., 2024] to generate the 3D asset. All experiments, including qualitative evaluation, quantitative evaluation and robotics task are all conducted on an A100 GPU.Baseline We compare ARCHITECT with state-of-the-art indoor scene generation approaches: (1) Holodeck [Yang et al., 2024c] , leveraging common sense from LLMs to generate floor plans and place objects;(2) Text2Room [Höllein et al., 2023] , utilizing the knowledge from image diffusion models and depth estimation models to generate the entire mesh of a scene; (3) DiffuScene [Tang et al., 2023a] , learning a diffusion model to generate the layout of 3D objects in a scene. In addition to the methods mentioned above, AnyHome [Wen et al., 2023] is another baseline we would like to compare with. However, the method is not yet fully open-sourced, so we will leave it for future work.Metric To evaluate the semantic correctness of the generated scenes, we use the following metrics that calculate the similarity between a rendered image and a given caption: (1) CLIPScore [Hessel et al., 2021] , computing the correlation between image feature of the rendered image extracted by CLIP image encoder and text feature of the caption extracted by CLIP text encoder; (2) BLIPScore, using the image-text-matching head of BLIPv2 [Li et al., 2023b] to compute alignment between the rendered image and caption;(3) VQAScore [Lin et al., 2024] , feeding the rendered image and caption into a VQA model, returning the probability that the answer to the question "Does the image show caption" is "Yes" as the score; (4) GPT4o Ranking, asking GPT4o to rank rendered images of generated scenes, and calculate the average ranking as the score. We also conduct a user study to evaluate various aspects of the generated scenes. We ask users to rate the following four indicators from 1 to 5: Visual Quality (VQ), Semantic Correctness (SC), Layout Correctness (LC) and Overall Preference (OP).Qualitative Evaluation We compare rooms generated by our model with those generated by other methods. Figure 3 shows the results for both household and non-household scenes. The comparisons with Text2Room and DiffusScene are provided in Appendix B, as these methods generate either non-interactive scenes or lack diversity. Figure 3 : We compare ARCHITECT with other methods in both household scenes(living room and dining room) and other non-household scenes. We only compared the household scene generated by Diffuscene due to its limitations in Figure 7 and compared with Text2Room in Figure 8 .Compared to Holodeck, our scenes are more realistic, leveraging the capabilities of 2D diffusion models. For example, in the hair salon scenario, Holodeck fails to generate semantically correct scenes because the spatial constraints and objects are entirely generated by LLMs. Additionally, our work demonstrates the ability to generate more complex and detailed placements of small objects. For instance, the shoe store filled with paired shoes, toys on the children's room shelf, and the organized placement of items on the dining room table all exceed the generative abilities of Holodeck. This comparison highlights the effectiveness of our iterative and hierarchical inpainting process, showing that the use of 2D generative models indeed brings more spatial priors compared to LLMs.Quantitative Evaluation We compare ARCHITECT to other state-of-the-art indoor scene generation results using 2D image scores and user studies. The results are shown in Table 2 and Table 4 Figure 5 : Left: the robot organizes the room by pushing the chair under the table and pushing the keyboard inside the table. Right: the robot opens the fridge door, grasps the mango and puts it into the fridge, opens the kitchen-dining room door, grasps the soda can and puts it on the dining room table, and finally closes the fridge.Inspired by previous work RoboGen [ Wang et al., 2023b] , we are making efforts to collect large-scale data for long-term embodied or robotics tasks in our generated scene.A significant challenge we face is that the inclusion of all the detailed small objects in the simulation significantly slows down the speed of the embodied environment. To address this, after generating the scene, task and task decomposition, we use LLMs to select relevant objects for each substep, while all other objects are designated as background objects and will not be physically simulated during this substep.Given our house-level scene generation pipeline, we can now extend RoboGen to generate action trajectories for skills that require long-distance navigation and more complex tasks. Specifically, by inputting the floor plan, large furniture, and small objects into GPT-4, it first generates a task related to the existing objects in the scene. Then, as in RoboGen, it decomposes the task and filters out irrelevant objects to the background. Finally, we leverage action primitives and training supervision generated by LLMs to obtain a trajectory of actions to solve the task. We demonstrate two example generated tasks in Figure 4 , and two other task with corresponding collected trajectories in Figure 5 .The comparison of diversity of is shown in Table 3 by the self-BLEU score of task discription.Baby's CribFigure 6 : Examples of generated small objects and large furniture.To address one of our limitations, the dependence on a large furniture database, we apply a pipeline to generate high-quality large furniture. It optimizes a differentiable tetrahedron mesh with SDS loss [Guo et al., 2024] , using the normal-depth diffusion model and albedo diffusion model provided by RichDreamer [Qiu et al., 2024] as the main supervision signal. This pipeline is capable of generating high-quality object meshes from text guidance, specifically large furniture in our case. Some results are shown in the right part of Figure 6 right part. We also show some qualitative resutls about small object generation in Figure 6 left part, which is another crucial factor of the quality of generated scenes. 
conclusion and future work: 
 In this paper, we propose ARCHITECT, a generative framework capable of creating diverse, realistic, and complex Embodied AI environments. Leveraging pre-trained 2D image inpainting diffusion models that better capture scene and object configurations compared to LLMs, ARCHITECT iteratively extracts diverse and realistic layouts from image inpainting results. We also propose to control this inpainting process by processing geometric cues in the background of a rendered image. This process effectively controls the camera parameters and depth scale for the generated image, allowing us to project it back into 3D point clouds. The scenes generated by ARCHITECT provide realistic and complex environments for downstream Embodied AI and robotics applications. In qualitative and quantitative comparisons, ARCHITECT outperformed baseline methods in both realism and diversity. We believe ARCHITECT is an important step towards creating large-scale interactive 3D environments.Limitation and Future work Currently, ARCHITECT retrieves furniture and large objects from datasets. This means that the diversity of furniture in our results is inherently limited by the dataset.In the future, we will explore generative methods to create more high-quality and articulated objects to further enhance the diversity of the generated scenes. Different from Holodeck that is using LLMs to generate all the constraints, all above constraints are generated by sorting floor objects by size and traversing them based on the position of their bounding boxes except for the rotation constraints. Specifically, for each floor object's bounding box, constraints of each type are assigned based on the distance and directional relationship between its boundaries and those of other floor object bounding boxes. In particular, rotation constraints cannot be solely determined by the bounding box, so an LLM is consulted to obtain the rotation constraint based on common sense (e.g., chairs facing a table). After obtaining the complete constraints, a DFS algorithm is utilized to explore possible placements for each item. Placements that do not meet the hard constraints are filtered out, and the highest scoring placements are selected based on the soft constraint scores. Here, hard constraints refer to mandatory constraints, such as global constraints and position constraints. Soft constraints refer to cumulative scores, where the highest scoring options are prioritized, such as location constraints. In practice, we applied a greedy pruning mechanism to the DFS algorithm, exploring only 3 nodes with highest score at each time. The score S p for each placement is calculated as follows:S p = W loc •   i̸ =i ∆ i • w i + w cur ∆ cur + C   + W rotation • ( n i=1 1 ri )For each current object, W represents the weight of constraints; w represents the weight of each object; ∆ represents the deviation from the reference, which hopes to be close to the reference and away from other items already placed; C represents constants to keep result positive; 1 is a indicator function that equals 1 if the rotation satisfies the constraint; r represents the rotation of the item.As for wall objects, most of its constraint comes from floor. Wall objects constraints are as follows:/* Global Constraint */ Above, object: close to the wall, above a specific floor object./* Soft Location Constraint */ Location, (x, y): predicted bounding-box location./* Position Constraint */ Height: The height of the object.The placement of wall objects is relatively simpler because it does not require specifying an orientation; by default, they face away from the wall. Additionally, the likelihood of conflicts on the wall is lower, and using soft location constraint suffices for effective arrangement. And the searching process of wall objects is just almost the same as floor objects. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the positive and negative social impacts thoroughly in Appendix F.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks, we are using all existing pre-trained models and datasets rather than releasing new ones.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets, used in the paper are properly credited and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The new ARCHITECT method is well documented and the documentation is provided alongside the assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: The paper include the full text of instructions given to participants and screenshots in appendix. The participants are volunteers with no compensation.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: Following Holodeck, for each piece of large furniture, we first retrieve multiple candidates from the dataset using text descriptions of the assets. Then, we select one asset from the retrieved candidates based on scale similarity, which is calculated as the L1 difference between the scale of 3D bounding box of object point cloud and the 3D bounding box of object mesh. Additionally, we integrated image similarity using the cosine similarity of CLIP features in the selection process in our latest pipeline.Here, scale similarity and image similarity are used only in the candidate selection process rather than the retrieval process, since there could be significant occlusions in the image (e.g., a chair behind a table) that could greatly influence the accuracy of retrieving.We use a text-to-3D pipeline (text-to-image and image-to-3D) to generate 3D assets for small objects.To make the scene more reasonable and resemble the inpainted image, we generate multiple candidates for each type of object and use the cosine similarity of DINO features to select from the candidates. We also experimented with an image-to-3D pipeline, starting from the object image segmented from the inpainted image. However, the resolution of the segmented image is low, resulting in sub-optimal 3D shapes and textures.For large furniture placement, we heuristically select up to three views (right-back corner to left-front corner, front middle to back middle, and left-back corner to right-front corner) that can cover the whole room area for inpainting. Assuming the room ranges from (0, 0) to (x, y), the three views would be looking from (x, y, 1.8) to (0, 0, 0.5), from ( x 2 , 0, 1.8) to ( x 2 , y, 0.5), and from (0, y, 1.8) to (x, 0, 0.5). We stop inpainting from new views when the occupancy of the room is larger than 0.7 or it has been inpainted from all three views. Additionally, we use an 84-degree FOV for our camera during rendering, a standard parameter for real-world cameras. Consequently, for a square room, this setup results in approximately 95 percent of the room being visible from a single corner-to-corner view.For small object placement, we first ask LLMs to determine which objects can accommodate small objects on or in them, and then inpaint each of them with heuristic relative views. For objects like tables or desks on which we are placing items, we use a top-down view. For shelves or cabinets in which we are placing objects, we use a front view. The distance of the camera from the object is adjusted according to the scale of the object and the camera's FOV, ensuring the full object is visible during inpainting.As shown in Figure 7 and 8, compared to Text2Room, our method generates more photorealistic scenes. However, since Text2Room directly projects RGB pixels into 3D space using depth maps, artifacts are unavoidable, the geometry of the generated 3D mesh is distorted, and it can't serve as an embodied environment since it's not interactive. These issues are addressed in retrieval-based methods. Compared to Diffuscene in Figure 7 , our method generates more reasonable and detailed scenes. Our results exhibit more detail and precision. Additionally, the dataset lacks information on the placement of small objects, limiting the diversity of small object placement in Diffuscene-generated scenes. The comparison with PhyScene is shown Table 4 , where we present the comparison results for generated living rooms (only the weight of living room generation is released).In our experiments, we aim to provide a general comparison with three types of related works: works that generate only the static mesh, works that are trained on existing datasets and works that generate open-vocabulary scenes using foundation models.It's challenging to make a completely fair comparison between our methods and the Text2Room method since they serve different purposes. While Text2Room generates the entire mesh without retrieving objects, none of its assets are interactive and it may achieve higher photorealism by directly generating meshes from 2D images.In short, our method combines a diffusion-based pipeline with an LLM-based method, which still possess the ability of controlling and editing. The inpainting-to-layout pipeline functions can be considered as an API function callable by the LLM. Our approach aims to generate scene configurations seeded from diffusion models, with scene editing as an othogonal feature enabled by LLMs. Specifically, the scene configuration generated by our pipeline can be represented by each object's name, position, scale, bounding box, orientation, and asset UID, which can be easily converted to text representations. This allows us to feed this information directly into LLMs to further control or edit the scene layout. Corresponding experiments could be found in Appendix C.It's challenging for LLMs to directly solve arrangement problems. First, for small object placement on shelves, LLMs lack information about supporting surfaces, making it impossible for them to solve this issue. Second, for placement on tables, while we might know the supporting surfaces given the bounding boxes, LLMs struggle with object orientations, often resulting in less complex scenes or scenes with severe collisions. We show a comparison of small objects generated by our methods and LLMs in the middle part of Figure 9 and in Table 3 .We've also evaluated the image similarity between inpainted images and images of generated scenes against empty scenes, as shown in Table 3 . The results indicate that, although not exactly the same, the generated scenes are to some degree faithful to the image generation results.The appearance of the masked area is consistent with other areas both stylistically and geometrically. We also apply a commonly used technique, softening the boundary of inpainting masks, to improve consistency. A comparison of the results before and after using softened inpainting masks is shown in the left part of Figure 9 . To demonstrate that our pipeline is compatible with scene editing and complex text control, we implemented additional APIs to add, remove, and rescale objects, enabling LLMs to edit the scene.Initial results for scene editing are shown in the right part of Figure 9 . We issued commands to LLMs such as replace the books on the shelf with vases, replace the bookshelf with a cabinet, and make the bookshelf smaller. The LLMs achieved the correct results by calling the provided APIs. We conducted comprehensive human evaluations to assess the quality of ARCHITECT scenes, with a total of 115 undergraduate students and graduate students participating in the user studies. All participants were volunteers without compensation.We first provided participants with two minutes to read the instructions, the specific content of which was as follows:• Thank you for your participation! This questionnaire is used for the experimental part of scientific research articles, which requires human evaluation. We will keep the information of the participants confidential.• The estimated total time is about 10 minutes.• There are some pictures in the questionnaire from different utils. Just observe and score all of these pictures. The higher the score, the better the quality.• An example is shown in the following image: Then, we randomly assigned each volunteer 23 scenes and asked them to score the scenes from one to five, considering visual quality, semantic correctness, layout coherence, and overall preference. Each volunteer received only 23 scenes to ensure they could complete their responses in approximately 10 minutes.We collected ratings from all participants and calculated the average scores for these four metrics. These average scores were used to evaluate our model, Holodeck, Text2Room and DiffuScene allowing us to compare the scene generation performance of our model with the two baseline models.The average response time was 525 seconds, with the longest response time being 1113 seconds and the shortest being 150 seconds. Responses with a duration of less than 230 seconds were filtered out.Detect all objects in the picture, generate a description for each object, and decide whether it is floor-object or wall-object.Here are the definitions of object types: floor object: object that is placed on floor or in direct contact with the floor. wall object: object that is placed on wall and not in contact with the floor.Here is a sample answer: The prompt above is fed into GPT-4V along with an image generated by a 2D inpainting model. The prompt asks GPT-4V to recognize all objects in the inpainting image, briefly describe them, and then classify them as either objects on the floor or objects on the wall. The generated object names will be used to prompt Grounded-SAM. The generated descriptions will be used for retrieving objects or for text-to-3D generation.Given the objects in the current scene, please list which objects have already reached their potential limits, and the objects are still lacking.Your answer should be in the following format: reached limit: object A, object B, ... lacking: object C, object D, ... The objects in the current scene are: /* a list of objects with quantities, eg: 2 sofa, 1 coffee table, 1 TV*/ Remember, do not answer anything not asked. The lacking objects should ideally contain objects that are not in the scene. The lacking objects you list should be precise, do not give things like "other furniture".The prompt above asks GPT-4V to provide negative prompts and positive prompts in addition to the room caption for the inpainting model. ROOM-CAPTION will be substituted with the actual room caption. The lacking objects will be added to the positive prompt, and the objects that have reached their limit will be added to the negative prompt.F.1 Positive Impacts• Advancements in Robotics and AI: ARCHITECT enhances the development of versatile robots capable of assisting in various tasks.• Educational Tools: Generated 3D environments can be used for immersive learning experiences, aiding in the understanding of spatial relationships and complex systems. • Accessibility: Improved AI environments can lead to the development of assistive technologies for individuals with disabilities, enhancing their quality of life.• Job Displacement: Advanced AI and robotics could potentially displace jobs in certain sectors, necessitating consideration of economic and societal impacts. • Bias and Fairness: Ensuring training data and algorithms are representative and fair is crucial to avoid perpetuating existing biases. • Misuse of Technology: Inferring internal geometric structures of 3D objects could be misused for unauthorized reproduction or surveillance, leading to ethical and legal concerns.NeurIPS Paper ChecklistJustification: There are no potential risks for participants and the IRB approvals were obtained.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 


Decision: The submission initially received mixed reviews; the authors did a great job during the rebuttal, after which all reviewers became positive about the submission.  The AC agrees with the recommendations.  The authors should incorporate the rebuttal into the camera ready.

Reviews: review: paper_summary: paper_summary: The paper introduces ARCHITECT, a generative framework designed to create complex and realistic 3D environments for Robotics and Embodied AI research. Unlike traditional methods that rely on manual design, predefined rules, or large language models (LLMs), ARCHITECT utilizes pre-trained 2D image generative models to capture scene and object configurations more effectively. To address the problem of camera parameters and depth estimation, ARCHITECT utilizes a hierarchical inpainting pipeline.
Specifical contributions are: 1. The paper proposes to leverage 2D prior from vision generative models for the 3D interactive scene generation process to generate complex and cluttered object arrangements; 2. The paper introduces a controlled inpainting process that utilizes a controllable rendering process and integrates geometric cues from the background to guide the generation of the foreground, allowing for better control over camera parameters and depth perception.
strength_weakness: strength_weakness: 1. For methods based on rules and LLMs, we can directly edit the positions, types, and other conditions of objects to control the output. However, when using a pre-trained inpainting model to plan object layouts, it becomes difficult to make such detailed edits, which seems to result in a loss of some controllability.
2. Although the goal of the paper is to generate scenes using multiple viewpoints, the paper only provides the method of selecting the first camera viewpoints. I hope the authors can further clarify the specific rules for selecting multiple viewpoints.
3. Heuristically choosing the view that spans from one corner to the opposite corner results in only half of the room being occupied by objects, which would make the room appear unrealistic.
questions: questions: 1. Can the inpainting model ensure that the appearance of the masked area is consistent with other areas?
2. Could you provide some results about generating using image-to-3D generative models? I found that the paper mentioned this approach in the introduction section but did not provide subsequent experiments.
3. How do you perform object retrieval? Is it like the HOLODECK method?
4. Can the algorithm based on LLMs solve the problem of small object arrangement by modifying the example in the prompt?
limitations: limitations: The authors have addressed the limitations. However, the limitations of inpainting models still need to be further elucidated.
paper_summary: paper_summary: This paper proposes a hierarchical diffusion-based 2D inpainting method for creating interactive 3D scenes. By leveraging the generative prior of 2D diffusion models, the proposed method could generate more realistic and diverse object layouts compared with Holodeck [Yang et al., 2024c] that depends on LLMs which lack 3D spatial reasoning ability. Specifically, it renders a 3D scene (initialized with empty scene) and utilizes its GT depth and camera parameters for the back-projection from 2D inpainted components to 3D point clouds. Based on this layout, it determines how to place 3D objects into the interactive 3D scene, where large objects are retrieved from databases and small objects are generated with text-to-3D and image-to-3D generative models.
strength_weakness: strength_weakness: This paper has the following weaknesses:

(i) The diversity of inpainting results might be affected by the bias of LLMs, sinch inpainting prompts are generated by LLMs. According to L191-193, it says “TV” will be placed into the negative prompt since a living room normally has only one TV set. Such bias might enhance the realism but limit the diversity of inpainting.

(ii) The proposed method essentially enables generating open-vocabulary scenes, but it currently retrieves large furniture pieces from databases. 

(iii) Several implementation details are not fully described. For example, it does not elaborate the predefined criterion used in the filtering (L194-195) and how to finally select large furniture (L224-228).
questions: questions: (i) The proposed method generates small objects using a text-to-3D generation method (MVdream) and an image-to-3D generation method (InstantMesh), but it retrieves large furniture pieces instead of such generation. Is this because the large furniture pieces determine how small objects are placed and thus should be neat? What is the reason for the retrieval?

(ii) According to L194-195, the proposed method filters out images if the number of recognized objects falls below a pre-defined criterion. What is the criterion? Is there also an upper bound? 

(iii) According to L224-228, the proposed method finally selects large furniture pieces based on feature similarity and proportionality of their scale. How did the authors compute feature similarity and balance between the similarity and proportionality?
limitations: limitations: Yes, in page 9 and 18.
paper_summary: paper_summary: This paper considers creating large 3d indoor scenes (e.g. an apartment or grocery store) by an hierarchical generation procedure. The key idea is to use diffusion models for inpainting to guide where to place the objects in the scene. This is done for both large objects (e.g. table or couch) and also small objects relative to the larger ones (e.g. books or a coffee cup on a coffee table). The objects are retried from both large scale object datasets (Objaverse) and image-to-3d models. The paper is compared to several existing methods for scene generation, and found to perform well, generating scenes that adhere to the text prompt. The scenes generated by the method could be useful e.g. for robotics and for training embodied agents, although such experiments are not presented.
strength_weakness: strength_weakness: - The 2d inpainting is used only partially. For the large objects, it is only used as a means of finding object placements in the form of 3d bounding boxes, and then the content within the bounding box (inpainted image, point cloud etc) is more or less discarded and an object is retrieved from Objaverse instead and placed in the 3d bounding box. This is not the case for small objects where the inpainted images are fed to an image-to-3d model, and it is unclear why this distinction between large and small objects was made.
- Several of the closest related works, with two examples being Holodeck [Yang 2024c] and procTHOR [Deitke 2022], evaluate how agents for e.g. object navigation perform after training on their proposed datasets, and evaluate how it generalizes to other datasets. Such policy learning experiments are missing from the current paper.
- Qualitatively, it clearly looks synthetic and not photo-realistic both for the scene layout with the floor and walls, and also for the large objects, which are retrieved from partially synthetic datasets. This is also the case for the closest related work (e.g. Holodeck), so it is not a major issue because of this.
questions: questions: See weaknesses
limitations: limitations: This is adequately addressed
paper_summary: paper_summary: This paper aims to generate interactive 3D scenes suitable for downstream tasks like robotics. The paper first generates an empty scene and then utilizes pre-trained 2D inpainting models to fill in the ‘foreground’ and apply visual recognition and depth estimation models to ‘lift’ the 2D objects to 3D space via retrieval or off-the-shelf image-to-3D models. Finally, to generate complete and complex scenes observable from multiple views, this paper further applies iterative and hierarchical inpainting. In summary, the framework can generate complex and interactive scenes with detailed asset placement and shows superior results compared with previous works in complexity and realism.
strength_weakness: strength_weakness: 1. Faithfulness to image generation results? The paper seems to be using retrieval models or text-to-3D models after acquiring coarse 3D information about an object, which may be unfaithful to the appearance cue provided by the diffusion model.
    
2. Unfair comparison. The comparison with Text2Room does not seem very fair as this paper retrieves furniture items from carefully designed datasets while Text2Room generates furniture items from scratch. On the other hand, as the paper focuses on embodied AI tasks, PhyScene[1] can serve as a potential baseline that considers many physical plausible constraints and it would be interesting to see the comparison results.
    
3. Limited technical contribution. Though the paper mixes lots of off-the-shelf models, the main idea is to acquire layout from pre-trained image diffusion models, whose capability can be justified in detail.
questions: questions: Apart from the weakness section, I still have the following questions.

1. The idea of leveraging pre-trained image diffusion models to provide a layout prior to the scene is interesting. However, how well can image diffusion models provide scene layout prior can be discussed more thoroughly. For example, the diffusion model seems to be applied from a certain perspective, lacking the awareness of the holistic scene. Will it generate semantic implausible scenes like two beds in one scene?
2. The 3D constraint seems to be a 3D bounding box generated by GPT4v, how well can it guarantee that objects will not be floating above the surface or will there be other physical constraints since a physically plausible scene is important for embodied AI tasks?
3. Is the small objects decomposable? For example, in the Living Room & Dining Room case in Fig.3, there exists a plate with multiple objects on it. Are these objects decomposable in the pipeline of GPT4v--Grounding DINO--2D to 3D? If not, will this fine-grained synthesis results hinder downstream tasks like grasping?
limitations: limitations: As discussed in the paper, they use off-the-shelf generative models or retrieved models. The diversity of retrieval results is limited and the quality and articulation structure of generative models is also limited.
paper_summary: paper_summary: The paper proposed a pipeline for 3D scenes generation, which leveraging 2D prior from  diffusion-based image and depth generation. The main proposed point is the hierarchical inpainting, a inpainting mask is generated from a simulated 3D environment, providing a good condition to control the 2D image generation.
strength_weakness: strength_weakness: 1. The effectiveness of the proposed hierarchical inpainting is not well verified, without the control of *3D constraints, how is the quality of the generated scenes?*
2. In 4.2, the paper presents a quite simple example for embodied tasks, this is an important application scenario for 3D scene generation, more results are required to verify the effectiveness of the proposed methods can be used for embodied tasks.
3. typo  in line 16: hierachical → hierarchical
questions: questions: 1. The operation of the depth map is not clear, what is the meaning of *estimated depth map and referenced depth map,  what is the goal of the scaling? 
2.   How many iterations is needed? and what is the ending condition?
3. the showed results are conditioned simple texts, does the method supports more complex text descriptions?
limitations: limitations: I don't see any negative societal impact.

Comments: title: Thank you and we are looking forward to your post-rebuttal feedback!
comment: Thanks for the effort in the rebuttal. My concerns are resolved to a large extent so my score will remain unchanged.
I thank the authors for the answers which clarified my concerns, and I appreciate the experiments with embodied skill acquiring (sec. 2A in the general response post). I updated my score one step to weak accept.
We are glad to know that you are leaning to accept this paper and that your concerns have been addressed. If there’s anything more we can do to further enhance your view of our work, we would be glad to provide additional details. Thank you again!
We are pleased to hear that your evaluation has shifted positively and that your concerns have been addressed. We plan to incorporate these details into the main paper in future versions. If there’s anything else we can do to further elevate your opinion of our work, we would be happy to provide additional information. Thank you once again!
Dear reviewer yxqm,

We are truly grateful for your insightful comments and advice, which have played a significant role in enhancing the quality and clarity of our paper.

We hope that the additional details and experimental results we provided have effectively addressed your concerns. As the rebuttal period comes to an end, we kindly request your thoughts on our rebuttal and ask that you consider raising your score accordingly. If there are any remaining concerns, please feel free to share them with us.

Once again, we deeply appreciate your thoughtful review and constructive feedback.

Best,

Authors
I have read the rebuttal carefully and most of my concerns are addressed. Hence, I decide to keep my original score and am leaning to accept this paper.
Dear reviewer YpUs,

Thank you once again for your insightful comments and advice, which have been instrumental in improving the quality and clarity of our paper.

As the rebuttal phase is drawing to a close, we are eager to hear your thoughts on our response. We hope our additional details and experimental results have addressed your concerns. We kindly ask if you could provide your opinion on our rebuttal and adjust your score accordingly. If you have any additional concerns, please do not hesitate to share them with us.

Once again, we sincerely appreciate your thoughtful review and constructive feedback.

Best,

Authors
Thanks for the response, and most of my concerns are addressed. I suggest the authors to put some important details into main paper. I have raised my score.
Dear AC and all reviewers:

Thanks again for all the insightful comments and advice, which helped us improve the paper's quality and clarity.

The discussion phase has been on for several days and we are still waiting for the post-rebuttal responses.

We would love to convince you of the merits of the paper. Please do not hesitate to let us know if there are any additional experiments or clarification that we can offer to make the paper better. We appreciate your comments and advice.

Best,

Authors
Dear Authors,

Thanks for the response! My concerns were addressed in a proper way. 

I think this paper is worthwhile to be accepted.

Best regards,

Reviewer YmnU