From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries
Authors: Joy Hsu
Emily Jin
Jiajun Wu
Niloy Mitra
Venue: NeurIPS 2025 poster ()
Keywords: factorized scene generation
program structure
object poses
Primary area: applications

Abstract: Real-world scenes, such as those in ScanNet, are difficult to capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a framework that synthesizes realistic 3D scenes by leveraging the underlying structure of rooms while learning the variation of object poses from lived-in scenes. We introduce a factored representation that decomposes scenes into hierarchically organized concepts of room programs and object poses. To encode structure, FactoredScenes learns a library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regularized by the learned library. To represent scene variations, FactoredScenes learns a program-conditioned model to hierarchically predict object poses, and retrieves and places 3D objects in a scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.

From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries
Joy Hsu
Emily Jin
Department of Computer Science Stanford University joycj@stanfore edu
Department of Computer Science
Stanford University
ord. edu
Jiajun Wu
Niloy J. Mitra
Department of Computer Science Stanford University jiajunsuccs, stanford.cccommo edu
Department Computer Science University College London n mmiirracs....mmmmmmmmmmmmmmmmc.. ucl ac uk
Abstract
Real-world scenes, such as those in ScanNet, are difficult capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a frame- work that synthesizes realistic 3D scenes by leveraging the underlying structure of rooms while learning the variation of object poses from lived-in scenes. We intro- duce factored representation that decomposes scenes into hierarchically organized concepts of room programs and object poses. To encode structure, FactoredScenes learns library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regu- larized by the learned library. To represent scene variations, FactoredScenes learns a moogram-conditione model to hierarchically predict object poses, and retrieves and places 3D objects in scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.
Introduction
Real-world scenes are inherently noisy, varied, and lived- in. For instance, chairs often arranged based on how people interact within a room, and monitors may be angled to face specific seating arrangements. Capturing these nu- anced rooms with noisy object poses remains challenging hence, high-qualit, 3D scene datasets such as ScanNet [1] are still scarce. How can we learn to generate such realistic scenes from limited data?
Progamm gemention
eoffiiee pubite exell (200, 531
and
perdiction
Our key insight that, despite inherent noisiness, indoor scenes retain significant underlying structure based on how rooms were intentionally designed, following social norms and preferences. Chairs are grouped around tables. and coffee tables are positioned by couches. We propose to leverage this (hidden) structure by first generating pro- grammatic layouts that align with the foundational design of rooms, then modeling the realistic variation of lived-in scenes through pose prediction model for orienting ob-
Factored scene representation
Figure 1: We propose FactoredScenes, a framework that generates layout pro- grams which encode underlying room structure, and predicts objects poses that capture nuanced variation real-world, lived-in scenes.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

jects in the room (See Figure layouts and object poses.
1. Our goal is to synthesize ScanNet-like data with diverse room
To this end, we introduce FactoredScenes, a framework that uses a factored representation to represent scene (See Figure We decompose complex scene generation into five steps: (i) learn a library of programs that capture room structures, (ii) generate scene program using large language models and the learned library, (iii) execute the program retrieve axis-aligned layouts, (iv) predict object poses with program-conditione model, and (v) retrieve object instances based on program structure and predicted dimensions. Notably, such decomposition of room into hierarchical concepts eliminates the need to directly generate room sampled from the full scene distribution, learned solely from ScanNet data. Instead, this approach enables FactoredScenes leverage different data sources and methods to model different components of scenes' structure- effeeciveel bootstrapping learning of the full scene distribution despite limited real-world data. Importantly, this modular design made possible due to the appropriate levels of abstraction that enable the interface between each component. Semantic knowledge from L.LMs distilled into programs, regularized by learned libraries, which operate on pext-parameterized objects, with numeric values predicted by neural networks.
Our framework first learns a space of programs that can generate room layouts from 3D-Front 21. large-scale dataset of synthetic indoor scenes that are professionally designed. We use this dataset learn a library of reusable programs that captures room structure patterns. FactoredScenes then leverages the generalization capabilities of large language models to create diverse new layouts, guided by our learned library. We demonstrate that this library learning step essential in capturing structural relationships between objects in rooms, opposed to relying solely on an inference-based library that has never seen example scenes.
Given the program and generated layout, FactoredScenes learns to predict object poses, using a much smaller ScanNet [I] dataset. Here, the layout program serves as a form of regularization, enabling effective learning from (very) limited real-world data. FactoredScenes' 's object pose model orients bounding boxes hierarchically given this program. It first predicts poses of primary objects (e.g., table), and then predicts those of dependent objects (e.g., chairs grouped around the table) based on primary pose predictions. Finally, FactoredScenes retrieves object instances based their predicted dimensions, completing the full 3D scene.
FactoredScenes demonstrates significant improvements over prior work in generating realistic Scan- Net oriented layouts by FID and KID metrics. We also quantitatively evaluate our learned library's ability compress room structure compared to inference-only library, and see 644.1% relative improvement in function use. In addition, we evaluate our object pose model's performance, which shows 11 11.4% relative improvement in the prediction of dependent object poses. Finally, we conduct a human study comparing FactoredSeenes's rooms to real ScanNet rooms, and demonstrate that our generated 3D scenes are difficult for humans to distinguish from real scenes. We believe that FactoredScenes is a step toward realistic, real-world scene generation from programs to poses.
In summary, key contributions the following:
We propose a library-learning approach to capture the underlying programmatic structure of rooms from 3D-Front.
We introduce program- coggaam-onditioned model for object pose prediction, leveraging hierarchical dependencies between objects and training limited ScanNet data.
We validate that FactoredScenes significantly improves upon prior work in generating realistic, oriented layouts.
We show through human studies that our generated scenes are difficult to distinguish from real ScanNet scenes.
2
Related Works
Indoor scene synthesis. A plethora of prior works have been proposed for 3D scene generation
3.
with approaches ranging from employing 2D generation models then projecting images to
3D
5 7) leveraging VAE and GAN architectures with priors [8 9 TO
[1 12], and using
diffusion- backbones in hierarchical and compositional manner also ingest wide range of input: from using layout as input in 2D image form and scene graph
13
14 15 These works

---

P(library)
Piprugram librarry)
P(layout - (prugramm)
Pipuse layout, progrum) Pg ... pose, program)
Larted Heary
cabla_1
parallo]
thaice_sioe
alige
cheice c2ussae www.ppaaa.seeee...o (ta offents.
thaice si2et
grid
[hate_1 chairolt)
grid K2E3
chasc_3 coxten(2)
2 chstea(2]
bookebalf_
Surcituce
100, 24301
cluster
Furnituuee2200, 220, 1350,
1amp_3
Tornitoree: 526, 1820, 1598, 16500
Figure 2: The FactoredScenes framework. FactoredScenes (i) learns a library programs, (ii) gen- erates a scene program with the learned library, (ii)) executes the program to retrieve layouts, (iv) predicts object poses given the program, and (v) retrieves object instances for the full scene. This factorization enables our framework to use different sources of data to generate real-world scenes.
form 10 21 22 23
16
18
19
20
using text prompts input processed by large language models
7
In this work. we primarily focus the unconditional scene synthesis task. We highlight four state-of-the-an art methods spanning various architectures and designs. ATISS (24) is an autoregressive transformer that predicts plausible room layouts unordered sets of objects. DiffuScene (25) is diffusion network that synthesizes 3D indoor scenes by denoising a set of unordered object attributes. Sync2Gen (66 is variational auto-encoder that learns a latent space of object arrangements. Layou [27] is a large language model-based planner that generates realistic layouts. Similar to these works, our method first predicts scene layout and then replaces objects with meshes or point clouds from a set of assets. However, in contrast to these prior works, we focus on the task of real-world unconditional scene synthesis, with lived-in scenes. Instead of parameterizing the scene as a collection of uxis-aligned labeled bounding boxes, we model each object pose-with not only its bounding box coordinates and size but also its orientation.
Program-based generation. Our framework leverages programs one part of its factored scene representation. Prior works such as Holodeck (1 have also proposed modeling the scene as a program, with similar LLM generation of the scene, but across specific modules designed for floorplan, doorway, objects, and layout. Aguina-Kamn et al. [22) take a similar high-level approach, with Cromain-specifee language ingested by the LLM produce layouts. Other works have also proposed leveraging large language models for scene generation, with templates designed to retrieve structured representations, language facilitate multi-round interactions
28
29
and
programs and constraints used in various 2D and 3D modeling tasks (30 31 27
32
33
34
35
Notably, these methods commonly operate pre-defined languages, which require domain-speciff knowledge priori and cannot flexibly generalize to potential new scene types. They do not see examples of real scenes learn this library from input dataset. In contrast, FactoredScenes conducts library learning on a large-scale dataset to understand underlying room structure [3). We show in experiments that our learned program library significantly improves upon an inference- only library in representing scenes compactly. While works such as ShapeCoder (37 follows a similar paradigm, it focuses training recognition networks for parsing input shapes; in contrast, FactoredScenes uses text-based representations and L.MSs to propose and parse named abstractions for downstream generation. The library learning of functions via LLMs enables LLMs determine the suitable level of abstraction for more precise generation.
3 Method
FactoredScenes models rooms as the output of a factorized generative model. We denote scene S as containing objects [x1, .2....., where each object x is represented as an oriented point cloud or mesh. To generate new scene, want sample S ~ P(S). However, learning P( intractable due to limited woal-world data (e.g.. ScanNet). Instead, we approximate P(S) with a structured factorization into conditional probabilities (See Figure 2). enabling efficient learning of scene structures and the use of a variety of data sources (e.g.. LLMs, synthetic layouts, real scans).
Concretely, decompose the scene generation process with objects [21, ... into the following:

---

P(11.22....
P( pose. program) P(pose
layout, program)
P(layout (program)) P(program
library) (atary))PPiibaaaaa P(library).
With this decomposition, generate real-world scenes in five steps, from programs poses:
learning a library of programs that capture room structures to model P(library);
(ii) generating scene program with LLMs regularized by the learned library to model Pprogrrm library);
(iii)
executing the program to retrieve layouts to model P(layout fprogram))
(iv) predicting object poses with a program program-conditinn conditioned model to model P(pose layout, program);
(v) retrieving object instances based on predicted poses and program structures to model P 22.-" pose, program).
This factorization enables FactoredScenes to learn from different data sources (e.g.. library learning of programs from large-scale synthetic data and object pose model training on real-world ScanNet), well as appropriate methods for modeling different components (c.g., LLM generation to generalize to new programs and neural networks to predict precise numeric values). FactoredScenes uses LLM-generated programs to determine object locations via commonsense knowledge, and leaves more complex numeric calculations for the pose model trained on real-world orientation variations. Each module leverages its strengths and data available. We describe how learn each component in the sections below.
3.1 Library Learning 3D-Front
FactoredScenes does not require hand- designed domain-specific functions for generation, but instead learns functions through library learning. Its learned li- brary contains functions that capture the structure underlying rooms and specify the possible high-level relationships be- tween objects in scene; for example, the cluster puster_ppaceee pplaceeent function for group- ing objects like chairs around a table. The functions represent reusable layout patterns from which rooms are drawn, which can be used to generate the library as a discrete uniform distribu- tion over available functions in L, where Pliirarry) = Uniform(L))
Wake stage: Parse and reconstrict
Program represeniation
nighisstands 825 (righh stand 400, 11
Python Exccution
LLM
Sleep stage: Propose
Learned library
nightabbanda allggn (hoght
Function proposal
pacaile.
acand J. 2,
LLM
olign clarter placeeent
400, 13
Figure 3: FactoredScenes employs an alternating wake- sleep formulation; in the wake stage, an LLM generates the underlying program of a layout, and in the sleep stage, an LLM proposes new abstractions for the library given successful reconstructions.
scenes. We model
Importantly, this library can be learned from synthetic data of professionally designed scenes, without requiring real-world examples, or notably, the need to process complex scenes. Hence, we learn our program library 3D-Front [2) a large-scale dataset of synthetic indoor rooms with axis-aligned objects. The dataset consists of axis-aligned bounding boxes scene (e.g., a layout) and their semantic object labels.
We employ wake-sleep framework for library learning based on the DreamCoder formulation (36). alternating between generating possible underlying programs and proposing new abstraction functionss for the library. Importantly, we propose using large language model (LLM) a text-based reasoning model, leveraging the natural parameterization of object bounding boxes into semantic text forms (See Figure 33. In the wake stage, we use an LLM recognition model predict the underlying program given set of bounding boxes and library of functions. We bootstrap the library with just two functions: object instantiations and the parallel function. The LLM generates the function implementations in Python, and we directly execute the programs as the generation model to output the layout. We then verify whether the predicted programs correctly reconstruct the input bounding boxes, and use it to correct the LLM recognition model and retrieve successful programs.

---

objects
co 1820
Deperddent
Predicted object poses
Maatha 700
walls
Based Model
you
pin
beages: 660
Jabel Ere
Dependent objects
Lotect stait
1000
Eac
Closest Enc
Wall Enc
(bb
Ene
Malk-
heteeht 80
X22 1500
1500
Attemion-
1300
1000
Uqp dije.
wiatho
Text
Enc
will: [_
Theta Enc
cluated placesent.
deponaion dbjocts
Figure 4: FactoredScenes hierarchically predicts poses for objects based the underlying program. Our model first predicts primary objects' poses with an attention based model (e.g., table), then predicts dependent objects' poses additionally conditioned the orientation and program of their dependency target (e.g., the chairs' poses are dependent on that of table's) s).
In the sleep stage, given successful programs, we use an LLM as the abstraction proposal model to propose new functions, including their signatures and implementations. We conduct this wake-sleep paradigm iteratively and refine our library for program generation. FactoredScenes discovers the following functions: align (for aligning objects like bookshelves in row), grid (for creating grid of objects like chairs), grid_with ttthoffetttttt (for creating a messy grid of objects), symmetrin (for placing objects symmetrically around central point), and cluster canssterpplacmmeet (for grouping objects like stools around table). We describe full function signatures for our library in the Appendix.
In this library learning process, we retrieve an abstraction library that captures the underlying structure rooms, and validate our LLM-based method for parsing the imput layout of bounding boxes into programs. We show in ablations that this learned library significantly outperforms an inference-ons LLM library, which has never seen example scenes. Notably, compared prior works that also use LLMs for generation, FactoredScenes's 's use of LLMs for library learning enables the LLM itself to propose functions an appropriate level of abstraction for precise generation.
3.2 Program Generation via an LLM
Next, we sample P(program library) way of LLM that generates scene programs, conditioned on the learned library and parsed program examples from ScanNet. At a high-level, we distill semantic knowledge in LLMs into program samples, regularized by our library. Concretely, we first parse complex layouts from ScanNet into programs via our library learning framework. We find that the LLM (here, GPT-o1 388 can parse layouts into compressed programs that accurately reconstruct the input ScanNet bounding boxes. With these few-shot example programs sampled from ScanNet, the LLM generates new, diverse scene programs.
FactoredScenes then executes the I.LM-generated program to retrieve predicted layouts. This execu- tion gives P(layout (program)), where deterministic Python code interpreter generates layouts consisting axis-oriented bounding boxes parameterized by their coordinates. Importantly, using an LLM for program generation enables FactoredScenes generalize to novel scene constructions, while regularizing programs with the space of learned functions and parsed examples enables generation of realistic scenes that follow commonsense rules and structures.
3.3 Object Pose Prediction on ScanNet
Given the predicted program and layout of axis-aligned bounding boxes, FactoredScenes learns model for P(pose layout, program). Here, the pose defined by the orientation for each object in the scene. By predicting oriented bounding boxes, instead of axis-aligned in most prior work, FactoredScenes able to capture realistic variation real, lived-in scenes.
We train our object pose model on ScanNer, a dataset with 707 unique scenes, each with annotated segmented objects. From these rooms, we extract axis-aligned bounding boxes based on the objects' boundaries. To retrieve oriented bounding boxes ground truth labels for the orientation 0
5

---

enumerate through 180 degrees of rotation and find the tightest bounding box. Through this process, we generate small set of real-workd examples consisting of axis-aligned bounding boxes and their transformed object pose. For each ScanNet scene, we also retrieve the underlying program via the wake stage of our library learning framework as described.
As there is limited data in ScanNet, we propose leveraging the program structure directly in the forward pass of FactoredScenes's 's object pose model (See Figure I. At a high level, FactoredScenes hicrarchically predicts object poses for specific objects based on the program context. FactoredScenes first separates objects as primary or dependent based on the program. Primary objects are those that are directly initialized by global coordinates, while dependent objects are defined by relative relations dependency target, a previously existing object. through functions in the learned program library. As an example, a table may be primary object and dependency target that several dependent chair objects may be clustered around. To capture the hierarchical dependency that people move chairs to orient around tables, FactoredScenes first predicts the table's pose, then uses its orientation as input to predict the chairs' poses. All dependent objects' pose predictions are conditioned upon their dependency target as specified by the program.
More concretely, our model predicts the orientation 0 of primary objects, by first encoding the input object label (e.g., table), its axis-aligned bounding box (as specified by Emins Mmmn maax Mmaz)) ) the closest wall (as specified by 21: V1, T2. 82 representing two endpoints as well as orientation). and all walls in the scene. These embeddings are fused and processed jointly with multi-head self-attention over object slots, then used to predict 9. For dependent objects, in addition to the prior elements, the model also encodes the predicted orientation of the object's s dependency target, as well as text embeddings of the program function that instantiated the object (e.g., chair__ _1 cluster cwwster_ppacemeeeeeeeeeeeeeeeeee placement (table_1, offsets, (90, )), then passes the result through the same attention stack to predict object poses.
We train FactoredScenes to predict the orientation 0, by treating angles modulo 180 and discretized into 36 classes (5 bins). The loss is cross-entropy over objects, summed over the independent and dependent stages:
K Σ 1
cindep C Lg
+ L
L
[Stage
Viy log Uij.k:
where K is the number of orientation classes, Dij.& is the predicted probability for object j in scene belonging to class ki and V.j.& is the one-hot ground truth label. During training, we upsample scenes where the percentage of difficult-to- -reediccttt orientations is high. At inference, we let Og 5 arg maxa Dej.a- With FactoredScenes's 's pose prediction model, we can generate oriented bounding boxes for each object in given layouts. Notably, our model uses programs as regularization, ensuring that the model generalizes well even when trained on a limited ScanNet dataset. In the Appendix, we include experiments adapting FactoredScenes to infer a pose distribution.
3.4 Object Retrieval of ScanNet Objects
we retrieve specific 3D instances based on object bounding boxes. Concretely, object retrieval is done by populating the scene with 3D objects whose class and dimensions are the nearest neighbor to the predicted oriented bounding boxes. The objects are then scaled, translated, and rotated to the full predicted poses. To match ScanNet scenes, we manually annotate ScanNet objects with facing directions, such that FactoredScenes's 's pose model can orient the object accordingly. The set of objects 22,.. form the final scene
Finally, we sample P(x1, 22,-" pose, program). Following prior works 24
In the object retrieval stage, we process the predicted orientation to set a facing direction, determined by the scene's 's underlying program. Each object is set face away from its automatically extracted region boundaries. For primary objects, the region boundaries are taken as those of the full room, containing all objects. For dependent objects, the region boundary defined as the tight bounding box enclosing the dependency target together with all dependents that share that target (e.g., cluster of chairs around table). Notably, this step affects only the facing direction used at retrieval time, and does not change the predicted orientation axis. Overall, our factored design keeps retrieval agnostic to object representation, enabling meshes or point clouds to be flexibly substituted while yielding useful object-centric scenes.

---

Table 1: Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) comparison of
FactoredScenes to prior works and variants of our framework on matching real, ScanNet layouts. The full FactoredScenes framework significantly outperforms all methods.
Bedroom FID
Living FID
Bedroom KID +
Living KID $
DiffuScene
135.57
186.54
0.130
0.177
Syne2Gen
26
126.67
139.30
0.127
0.117
ATISS
24
120.27
176.95
0.117
0,166
LayoutGPT
27
109.40
157.69
0.102
0.142
FactoredScenes wo/ poses
101.55
137.66
0.085
0,106
FactoredScenes w/ sampled poses
110.03
123.64
0,086
0.079
FactoredScenes (Ours)
67.51
83.49
0.020
0.024
Experiments
Our goal to generate real-world scenes akin to ScanNet, from programs to poses. Here, we evaluate the full FactoredScenes framework in Section its program library learning component in Section 4.2 and its pose prediction component Section 4.3 We discuss limitations and future directions Section 4.4
4.1 FactoredScenes Evaluation
Comparison to prior work. We evaluate FactoredSeenes's ability to generate realistic ScanNet-like layouts compared to prior state-of-the-art methods: ATISS (24). DiffuScene (25), Sync2Gen (26). and LayoutGPT [27]. We compare a set of 100 layout images from FactoredScenes and prior work to those of ScanNet. Each layout consists of a of bounding boxes uniquely colored by their object category. For fairness, we choose set of intersecting categories between all works and reduce the of objects ScanNet accordingly. The final categories are follows: armchair, bed, bookshelf, cabinet, chair, coffee table, couch, desk, dresser, lamp, nightstand, shelf. stool, table. The legend and examples are shown in Figure[5 [ Following prior work, we evaluate the generated layout images with Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) 39 40 41 ScanNet scenes, we highlight KID as a more robust metric.
Due to limited
In Table L we see results of FactoredScenes compared to prior works on bedroom and living room scenes. FactoredScenes significantly outperforms all prior works in FID and KID. On bedrooms, framework shows a 38.3% FID improvement over top prior works, and 80.4% KID improvement. On living rooms, FactoredScenes yields a 40. 40.1% FID improvement and 79.5% KID improvement.
Ablations. Notably, in Table we study the im- portance FactoredScenes's object pose model- by comparing FactoredScenes out poses (with axis-aligned bounding boxes), and a variant with sampled poses drawn from a normal distribution of orientations specified by class. From the no-pose variant, we see that the layouts created using our learned library already outperform top prior works, as FactoredScenes learns inherent structures within rooms. Impor- tantly, on bedrooms, our sampled-poss variant performs worse than the no-pose variant, high- lighting the difficulty of the object pose prediction task; random perturbations fail to capture relation- ships between objects, and instead yield overlap- ping boxes with illogical orientations. Overall, the full FactoredScenes framework yields signif- icantly stronger results.
variant with-
SaeNN
Ours
LayonCirr
SynchGe
ATTSS
DiffiSeaee
Human study of real-world scene generation. To evaluate the quality of full 3D scenes, we con- duct human study via Prolific 42) to compare
Ammchair
Lamp
Shelf
Bookshelf
Figure 5: Our predicted layout better matches that of ScanNet scenes compared to prior works.
7

---

Figure 6: We render diverse examples of FactoredScenes's generated 3D scenes with annotated ScanNet (top row) and ShapeNet (bottom row) objects. Note that ScanNet objects are often partial, hence we include scaled and interpolated ScanNet background for visualization.
FactoredScenes's 's generated rooms to ScanNet rooms. We render 3D scenes in a top-down view, Participants were given two rendered rooms, a ScanNet scene and FactoredScenes scene, and asked: "Which of these two rooms is more realistic and resembles a real-world room?" The questions were randomly ordered and the answer choices shuffled. Out of 400 answers total (20 pairs of scenes and 20 participant answers for each scene), the mean accuracy of choosing the ScanNet scene is 0.67, indicating that FactoredScenes's generations
difficult distinguish from that of ScanNet.
Qualitative examples. In Figure 6 we present qualitative examples of FactoredScenes' rendered 3D scenes with both ScanNet and ShapeNet objects on the same generated oriented layouts. Our framework is flexible, hence We see that FactoredScenes generates diverse rooms with appropriate structure and varied object poses, all with realistic perturbations (e.g., the orientations of chairs in the second column follows that the table that they are dependent on). In the Appendix, we provide more visualizations and analyses of rendered 3D scenes.
easily swap in objects of any type and texture render scenes.
4.2 Learned Program Library Evaluation
Here, we evaluate the diversity of functions used and accuracy of compression with FactoredScenes's learned program library. on both 3D-Front, the dataset learned from, as well as ScanNet, the target dataset that it generalizes to. We measure diversity as average high-level functions used per program, and accuracy compression as mean intersection over union (mIoU) of reconstruction.
We compare the quality of our learned li- brary to that of inference-only library in which the LLM proposes the same amount of functions, without seeing exam- ples scenes but given ample context about the task. We parse 100 3D-Front and 150 ScanNet scenes into programs with both li- braries, and report results in Table 2 First, we see that our library learned from 3D- Front is able to accurately reconstruct Scan- Net scenes, showing that our functions are reusable in the context of real scenes, and that there are underlying regularities across human-designed rooms. Second, our learned library that has seen examples scenes significantly outperforms a naive LLM-only approach for 3D-Front and ScanNet, on both diversity and accuracy. Compared to inference-only library, our learned library shows 170.1% % relative improvement in function use 3D-Front, and 644.1% improvement in ScanNet. We note that ScanNet, lower high-level function use with the inference-only library yields high mloU, there is always a trivial solution to
Table 2: Comparison of function diversity and compres- sion accuracy between our library and an inference-only library that has never seen example scenes.
3D-Front
ScanNet
Funes
mIOU
Funcs
mIOU
Inf-only library
1.07
0.92
0.34
0.96
Learned library
2.89
0.96
2.53
0.98

---

Figure 7: Visualizations of FactoredScenes's pose predictions the unseen ScanNet test set. Orange and green boxes are the original primary and dependent objects respectively. Red and blue boxes are the predicted primary and dependent objects. Here, highlight the natural variation in orientations of the primary objects, and the corresponding learned changes in the dependent objects. For example, in the top left scene, the bottom left table is rotated, and its dependent chairs correspondingly.
fully reconstruct the input without any compression via functions. Despite this, our library achieves an appropriate trade-off, and significantly outperforms the inference-only library diversity and accuracy. the Appendix, we provide examples of ScanNet scenes parsed into compressed programs via our library, and include evaluation of the generative performance from learned library.
4.3 Pose Prediction Evaluation
FactoredScenes's object pose model yields mloU of 0.745 on our unseen ScanNet test set. We visualize test results in Figure 7 and provide more examples and analyses in the Appendix. In Table 3 we report ablation results of our pose prediction model. We compare the model against variant that does not condition depen- dent objects' pose predictions on the predictions of their dependent targets. We see that while the orientation accuracy in degrees of primary objects is similar between the two models, the variant's performance drops significantly for dependent objects.
Table 3: Comparison of orientation predictions
between FactoredScenes and variant that does not rely pose predictions of dependent targets.
Primary acc.
Dep. 0 acc.
Ours wo/ dep.
0.537
0.397
Ours
0.542
0.442
4.4 Discussion
FactoredScenes is framework for real-world scene generation, which decomposes modeling of programmatic structure and varied object poses. Our method consists of library learning from synthetic data, LLMs to generalize program structure, and programs to regularize pose predictions. FactoredScenes enables learning with limited real-world data to achieve semantically meaningful layouts. However, it is still limited by the L.M''s ability to consistently generate valid room programs. Similar prior LLM-based works, generated scenes occasionally contain unnatural object placements (c.g., aligned nightstands at the middle of the bed, instead of at the head). Additionally, FactoredScenes' 's pose prediction model is limited by the heuristies used generate ground-truth oriented bounding boxes for ScanNet. Due to partial objects in ScanNet, extracted orientations are at times inaccurate, yielding illogical overlapping objects. With more accurate labeled data and continued advances in LLMs, we expect FactoredScenes' performance to naturally scale. Its interpretable framework allows each stage to be improved and evaluated independently. In addition, promising future direction is to include humans in the loop generate new scenes programs, thus dynamically producing more task-specific data. This high-level scene editing could achieved with FactoredSeenes either via direct program modification or through targeted natural-language language prompts.

---

Conclusion
Real-world scenes are complex, varied, with limited available data. We propose FactoredScenes solution for real-world scene generation, by introducing a factored representation of rooms, decomposed into underlying layout programs and varied object poses. We learn library of functions on professionally-dsigeeee synthetic data, and train a hierarchical object pose model on limited real data. Quantitative experiments and human studies demonstrate that FactoredScenes is promising step toward synthesizing scenes that are difficult distinguish from real ones.
Acknowledgments and Disclosure of Funding
We thank Niladri Shekhar Dutt for providing valuable feedback and visualization guidance. This work is in part supported by the Stanford Institute for Human-Centered AI (HAT), AFOSR YIP FA9550-23-1-0122 ONR 100014-23-1-2355 22-1-2740, and NSF RI #22112558 JH is also supported by the Knight-Hennese, Fellowship and the NSF Graduate Research Fellowship. NM was partially supported by gifts from Adobe Research and the UCL Al Centre.
ONR YIP N00014-24-1-211 1-2117, ONR MURI N00014-
References
[1] Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: annhtt-nnnttated 3D Reconstructions of Indoor Scenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5828-5823 2017.
[2] Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et 3D-Front: Furnished Rooms with Layouts and Semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10933-10942, 2021.
[3] Daniel Ritchie, Kai Wang, and Yu-an Lin. Fast and Flexible Indoor Scene Synthesis via Deep Convolu- tional Generative Models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6182 6190, 2019.
[4] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Iman Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7909-7920, 2023.
[5] Hao Ouyang. Kathryn Heal, Stephen Lombardi, and Tiancheng Sun. Tex22lmmersion: Immersive Scene with 3D Gaussians. arXiv preprint arXiv:2312.09243 2023.
Generative
[6] Hieu T Nguyen, Yiwen Chen, Vikram Voleti, Varun Jampani, and Huaizu Jiang. HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Model. arXiv preprint arXiv:2406.200 20077, 2024.
[7] Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi, RealmDreamer: Text-driven Scene Generation with Inpainting and Depth Diffusion. arXiv preprint arXiv:2404.07199 2024.
[8] Ming-Jia Yang, Yu-Xiao Guo, Bin Zhou, and Xin Tong. Indoor Scene Generation from Collection of Semantie-Segmentee Depth Images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15203-15212, 2021.
[9] Zifan Shi, Yujun Shen, Jiapeng Zhu, Din-Yan Yeung, and Qifeng Chen. 3D-aware Indoor Scene Synthesis with Depth Priors. In European Conference on Computer Vision, pages 406-422. Springer, 2022.
[10] Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Xingguang Yan, Gordon Wetzstein, Leonidas Guibas, and Andrea Tagliasacchi. CC3D: Layout-conditonne Generation of Compositional 3D Scenes, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7171-7181, 2023. 2023.
[11] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. VAY Stochastic Scene Layout Generation from Label Set. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9995-9904, 2019.
[12] Pulak Purkait, Christopher Zach, and lan Reid. SG- VAE: Scene Grammar Variational Autoencoder to Generate New Indoor Scenes. In European Conference on Computer Vision, pages 155-171. Springer, 2020.
10

---

[13] Xiaoliang Ju, Zhaoyang Huang, Yijin Li, Guofeng Zhang, Yu Qiao, and Hongsheng Li. DifflnDSeene: Diffusion-based High- Quality 3D Indoor Scene Generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 4526-4535, 2024.
[14] Zhennan Wu, Yang Li, Han Yan, Taizhang Shang. Weixuan Sun, Senbo Wang, Ruikai Cui, Weizhe Liu, Hiroyuki Sato, Hongdong Li, et al. Blockfusion: Expandable 3D Scene Generation Using Latent Tri-plane Extrapolation. ACM Transactions on Gruphics (TOG), 43(4):1-17, 2024.
[15] Xuanchi Ren, Jiahui Huang. Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3D Generative Modeling Using Sparse Voxel Hierarchies, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4209-4219, 2024.
[16] Xinpeng Wang. Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor Scene Generation with Transformers. In 2021 International Conference on 3D Vision (3DV), pages 106 115. IEEE, 2021.
[17] Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico Tombari. Graph-to-3D- End-to-end Genera- tion and Manipulation of 3D Scenes Using Scene Graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16352-16361, 2021.
[18] Madhawa Vidanapathirana, Qirui Wu, Yasutaka Furukawa, Angel X Chang, and Manolis Savva. Plan2Scene: Converting Floorplans to 3D Scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10733-10742, 2021.
[19] Aditya Chattopadhyay. Xi Zhang, David Paul Wipf, Himanshu Arora, and René Vidal. Learning Graph Variational Autoencoders with Constraints and Structured Priors for Conditional Indoor 3D Scene Gen- eration. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 785-794, 2023.
[20] Guangyao Zhai, Evin Pinar Örnek, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graphs. Advances in Neural Information Processing Systems, 36, 2024.
[21] Yue Yang, Fan- Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language Guided Generation of 3D Embodied Al Environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16227-16237, 2024.
Fun-Yun
[22] Rio Kanina-Kang. Maxim Gumin, Do Heon Han, Stewart Morris. Seung Jean Yoo, Aditya Ganeshan, R Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, and Daniel Ritchic. Open-Univers Indoor Scene Gener- ation using LLM Program Synthesis and Uncurated Object Databases. arXiv preprint arXiv:2403.09673 2024.
[23] Liangchen Song, Liangliang Cao, Hongyu Xu, Kai Kang. Feng Tang, Junsong Yuan, and Yang Zhao. RoomDreamer: Text-driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture. arXiv preprint Xiv:2300 1337, 2023.
[24] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. ATISS: Autoregressive Transformers for Indoor Scene Synthesis. Advances Neural Information Process- ing Systems, 34: 12011-12066. 2021.
[25] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias NieBner. DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis. Proceedings of the IEEF/CVF conference on computer vision and pattern recogmition, pages 20507-20518, 2024.
[26] Haitao Yang, Zaiwei Zhang. Siming Yan, Haibin Huang, Chongyang Ma. Yi Zheng, Chandrajit Bajaj, and Qixing Huang. Scone Synthesis via Uncertainty-drive Attribute Synchronization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5630-560 56400056,00250 2021.
[27] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He. Sugato Basu, Xin Eric Wang, and William Yang Wang. LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. Advances in Neural Information Processing Systems, 36, 2024.
[28] Rao Fu, Zehao Wen, Zichen Liu, and Srinath Sridhar. AnyHome: Vocablacy Generation of Structured and Textured 3D Homes. In European Conference on Computer Vision, pages 52-70. Springer, 2025.
[29] Yiqi Lin, Hao Wu, Ruichen Wang. Haonan Lu, Xiaodong Lin, Hui Xiong, and Lin Wang. Towards gunded
Language- guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedback. arXiv preprint arXiv: 15808, 2023.
11

---

[30] Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual Programming for Step-by-Step Texl-to-Image Generation and Evaluation. Advances in Neural Information Processing Systems, 36, 2024.
[31] Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang. Zishan Qin, and Stephen Gould. 3D-GPT: Procedural 3D Modeling with Large Language Models. arXiv preprint arXiv:2310.1294 2023.
[32] Alexander Raistrick, Lingje Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, al. Infinigen indoors: Photorealistic Indoor Scenes using Procedural Generation. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21783 21794, 2024.
[33] Angel Chang, Mihail Eric, Manolis Savva, and Christopher Manning. SceneSeer: 3D Scene Design with Natural Language. arXiv preprini 1703 2017.
[34] Rui Ma. Akshay Gadi Patil. Matthew Fisher, Manyi Sören Pirk, Binh- Son Hua. Sai-K Sai-Kit Yeung. Xin Tong, Leonidas Guibas, and Hao Zhang. Language-Drite Driven Synthesis of 3D Scenes from Scene Databases. ACM Transactions Graphics (TOG), 37(6): 1-1::1111... 16, 2018.
[35] Gabrielle Littlefair, Niladri Shekhar Dutt, and Niloy Mitra. FlairGPT: Repurposing LLMs for Interior Designs. arXiv preprint 2511 2025.
[36] Kevin Ellis, Catherine Wong, Maxwell Nye. Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B Tenenbaum. DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning. In Proceedings of the 42nd acm sigplan international conference on programmning language design and implementation, pages 835-850, 2021.
[37] R Kenny Jones, Paul Guerrero, Niloy Mitra, and Daniel Ritchie. Shapecoder: Discovering Abstractions Visual Programs from Unstructured Primitives. ACM Transactions on Graphics (TOG), 42(4):1-17, 2023.
[38] OpenAl. Introducing OpenAl ol. https://openai..oomooo 2024.
[39] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-scale Update Rule Converge to Local Nash Equilibrium. Advunces in neural information processing systems, 30, 2017.
[40] Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. arXiv preprint 1801 40100, 2018.
[41] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On Aliased Resizing and Surprising Subtleties GAN Evaluation. In CVPR, 2022.
[42] Stefan Palan and Christian Schitter. Prolific, ac ac-A Subject Pool Online Experiments. Journal of behavioral and experimental finance, 17:22-27, 2018.
12

---

Supplementary for Factored Real-World Scene Generation via Learned Program Libraries
The appendix is organized as the following, In Appendix A we provide additional experiment results of FactoredScenes across different prompt variations, as well as performance of FactoredScenes on generating new room type of office. In Appendix B present function signatures and descriptions FactoredScenes's learned library. Appendix we include examples of parsed programs that reconstruct ScanNet scenes. In Appendix D we report additional predicted orientations on ScanNet and analyze failure cases, as well as add evaluation of adapting FactoredScenes to infer a distribution of poses. In Appendix(E] we visualize examples of generated 3D scenes from FactoredScenes. Finally, we describe details of our model implementation, human study, and broader impact.
in Appendix F.
A Additional Results
Here, add experiments that test the robustness and sensitivity of program generation different prompt variations, under different few-shot prompting configurations (5-shot, 3-shot, 1-shot). In Table FactoredScenes shows consistently strong performance across variants, demonstrating its robustness in generation.
Table 4: FactoredScenes under different few-shot prompting configurations.
Robustness
5-shot FID
3-shot FID
1-shot FID
Ours (bedroom)
64.86 4.36
67.51
59.83 95.74
67.25
Ours (living room)
91.26 6.76
83.49
94.55
While we focus on bedrooms and living rooms in the main text following prior works, we expect the distributions learned from ScanNet transfer well across many indoor scene types. To showcase performance within indoor domains, we include new results on the office category in Table 5 We follow the same evaluation protocol in the main text and compute FID and KID between generated layouts and ScanNet office scenes.
Table 5: FactoredScenes generation results on the room category of office.
Office FID
Office KID
Ours
88.50
0.032
13

---

B Program Library
From Figures
14
wc detail the function signatures for all functions in FactoredScenes's learned
library.
furniture function
class furniture:
def
-- miniit.... (self, x_min, I_min, x_max, y_max), seli x_min x_min
self y_min y_min
self
self y manax y_max
Figure 8: The furniture function for primary objects.
parallel function
def parallel (obj (ob]) anchorr distance_apart, direction, Sarallel.objeet_elze--ee una
Place new furniture object parallel to an existing object based the
point specified distance. Optionally, apecify the size of the new object being placed.
obj_anchor: reference object to base the dietance_apart. distance between the two objects direction: (up), 2 (down), 3 (left),
object's position
(right)
objllle__ojjct.... size: optional tuple (width, height) to specify the size the object; defaults abj_anchor's 's size Returns: new furniture object positioned parallel to obj with the specified size
Figure 9: The parallel function for dependent objects.
align function
def align(obj_ree count, distance, direction): una
Create specified number of furniture objects aligned in given direction
with specified distance betreen them, based single reference object.
obj_ref: reference furniture object to be aligned count: aumber of objects to instantiate and align distance: distance between consecutive objects direction: (up), 2 (down), 3 (left), (right) Returns: list of aligned furniture objects
Figure 10: The align function for dependent objects.
14

---

grid function
(oid_rrr rows, cols, h_distance, v_distance);
Create a grid of furniture objecta based on a single reference object, with respect to its center.
The grid has specified rows and columns, with horizontal and vertical distances betueen objects.
The grid grows in both directions relative to the center of the obj_ref.
obj_ref: reference furniture object to generate the grid number of
in the grid cols: number of columns in the grid h_distance: horizontal distance between objects in the grid v_distance: vertical distance between objecta in the grid Returns: list furniture objects arranged in a grid
Figure 11:
The grid function for dependent objects.
wrid_with_at offeet function
def
grid_ grid_vith wit__titthheseeeeeeeeeeee offset (obj_ref, ref, rows, cols, h_distance, v_distance, sov_offset=-Nons col_offaet=NNoe))
Create a grid of furniture objects with optional rox and column
offsets,
with respect to the center of the reference object (obj_ref).
obj_ref: reference object to determine the grid's location and size rous: number rous in the grid
cols: number of columns in the grid
h_distance: horizontal distance betseen objects in the grid v_distance: vertical distance between objects in the grid rox_offsets: list of offsets for specific rows (optional) col.offsets: list of offsets for specific columns (optional) Returns: list of furniture objects arranged in grid with offsets relative to obj_ref's 's center
una
Figure 12: The cerid_with_offeee offset function for dep. objects.
symmetrical cal function
symaetrical (center,, distance_X, distance_y, paamatrriall.ojects.ddtt una
Place four objects symmetrically around central point, based the specified object size.
The placement is based the center of each symmetrical object.
center: (x, y) coordinates
central point
distance_x: horizontal distance from the center to the nev objects distance_y: vertical distance from the center to the new objects caanetrical.obecte_stt.
tuple (width, height) specifying the size
of the symmetrical objects
Returns: list of four furniture objects symmetrically placed around the center
una
Figure 13: The symmetrical function for dependent objects.
15

---

cluster pllcceeent function
def
(obj_ center, offsets, claterrd,
:
Place cluster furniture objects around central object based specified offsets.
The offeets are with respect the the obj_center center coordinates. Optionally, specify the size of the clustered objects.
obj_center: central furniture object used the anchor point offsets: list of (x_offaet, y_offset) tuples for placing
surrounding objects
size: optional tuple (vidth, height) specify the size the clustered objects;
defaults the size of the central object Returns: a list of furniture objects placed around the central object
Figure 14: The cluster, placement function for dep. objects.
16

---

C ScanNet Program Parsing
In Figure 13
we include examples of parsed programs that reconstruct ScanNet scenes. We highlight the apt uses of high-level functions, for instance, in the top example, the parallel functions for beds, desks, dressers, etc. In our experiments, we note that GPT-ol is exceedingly proficient at parsing input bounding boxes in text form into programs with our learned library, and has strong potential for extracting program-based representation from structured data.
bod Curiiluzzei (nio-37, 9.0221941,
shelf_ fucnituree 6a xine33, xixmm223, rax-582, xaX-2431
3ax-59t, y (5x62754)
bec _ pacalleic
shelt_: pacailei;
dbi endier-bedd_ diatanow wba.2m1144.00
001 Lanchor=abell 2, dia lance upert=1158:5,
dtrecttonnd, passlin. parslia._ctjjct, sespeett_tttttteee (562, 8131
directione,,
parallei_ catpe__jjeeee__eeeeeeee piizen1b4,, 1323
furnltureri ninedo, y nin-186, s.xaeett,, y.3028675)
radiator furnituceti nisw22,, y min-162,
v-raxe1313, y raxee225
recyolingg
rurntture v_auae336)
y_stne33r,
ScanNet scene
sucyeling pazallei
dizectioond,
passlie. objectt sizen (361, 434)
distance spart_22tttt
directions4,
dceases_ funiiturela zine221, xin
patsilul sujjsttjjeeaaeee pise--111, 152)
dreszen 1 paralisil abt moserdraeeeee 2, distance aparc=1315.0,
trach_cn can 2 farntturaiy, nto-t0s,, y_nine206,
K_ax--109,, (a
crash can 1 parallel
direationed, pazelie. cbjact, sizen (349, 257)
dn anchorstrach can 2,
diatance upart=53.0, disectioned,
office chuis furniturnte bn 21s, y ninse2)),
patailel sbjject pisuetllé, 104)
i ciaster_piacanee placement
bookesiit_
ransturri
(116ets=/:1303.00 155,671,
Reconstructed scene
clustered objests steen (100, 192)
Underlying program
1:01
pillows clnter, placeress (occh, 110% -i0 LM,, 1611
24B3 60-211
CALEA Euratas DLRGHIR,, L L 1200
paasaa contee, puacenars (atoe,), 118614 sith, cal, 2300
Fam mowitt s013 c.reeda (
ScanNet scene
effaees
chile
chis A1ann12)
Jemese scrett siedt aweass sae-d20)
Adites Farm steet seette 2 meentan
jofeert rested, treestil)
b. jorferr (rees
190 Eatratin be acoeett 3u 2u pa
1aap mecutastt mie- ale-tat aasttte,, y sis
Reconstructed scene
Underlying program
Figure 15: Examples of programs underlying complex ScanNet scenes, parsed with LLMs using FactoredScenes's 's learned library.
17

---

D Predicted Object Poses
In Figure 16 we present additional examples of FactoredScenes's predicted object poses unseen ScanNet test set. We highlight several failure cases the last row. In particular, the left scenes in the bottom row have infeasible rotations (e.g., kitchen cabinet and sink), while the right two scenes in the bottom row have no dependent objects, leading to less logically dependent poses. We believe that a combination of more accurately labeled data and a larger quantity of data would improve PactoredScenes s object pose model significantly.
Figure 16: Additional visualizations of FactoredScenes's pose predictions on the unseen ScanNet test set. Orange and green boxes are the original primary and dependent objects respectively. Red and blue boxes are the predicted primary and dependent objects.
18

---

We additionally report results of adapting FactoredScenes to infer a distribution of poses rather than single pose for given layout, trained with negative log likelihood (NLL). In Table we report NLL on the ScanNet test set, and see that our trained model significantly improves on both random (untrained) baseline and an informed fixed mean and variance baseline. Table 7 we report point estimate metric by taking the mean of the predicted distribution. Our generative model outperforms all top prior works (Sync2Gen for living rooms and LayoutGPT for bedrooms), though less than our predictive model. We note that in this framework, while the means and standard deviations of dependent objects are conditioned on that of dependency target, the sampling is conducted independently. Hence, it is unable to robustly align dependent objects' orientations correspondingly.
Table 6: NLL comparison of our pose model to baselines.
ScanNet Test NLL +
Untrained model baseline
1.795 1018
Fixed mean & variance baseline
2.279 x 106
Generative FactoredScenes
1.587 10
Table Comparison using mean-based point estimates.
Living room FID +
Bedroom FID |
Living room KID &
Bedroom KID
Top prior Generative
139.30
109.40
0.117
0.102
113.06
99.95
0.067
0.071
19

---

E
Generated Scenes
Figure 17: Additional examples of FactoredScenes's 's generated rooms.
We present additional examples and analyses of FactoredScenes's generations in Figure 17 as well showcase specific success and failure in the bottom row. In the green box of the left scene, we see that our framework correctly generates chairs of the same type oriented towards the table. Notably, this without a priori specifying any constraints around how chairs should be oriented around tables or that chairs around a table are usually of the same type. The predicted orientations and facing directions are natural. However, the red box, we see an example where not only are chairs of different types (as they are not accurately parsed in the program as objects dependent on the table), but are also faced in unusual ways (for similar reason, as the region boundaries here are the room walls).
20

---

Similarly, in the green box of the right scene, we see very natural setup, of couch facing a coffee table and stool across from with all objects oriented appropriately. But in the red box, we see chair facing towards the bed in an unrealistic pose.
Though FactoredScenes
accurately model detailed parts of scenes, such specific orientations of chairs, it occasionally errors in (i) generating valid programs with correct dependent objects and relations, and (ii) predicting reasonable object poses when objects are in unfamiliar positions. We believe these aspects can be improved with better-performing LLMs as well as human-annotated data for orientations instead of couristics-based labels.
21

---

Details
Model training. Here, we describe the FactoredScenes settings. The library learning component uses the 3D-Front (2) dataset, released under the CC BY-NC-SA 4.0 license; and the orientation model trained on the full ScanNet dataset M released under the MIT license. When training our pose prediction model, we used 581 scenes with accurately parsed programs out of 707 total unique scenes; our train set comprised 523 scenes and test set comprised 58 scenes. used the Adam optimizer with learning rate of 0.0001, and trained our models single Titan RTX GPU with 24 GB of memory.
Human experiments. We conducted human study via Prolific [42] to compare FactoredScenes's generated rooms to ScanNet rooms, with the following instruction: "In this study, you will be asked questions comparing two images of rooms: which of these two rooms is more realistic and resembles a real-world room?". The questions were randomly ordered
the answer choices shuffled. We queried 20 participants over 20 pairs of scenes each, with average compensation per participant of 25 USD per hour. We identified no potential risks for the study, as each image shown is rendering of 3D scene, and did not seek IRB approval, the task involved no personally identifiable information or sensitive content.
Broader impacts. Our work focuses generating realistic scenes, for the purpose of creating real-world, object-centric datasets that the community can build upon. While there are no direct routes to harm with model, acknowledge its potential misuse, such as in pipelines for generating fake content. In addition, although we use a large language model to produce layout programs regularized by our learned library, our framework remains susceptible biases inherent in such models. We encourage future work to explore bias mitigation strategies when deploying these systems.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state contributions in the abstract and introduction, and provide empirical evidence in the experiment section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See method details in Section S
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
We have released code to reproduce our work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See details in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars, both our method and certain baselines rely expensive API-based language models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the Code of Ethics and our work conforms to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See broader impacts in the Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See citations in the main paper and details in the Appendix.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
See details on our human study in the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
See details in the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
28

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We describe how LLMs used in our library learning and program generation process in Section 3
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
29


Reviews:
[{"review_id": "HIrk1DC5OF", "reviewer": "Reviewer_ME6t", "summary": "This paper proposes the FactoredScenes framework, which synthesizes realistic 3D real-world scenes by learning layout structures from synthetic data via library learning and predicting object poses with a program-conditioned model. The approach decomposes scene generation into hierarchical components—program generation, layout execution, pose prediction, and object retrieval—demonstrating significant improvements over state-of-the-art methods in generating ScanNet-like scenes, though it retains limitations in program logic and data processing.", "questions": "1. Does the LLM-driven library learning process explicitly differentiate between scene types (e.g., bedrooms vs. living rooms) when capturing structural patterns? Are there experiments validating this differentiation?\n2. Given the model is trained on only 707 ScanNet scenes, how is overfitting mitigated? \n3. The paper mentions \"commonsense rules\" for program generation, but how are these constraints formalized or quantified? Are there ablation experiments isolating their impact on scene realism?\n4. For non-axis-aligned objects, does the pose model consider functional geometric dependencies (e.g., monitor orientation relative to seating)? Current descriptions focus on hierarchical dependencies but lack clarity on geometric reasoning.\n5. Do the experimental comparisons include 2024–2025 advances in 3D scene generation (e.g., DiffInDScene, Blockfusion)? The baselines listed may not reflect the latest state of the art.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The factored representation decomposes scenes into structural programs and pose variations, leveraging synthetic 3D-Front data for library learning and LLMs for program generation.\n2. Experiments demonstrate substantial improvements.\n3. The paper is well-written, clearly articulating the problems to be solved and the problem-solving approach.\n\n**Weakness**\n1. The paper appears to overly rely on LLM programs for generation, which may lead to catastrophic generation results when unnatural object positions (such as nightstands in the middle of a bed) occur.\n2. The proposed method does not address the overlap issue in scenes.\n3. LLM inference requires high computational resources, and I am unsure whether the generative quality will be affected in other scenarios (e.g., offices).\n4. Although the paper claims to compare multiple methods, it lacks quantitative comparison experiments (such as KL, FID, SCA, etc.).", "was_revised": true, "final_justification": "The author seems to have addressed my concerns. However, since this submission was randomly assigned to me and it is not my area of expertise, despite my best efforts in reviewing, I still suggest that the AC take into account the opinions of other reviewers more. Thank you.", "rebuttal": "Thank you for your constructive feedback, which has greatly improved our paper. \n\n**Q: Quantitative comparisons.**\n\nWe clarify that our main quantitative evaluation includes FID and KID, reported in Table 1 of the main paper, across both bedroom and living room scenes. Thanks to your suggestion, we have now included KL divergence over object category distributions, following prior work (e.g., ATISS, DiffuScene). We see that FactoredScenes comfortably outperforms prior works on this metric across both scene types (lower is better), indicating stronger alignment with real-world category distributions.\n\n|            | Bedroom KL | Living room KL  | Bedroom FID | Living room FID | Bedroom KID | Living room KID |\n|------------|------------|------------|-------------|------------|-------------|------------|\n| DiffuScene | 7.861     | 6.257     | 135.57      | 186.54     | 0.130       | 0.177      |\n| Sync2Gen   | 8.052     | 6.402     | 126.67      | 139.30     | 0.127       | 0.117      |\n| ATISS      | 7.916     | 6.295     | 120.27      | 176.95     | 0.117       | 0.166      |\n| LayoutGPT  | 7.663     | 6.284     | 109.40      | 157.69     | 0.102       | 0.142      |\n| Ours       | **7.476** | **6.127** | **67.51**   | **83.49**  | **0.020**   | **0.024**  |\n\nWe also note that we provide a thorough quantitative analysis of each intermediate stage in our framework:\n1. Library learning: we measure the diversity of functions and accuracy of compression in Table 2, showing large improvements over an inference-only baseline. \n2. Program generation: we ablate our pipeline without the pose model to show the contribution of layout generation alone in Table 1. \n3. Program execution: as this step uses deterministic Python code, we assume exact execution. \n4. Pose prediction: we evaluate orientation predictions of primary and dependent objects in Table 3, showing that conditioning on program structure improves dependent objects’ predictions. \n5. Full scene generation: we evaluate via quantitative metrics in Table 1, as well as a human study in the main text. We will revise the main text to clearly highlight these evaluations and discuss their performance.\n\n**Q: LLM-based generation.**\n\nIndeed, FactoredScenes programs rely on LLMs’ commonsense knowledge as part of the generation process, as many state-of-the-art prior works do (e.g., LayoutGPT, Holodeck, LayoutVLM). However, unlike these prior works, we learn the underlying library representing structure in scenes, and constrain LLM generations via this learned library. We hypothesize that this learnt structural prior helps reduce unnatural generations by anchoring the generation space to abstractions learned from example rooms, which leads to the strong performance of the FactoredScenes wo/ poses variant in Table 1.\n\nWe agree that LLM-generated programs can occasionally yield unnatural scenes, which we highlight in the limitations section. Thanks to your suggestion, we have added experiments that test the validity of FactoredScenes’ LLM-generated programs, as well as the robustness of downstream results to potential errors in these programs. \n\nFirst, we report the percentage of FactoredScenes’s generated programs that execute successfully without errors and the percentage of programs that are semantically valid. As there is no ground truth for semantically valid programs, we define semantic validity using a conservative heuristic: a generated layout is deemed invalid if any pair of objects (excluding lamps and pillows) has IoU > 0.3. Here, we see that a high percentage of programs execute successfully, and even with this conservative, semantically valid criteria, we see a high percentage of programs marked as valid, which indicates that catastrophic errors are rare.\n\n| Across 100 rooms | % of program that executes successfully | % of programs that are semantically valid |\n|------------------|-----------------------------------------|-------------------------------------------------|\n| Ours             | 97%                                     | 90%                                             |\n\nSecond, we evaluate the effect of FactoredScenes’ program validity on downstream generation results, and report performance for scenes generated from valid vs. invalid programs. As shown below, semantically valid programs as per our heuristic produce lower (better) performance, although the difference is slight.\n\n| Generation | KID of semantically **valid** programs | KID of semantically **invalid** programs |\n|-----------------------|--------------------------------|------------------------------|\n| Ours                  | 0.0183                         | 0.0192                       |\n\nLastly, we investigate the robustness and sensitivity of our program generation to different prompt variations, under different few-shot prompting configurations (5-shot, 3-shot, 1-shot). FactoredScenes shows consistently strong performance across variants, significantly above that of prior works, demonstrating its robustness in generation.\n\n| Robustness        | All           | 5-shot FID | 3-shot FID | 1-shot FID |\n|-------------------|---------------|------------|------------|------------|\n| Ours bedroom      | 64.86 ± 4.36 | 67.51      | 59.83      | 67.25      |\n| Ours living room  | 91.26 ± 6.76 | 83.49      | 95.74      | 94.55      |\n\nWe thank you for the feedback and will incorporate these new analyses into the revised paper. \n\n**Q: Generation quality in other scenarios (e.g., offices).**\n\nAs suggested, here, we add additional results of FactoredScenes generating offices, a new room category. We follow the same evaluation protocol as in the main paper and compute FID and KID between generated layouts and ScanNet office scenes.\n\n|      | Office FID | Office KID |\n|------|------------|------------|\n| Ours | 88.50    | 0.032     |\n\n**Q: Modeling of geometric dependencies.**\n\nOur core hypothesis is that functional geometric relations stem from intentional semantic relations, and they can be captured via our framework. In cases like monitor orientation relative to seating, FactoredScene can encode such dependencies, as it arises from semantics (e.g., seat must face monitor, hence the generation should be dependent), which are then used by our pose model to predict geometry. However, we do not explicitly model purely geometric reasoning. We agree that incorporating these geometric constraints would be an interesting future direction. \n\n**Q: Differentiation of scene types in the library learning process.**\n\nFactoredScenes' LLM-driven library learning process does not explicitly differentiate between scene types, and captures underlying structure across all room designs in 3D-Front. This encourages the discovery of general, reusable structural abstractions that are applicable across diverse, open-world rooms.\n\nHowever, during generation, FactoredScenes does condition on the target scene type, which is passed as part of the prompt to the LLM, and hence the LLM chooses appropriate functions to use. We analyze the distribution of library function usage across room categories in the generated program, and see that, for example, the function parallel is used more in living rooms, while cluster_placement is used more in bedrooms. Such insights are valuable as they help us characterize structural differences between scene types.\n\n**Q: Comparison to DiffInDScene and Blockfusion.**\n\nDiffInDScene and Blockfusion focus on unconditional geometric generation, producing SDF volumes that are post-processed into 3D meshes (via marching cubes), without recovering semantic object-level layouts. To render scene visualizations, they use off-the-shelf tools to directly add texture to their scene geometry (e.g., DreamSpace, Meshy). We note that Blockfusion additionally allows users to specify a 2D layout as input, not output.\n\nIn contrast, FactoredScenes targets unconditional room generation via semantic scene layouts. We compare FactoredScenes against top representative works that solve this task, spanning different model classes: ATISS (autoregressive), DiffuScene (diffusion), Sync2Gen (VAE), and LayoutGPT (LLM-based). We will clarify this distinction in the main text and add additional discussion in the revised paper. \n\n**Q: Addressing the overlap issue.**\n\nCurrently, FactoredScenes does not explicitly address the overlap issue in scenes. Thanks to your suggestion, we follow prior work (e.g., LayoutVLM), and add a manual overlap checker in the final stage of our method, applied after pose predictions. Scenes with any pair of objects (excluding lamps and pillows) of IoU > 0.3 are filtered out. We see that this rejection sampling step improves layout quality across bedrooms and living rooms, and we will include this variant and its results in the revised paper. \n\n|                  | Bedroom KID | Living room KID |\n|------------------|-------------|------------|\n| Ours             | 0.020       | 0.024      |\n| Ours wo/ overlap | 0.016       | 0.023      |\n\n**Q: Overfitting mitigation.**\n\nWe highlight that we design FactoredScenes as a structured framework, precisely as fully end-to-end methods tend to overfit on small datasets like ScanNet. For the only component of FactoredScenes trained on ScanNet, the pose prediction model, we intentionally keep the architecture lightweight, limiting model capacity to reduce overfitting, as indeed, the primary constraint here is the availability of real-world datasets. \n\n**Q: Clarification of commonsense rules.**\n \nWe clarify that FactoredScenes does not define explicit commonsense rules. Instead, we use LLMs as a repository of implicit commonsense knowledge, from which we can generate programs when guided by our learned library. Given that LLMs are trained on extensive corpora of human data, we hypothesize that they contain commonsense knowledge about object placement in rooms for a wide range of room types. Thank you for the feedback. We will clarify this in the main text."}, {"review_id": "sipdbAAKYq", "reviewer": "Reviewer_ziVh", "summary": "This paper focuses on realistic 3D scene generation with varied object layouts and poses. A new framework, i.e., FactoredScenes, is proposed by leveraging the underlying room structure. A factored room representation is proposed to decompose scenes into hierarchically organized concepts of programs and object poses in lived-in scenes. The experimental results show the effectiveness of the proposed method for realistic room generation.", "questions": "1. Based on the results in Table 1, the results of the no-pose variant already outperform existing works. Why the proposed method can learn inherent structures better than existing works?\n2. Is it possible to be generalizable to other scenarios other than lived-in scenes?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The idea of decomposing the room into hierarchical concepts is interesting, making it possible to leverage different data sources for realistic scene generation. Semantic knowledge from LLMs is further distilled to capture structural relationships between objects in rooms. The results demonstrate the effectiveness of the proposed method.\n\nThe main weaknesses are listed as follows.\n1. The evaluations are not thorough. The proposed method consists of five steps for scene generation. The intermediate results from each step should be evaluated and discussed in the experiments. \n2. GPT-o1 is used in the current experiment, but the importance of GPT-o1 in the whole pipeline has not been studied yet. The ablation studies do not consider the effect of different LLM settings, including the parameters of GPT-o1 and replacing GPT-o1 with other methods.\n3. A detailed computational cost and running time comparison between the proposed method and existing works is necessary.", "was_revised": true, "final_justification": "I appreciate the authors' clarifications and additional comparisons. My main concerns have been addressed by the rebuttal. Although the proposed method is slower than baselines due to structured program execution and LLM querying, it still provides a new way to synthesize realistic 3D scenes with the discovery of reusable structural abstractions underlying scenes.\n\nI would lean to accept the paper by involving the additional evaluations and discussions in the revised version.", "rebuttal": "Thank you for your constructive feedback, which has greatly improved the quality of our paper. \n\n**Q: Effect of different LLM settings.**\n \nWe agree that it is valuable to explore the effect of different LLM settings. Based on your suggestion, we add experiments that test the robustness and sensitivity of our program generation to different prompt variations, under different few-shot prompting configurations (5-shot, 3-shot, 1-shot). FactoredScenes shows consistently strong performance across variants, significantly above that of prior works, demonstrating its robustness in generation.\n\n| Robustness        | All           | 5-shot FID | 3-shot FID | 1-shot FID |\n|-------------------|---------------|------------|------------|------------|\n| Ours bedroom      | 64.86 ± 4.36 | 67.51      | 59.83      | 67.25      |\n| Ours living room  | 91.26 ± 6.76 | 83.49      | 95.74      | 94.55      |\n\n**Q: Runtime comparisons.**\n \nThank you for the suggestion. We have now added a runtime and API usage comparison across methods, averaged over 10 generations on a single Titan RTX GPU. We see that FactoredScenes is slower than fully neural baselines like ATISS and Sync2Gen due to structured program execution and LLM querying, but remains comparable to DiffuScene and LayoutGPT. We will add this table and discussion to the revised paper.\n\n| On 1 TitanRTX | Generation time per layout  | API Calls |\n|---------------|-----------------------------|-----------|\n| DiffuScene    | 27.43s                    | No        |\n| Sync2Gen      | 0.10s                     | No        |\n| ATISS         | 0.41s                     | No        |\n| LayoutGPT     | 10.42s                    | Yes; o1   |\n| Ours          | 25.52s                    | Yes; o1   |\n\n**Q: Thorough evaluation of intermediate results and each stage of generation.**\n \nWe highlight that we provide a thorough quantitative analysis of each intermediate stage in our framework:\n1. Library learning: we measure the diversity of functions and accuracy of compression in Table 2, showing large improvements over an inference-only baseline. \n2. Program generation: we ablate our pipeline without the pose model to show the contribution of layout generation alone in Table 1. \n3. Program execution: as this step uses deterministic Python code, we assume exact execution. \n4. Pose prediction: we evaluate orientation predictions of primary and dependent objects in Table 3, showing that conditioning on program structure improves dependent objects’ predictions. \n5. Full scene generation: we evaluate via quantitative metrics in Table 1, as well as a human study in the main text.\n\nThank you for your suggestion, we will revise the main text to clearly highlight these intermediate evaluations and discuss their performance.\n\n**Q: Generalization to scenarios other than lived-in scenes.**\n \nYes, FactoredScenes can generalize to non-lived-in scenes (e.g., designed or staged rooms) with just our program generation step alone, as shown in the wo/ poses variant in Table 1. For generalization to entirely different domains (e.g., outdoor scenes), pose prediction would require retraining on new orientation data, and potentially our learned library would need to be augmented with new wake-sleep stages (without requiring handcrafted functions).\n\nHowever, we expect the distributions learned from ScanNet to transfer well across many indoor scene types. To showcase performance within indoor domains, we include new results on the office category. We follow the same evaluation protocol as in the main paper and compute FID and KID between generated layouts and ScanNet office scenes.\n\n|      | Office FID | Office KID |\n|------|------------|------------|\n| Ours | 88.50    | 0.032     |\n\n**Q: Ability to capture inherent structures compared to existing works.**\n \nCompared to existing LLM-based works which rely on handcrafted and fixed libraries, FactoredScenes learns its library from example rooms, enabling discovery of reusable, data-driven structural abstractions underlying scenes. FactoredScenes’ LLM generations are constrained via this learned library, and we hypothesize that this learnt structural prior helps reduce unnatural generations by anchoring the generation space to abstractions learned from example rooms.\n\nCompared to non-LLM baselines, FactoredScenes explicitly models discrete, hierarchical structure, which we hypothesize improves generalization to novel room layouts, as we see strong performance even in the no-pose setting. Thank you for your feedback, we will add these discussions in the revised text."}, {"review_id": "ZteIcBQV7R", "reviewer": "Reviewer_uVTa", "summary": "This paper introduces FactoredScenes, a framework for real-world unconditional scene synthesis with lived-in scenes, by decomposing the scene into programs encoding the layout and object poses capturing real-world variation. \n\nFirst, FactoredScenes learns a library of functions from a synthetic dataset through an iterative wake-sleep formulation using LLMs. Then, it generates new programs using LLMs conditioned on the learned library and parsed program samples from a real-world dataset. A neural network learns predicting the exact object poses hierarchically and finally the framework retrieves the object instances producing the 3D scene.", "questions": "All of my questions are listed in the weaknesses section, and I may adjust the rating if they are well addressed.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tReal-world unconditional scene synthesis with lived-in scenes is an interesting problem, as existing methods mostly employ synthetic datasets and attempt to produce similar scenes.\n2.\tLearning a library of reusable functions to capture structural relationships and generating scene-level programs based on this learned library is novel.\n3.\tDecomposing scene generation in several steps and hierarchical pose prediction based on object dependencies is intuitive.\n\nWeaknesses:\n\n1)\tThe authors claim their method generalizes to new scene types. However, all experiments are conducted only on the ScanNet dataset, which is also used to train the object pose estimation network. This raises concerns about dataset-specific biases in pose distributions and scene layouts that may not apply to other real-world indoor scenes. Further evaluations on different real-world datasets are required to support this claim.\n2)\tLines 168–169: The verification and correction mechanism is not clear. Is there a feedback loop that runs until the predicted programs accurately reconstruct the input bounding boxes for a given scene?\n3)\tThe success of library learning (Section 3.1) and program generation (Section 3.2) relies heavily on LLM prompting. However, the paper does not specify how to interact with the LLM during library learning or how these instructions are structured.\n4)\tThe paper performs the final evaluation on ScanNet. However, the baseline methods, whether using in-context learning or fully learning-based approaches, have only seen synthetic datasets and cannot capture real-world variations. To fairly evaluate scene-level program generation against these baselines, layouts should either also be generated and evaluated on 3D-FRONT (eliminating the real-world component) or the proposed pretrained pose estimation model should be applied to the layouts produced by the baseline models to produce ScanNet like scenes.", "was_revised": true, "final_justification": "Thank you for responding to my questions, the rebuttal addresses most of my concerns. I will keep my rating positive.", "rebuttal": "Thank you for your constructive feedback, which has greatly improved the quality of our paper. \n\n**Q: Additional evaluations on 3D-Front without the real-world component.**\n\nWe highlight that one of the main contributions of FactoredScenes is precisely that it enables training across diverse datasets, such as ScanNet (and hence generate realistic, lived-in scenes with object layouts), while prior works cannot, because they are limited to learning from axis-aligned layouts from datasets like 3D-Front. While not the focus of our paper, we add additional evaluations of FactoredScenes and prior works on 3D-Front, removing the real-world component. We see that FactoredScenes outperforms prior works on bedrooms, while achieving competitive performance to prior works on living rooms, despite not being designed to optimize for this task. Thank you for your suggestion, and we will include this evaluation in the revised paper and clarify our focus.\n\n| 3D-Front | Bedroom KID | Living KID |\n|--------------------|-------------|------------|\n| DiffuScene         | 0.036      | 0.088     |\n| Sync2Gen           | 0.010      | **0.030**     |\n| ATISS              | 0.052      | 0.093     |\n| LayoutGPT          | 0.030      | 0.038     |\n| Ours               | **0.009**      | 0.031     |\n\n**Q: Prompts to LLMs and their robustness.**\n\nWe provide the prompts used for both library learning and program generation below, and will include this in the revised paper. \n\nLibrary learning:\n\n*You have the following functions: [Current set of functions in the library]. Can you parse the below bounding boxes into programs with the above functions, such that they can be accurately reconstructed? The bounding boxes are defined by name and corresponding coordinates x_min, y_min, x_max, y_max representing the bottom left and top right corners of the bounding box. Please parse the bounding boxes into programs such that we only need to define the coordinates of minimal furniture only. The rest of the objects should be defined as so without coordinates and named (e.g., object_x = function(...)). Return code to generate the objects in a dictionary named objects, where the key is the original name of the object, and the value is the newly created object by the program. Bounding boxes: [Input bounding boxes].*\n\n*The following are programs describing layouts with bounding boxes, each program separated by ---. The current functions used are as follows: [Current set of functions in the library]. Can you propose new functions to compress these programs while retaining the same bounding box reconstructions? Compress programs by minimizing the number of objects that need explicit coordinate definition. Please generate the full definition and code for these functions.*\n\nProgram generation: \n\n*You have the following functions: [Full set of functions in the library]. These functions generate bounding boxes representing rooms. The bounding boxes are defined by name and corresponding coordinates x_min, y_min, x_max, y_max representing the bottom left and top right corners of the bounding box. Below are programs that generate a [Room type]. Note that each object is named based on its semantic class: [Few-shot examples]. Now, please generate another [Room type] using the given functions. You can generate this room by using different functions, locations, and object classes, but make sure they are reasonable objects to be in the room. Think step-by-step about where objects should be placed and what their sizes are, such that they are reasonable within the example program. Still return a dictionary called objects, do not repeat the given function implementations. Return code only. Only use the following object classes: [Objects categories from ScanNet].*\n\nIn addition, we add experiments to investigate the sensitivity of our program generation to different prompt variations, under different few-shot prompting configurations (5-shot, 3-shot, 1-shot). We see that FactoredScenes shows consistently strong performance across variants, significantly above that of prior works, demonstrating its robustness in generation.\n\n| Robustness        | All           | 5-shot FID | 3-shot FID | 1-shot FID |\n|-------------------|---------------|------------|------------|------------|\n| Ours bedroom      | 64.86 ± 4.36 | 67.51      | 59.83      | 67.25      |\n| Ours living room  | 91.26 ± 6.76 | 83.49      | 95.74      | 94.55      |\n\n**Q: Generalization to other real-world indoor scenes.**\n\nThank you for raising this point. We clarify that FactoredScenes aims to generalize in two key ways: (1) our LLM-based program generator does not rely on handcrafted & fixed libraries and can generate novel scene layouts beyond ScanNet (as well as learn new libraries as needed), and (2) our pose model is evaluated on new LLM-generated rooms not seen during training, demonstrating generalization beyond the training ScanNet scenes. We show in the main text that our pose prediction model can accurately generate poses for such new indoor rooms.\n\nHowever, the pose model is trained on ScanNet object orientations, so generalization to domains with fundamentally different pose distributions (e.g., outdoor or industrial scenes) would require new pose data, though we expect the distributions learned from ScanNet to transfer well across many indoor scene types. To showcase performance within indoor domains, we include new results on the office category, which did not require newly handcrafted function definitions in order to generate. We follow the same evaluation protocol as in the main paper and compute FID and KID between generated layouts and ScanNet office scenes.\n\n|      | Office FID | Office KID |\n|------|------------|------------|\n| Ours | 88.50    | 0.032     |\n\nWe agree that the limiting aspect here is the availability of real-world datasets, which we will highlight in our limitations section. Additionally, we note that while ScanNet is small, it spans diverse real-world indoor scene categories: apartment, living room, lounge, kitchen, hallway, bedroom, hotel, lobby, bathroom, office, classroom, gym, mail room, bookstore, library, stairs, mailboxes, computer cluster, storage, basement, garage, conference room, closet, dining room, copy room, game room, laundromat, laundry room. We will expand on this discussion in the revised paper.\n\n**Q: Verification and correction mechanism.**\n \nIndeed, in the library learning stage, FactoredScenes uses a reconstruction-based verification loop. If the program-generated layout differs from the ground truth input by more than a distance threshold, it is considered incorrect. We allow the LLM up to 16 attempts per scene to refine the program until a successful reconstruction is achieved. Thank you for your feedback, and we will clarify this in the main text."}, {"review_id": "NKZYrVnvj3", "reviewer": "Reviewer_e9Et", "summary": "FactoredScenes presents a modular framework for 3D scene generation that combines layout programs with learned object pose prediction. A function library is first learned from 3D-Front data. A neural model then predicts object poses conditioned on the program structure, and real 3D assets are retrieved and placed accordingly. The method achieves high-quality, diverse scene synthesis compared to previous methods.", "questions": "Can you elaborate on the model details, particularly the architecture of the pose prediction network? how the model architecture was chosen and ablated? Adding illustrations or diagrams to clarify the model architecture and technical components might further help.\n\nCould you provide more analysis on the system’s robustness and any other failure modes? Specifically, how sensitive is the pipeline to errors in program generation, and how do such upstream errors affect downstream pose prediction and object retrieval quality?", "limitations": "While the use of LLMs for layout program generation is effective, it is not particularly novel and the presented results are still of relative low visual fidelity. Following the line of LayoutGPT, Holodeck, and LayoutVLM, the proposed tailored mechanism library learning contributes to automating the extraction of a relation library on the code-level. However, the demonstrated improvements with more relation descriptions and noisy object pose remain relatively incremental. Compared to the current code-based approach, VLM-based extraction appears more promising for directly capturing relational patterns from real-world examples in the long-term. It may be interesting to incoporate more advanced explorations, such as expanding ground-plane layouts only to full 3D environments with vertical arrangements, or multi-level structures with hierarchical dependencies. Physical plausibility is not explicitly modeled or enforced, which may be considered to be added in LLM program generation. Including discussion or experiments on these fronts would strengthen the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Overall it proposed a relative comprehensive solution with well-engineered, modular pipeline with clear stage-wise design. The design is well-motivated and the leverage of LLM and neural model for pose prediction is quite reasonable. \n\nThe final rendered visuals exhibit relatively low fidelity. The two datasets used (3D-Front and ScanNet) are relatively limited in diversity and scale. Although the LLM-based program generation is a central component, it is treated largely as a black box, with little discussion of prompt sensitivity or the validity rate of generated programs. Some model details are under-specified, such as the pose predictor, which is only described as “attention-based.”", "was_revised": true, "final_justification": "After reading the rebuttal, I'd like to raise my score.", "rebuttal": "Thank you for your constructive feedback, which has greatly improved the quality of our paper. \n\n**Q: Additional analyses on robustness and failure modes of program generation.**\n \nThanks for the suggestion. We agree that it is valuable to analyze the capabilities of FactoredScenes’ LLM-based program generation. Here, we add experiments that test the validity of FactoredScenes’ generated programs, as well as the robustness of downstream results to potential errors in these programs. \n\nFirst, we report the percentage of FactoredScenes’s generated programs that execute successfully without errors and the percentage of programs that are semantically valid. As there is no ground truth for semantically valid programs, we define semantic validity using a conservative heuristic: a generated layout is deemed invalid if any pair of objects (excluding lamps and pillows) has IoU > 0.3. Here, we see that a high percentage of programs execute successfully, and even with this conservative, semantically valid criteria, we see a high percentage of programs marked as valid, which indicates that catastrophic errors are rare.\n\n| Across 100 rooms | % of program that executes successfully | % of programs that are semantically valid |\n|------------------|-----------------------|-----------|\n| Ours             | 97%                      | 90%             |\n\nSecond, we evaluate the effect of FactoredScenes’ program validity on downstream generation results, and report performance for scenes generated from valid vs. invalid programs. As shown below, semantically valid programs as per our heuristic produce lower (better) performance, although the difference is slight.\n\n| Generation | KID of semantically **valid** programs | KID of semantically **invalid** programs |\n|-----------|------------|----------|\n| Ours       | 0.0183     | 0.0192  |\n\nLastly, we investigate the robustness and sensitivity of our program generation to different prompt variations, under different few-shot prompting configurations (5-shot, 3-shot, 1-shot). FactoredScenes shows consistently strong performance across variants, significantly above that of prior works, demonstrating its robustness in generation.\n\n| Robustness        | All           | 5-shot FID | 3-shot FID | 1-shot FID |\n|-------------------|---------------|------------|------------|------------|\n| Ours bedroom      | 64.86 ± 4.36 | 67.51      | 59.83      | 67.25      |\n| Ours living room  | 91.26 ± 6.76 | 83.49      | 95.74      | 94.55      |\n\nWe also provide the prompts used for both library learning and program generation below, and will include this as well as expanded discussions in the revised paper. \n\nLibrary learning:\n\n*You have the following functions: [Current set of functions in the library]. Can you parse the below bounding boxes into programs with the above functions, such that they can be accurately reconstructed? The bounding boxes are defined by name and corresponding coordinates x_min, y_min, x_max, y_max representing the bottom left and top right corners of the bounding box. Please parse the bounding boxes into programs such that we only need to define the coordinates of minimal furniture only. The rest of the objects should be defined as so without coordinates and named (e.g., object_x = function(...)). Return code to generate the objects in a dictionary named objects, where the key is the original name of the object, and the value is the newly created object by the program. Bounding boxes: [Input bounding boxes].*\n\n*The following are programs describing layouts with bounding boxes, each program separated by ---. The current functions used are as follows: [Current set of functions in the library]. Can you propose new functions to compress these programs while retaining the same bounding box reconstructions? Compress programs by minimizing the number of objects that need explicit coordinate definition. Please generate the full definition and code for these functions.*\n\nProgram generation:\n\n*You have the following functions: [Full set of functions in the library]. These functions generate bounding boxes representing rooms. The bounding boxes are defined by name and corresponding coordinates x_min, y_min, x_max, y_max representing the bottom left and top right corners of the bounding box. Below are programs that generate a [Room type]. Note that each object is named based on its semantic class: [Few-shot examples]. Now, please generate another [Room type] using the given functions. You can generate this room by using different functions, locations, and object classes, but make sure they are reasonable objects to be in the room. Think step-by-step about where objects should be placed and what their sizes are, such that they are reasonable within the example program. Still return a dictionary called objects, do not repeat the given function implementations. Return code only. Only use the following object classes: [Objects categories from ScanNet].*\n\n**Q: Comparison to prior LLM-based works and the importance of pose predictions.**\n \nWe highlight that, unlike prior LLM-based methods (e.g., LayoutGPT, Holodeck, LayoutVLM) that operate with fixed, hand-crafted libraries, FactoredScenes learns a library of structural abstractions directly from data, and constrains LLM generation using this learned library. We hypothesize that this learnt structural prior helps reduce unnatural generations by anchoring the generation space to abstractions learned from example rooms, which leads to the strong performance of the FactoredScenes wo/ poses variant in Table 1.\n\nAdditionally, a key contribution of FactoredScenes is that it enables training on diverse datasets with noisy object poses, such as ScanNet (and hence generate real-world, lived-in scenes with object layouts), while prior works cannot, because they are limited to learning from axis-aligned objects from datasets like 3D-Front. We propose FactoredScenes as a structured framework precisely to enable learning from diverse data sources, in a data-efficient way with appropriate underlying models, such that we can generate realistic scene layouts.\n\n**Q: Clarification of model details.**\n \nOur pose prediction model predicts object poses in a hierarchical manner using per-object MLP encoders and self-attention modules, shown in Figure 4. For each object, we concatenate features including: category label, bounding box coordinates, closest wall encoding, and global wall features. These are passed through a Transformer-style self-attention block. Specifically, we apply layer normalization followed by learned linear projections to obtain queries, keys, and values, which are then processed via a multi-head attention module. The attention output is added back through a residual connection and passed through a feedforward network (two-layer MLP with ReLU) to predict pose. For dependent objects, we additionally encode the dependency function and the orientation of the source object. We include an ablation of the hierarchical design in Table 3, and will clarify these architectural details in the main text and add additional figures to illustrate the specific technical components.\n\n**Q: Limitations of 3D-Front and ScanNet.**\n\nWhile most prior works only train on 3D-Front, FactoredScenes additionally enables training on ScanNet, which allows learning of object poses in real-world, lived-in scenes. We agree that the limiting aspect here is the availability of real-world datasets, which we will discuss in depth in our limitations section. However, we highlight that while small, ScanNet covers a wide range of indoor scene categories: apartment, living room, lounge, kitchen, hallway, bedroom, hotel, lobby, bathroom, office, classroom, gym, mail room, bookstore, library, stairs, mailboxes, computer cluster, storage, basement, garage, conference room, closet, dining room, copy room, game room, laundromat, laundry room. \n\nHence, we expect the distributions learned from ScanNet to transfer well across many indoor scene types. To showcase performance within indoor domains, we include new results on the office category. We follow the same evaluation protocol as in the main paper and compute FID and KID between generated layouts and ScanNet office scenes.\n\n|      | Office FID | Office KID |\n|------|------------|------------|\n| Ours | 88.50    | 0.032     |\n\n**Q: Quality of rendered visuals.**\n \nWe clarify that the final rendered visuals use objects from ShapeNet and ScanNet, which are often partial or low-resolution. However, FactoredScenes’s generated layout enables us to flexibly substitute higher-fidelity meshes or point clouds during retrieval to improve visual quality without changing the generation pipeline. We will clarify and aim to improve visualizations in the revised paper. \n\n**Q: Discussions on VLM-based extraction, vertical arrangements, multi-level structures, and physical plausibility.**\n \nWe thank the reviewers for these suggestions, and agree that incorporating VLM-based extraction is a promising direction, especially for unlabeled scenes. We note that FactoredScenes currently captures relational patterns from real-world data via annotated object layouts in ScanNet. In the absence of such ground truth object poses, VLM-based extraction would be a valuable approach, and we view this as an exciting direction for future work.\n\nWe also agree that expanding to vertical arrangements, multi-level structures, and physical plausibility constraints is exciting and would augment the generation process. Currently, we focus on ground-plane layouts with a two-level hierarchy due to limited ScanNet data. However, the modular nature of FactoredScenes allows for incorporating more geometric reasoning and physical constraints (or other text-conditioned constraints) into future generations as data and models improve. Thank you for the feedback, and we will add discussions on these directions in the revised paper."}]

Author rebuttal:

Meta-review: The paper presents a method for generating 3D scenes by generating layouts and object poses using program synthesis with LLMs . The reviewers appreciated the method and the results, but initially raised concerns over limited evaluations and differences with the state of the art. The rebuttal addressed these concerns very well, and all reviewers are in favor of acceptance. The paper advances this problem by being able to synthesize real-world scenes with noisy data.
Decision: Accept (poster)