Video Perception Models for 3D Scene Synthesis
Authors: Rui Huang
Guangyao Zhai
Zuria Bauer
Marc Pollefeys
Federico Tombari
Leonidas Guibas
Gao Huang
Francis Engelmann
Venue: NeurIPS 2025 poster ()
Keywords: 3D scene synthesis
Video generation model
Primary area: applications

Abstract: Automating the expert-dependent and labor-intensive task of 3D scene synthesis would significantly benefit fields such as architectural design, robotics simulation, and virtual reality. Recent approaches to 3D scene synthesis often rely on the commonsense reasoning of large language models (LLMs) or strong visual priors from image generation models. However, current LLMs exhibit limited 3D spatial reasoning, undermining the realism and global coherence of synthesized scenes, while image-generation-based methods often constrain viewpoint control and introduce multi-view inconsistencies. In this work, we present Video Perception models for 3D Scene synthesis (VIPScene), a novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models to ensure coherent scene layouts and consistent object placements across views. VIPScene accepts both text and image prompts and seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene. This enables flexible scene synthesis with high realism and structural consistency. For a more sufficient evaluation on coherence and plausibility, we further introduce First-Person View Score (FPVScore), utilizing a continuous first-person perspective to capitalize on the reasoning ability of multimodal large language models. Extensive experiments show that VIPScene significantly outperforms existing methods and generalizes well across diverse scenarios.

Video Perception Models for 3D Scene Synthesis
Rui Huang 1+ Guangyao Zhai
Zuria Bauer
Mare Pollefeys 3.5
Federico Tombari Leonidas Guibas
Gao Huang
Francis Engelmann
SETH Zurich *Munich Center for Machine Learning Microsoft
TTinngua University Technical University of Munich
Stanford University
https: /ve: https://vipscenee .ei.ggithutte github.io
ViPScene
Reconstraction and 3D Object Detections
WIP
Asset Retriever
Test Image Condition
VIP
T
Synthesized Scene
theing coee with bedlstelf Briplace..
Vidco Generator
Object Detector
Reconstructor
Tracker
Figure I: Schematic of VIPSCENE. Conditioned on text or image prompts, VIPSCENE generates scenes by leveraging the commonsense priors of video generation models for scene layout and object placements. From the generated video, reconstruct the 3D scene and extract individual objects. The final scene is synthesized by replacing detected objects with high-quality assets from an object database.
Abstract
Automating the expert-dependent and labor intensive task of 3D scene synthesis would significantly benefit fields such architectural design, robotics simulation, and virtual reality. Recent approaches to 3D seene synthesis often rely on the commonsense reasoning of large language models (LLMs) strong visual priors from image generation models. However, current LLMs exhibit limited 3D spatial reasoning, undermining the realism and global coherence of synthesized scenes, while tere--nnnratttonbbeeeeeee methods often constrain viewpoint control and in- troduce multi-view inconsistencies. In this work, we present Video Perception models for 3D Scene synthesis (VIPSCENE). novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models to ensure coherent scene layouts and consistent object placements across views. VIPSCENE accepts both text and image prompts and seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object scene. This enables flexible scene synthesis with high realism and structural consistency. For a more sufficient evaluation on coherence and plausibility, we further introduce First-Person View Score (FPVSCORE), utilizing continuous first-person perspec- tive to capitalize on the reasoning ability multimodal large language models. Extensive experiments show that VIPSCENE significantly outperforms existing methods and generalizes well across diverse scenarios.
"Equal contribution.
1Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
(b)
(c)
Image prompt
sull
Text: bookstore with of booksheevess
Text: children's room
Figure 2: Diverse Synthesized Environments. From text prompts, VIPSCENE generates bookstore and (b) children's 's room. From an image prompt (c), VIPSCENE synthesizes a corresponding outdoor scene. Across settings, objects are semantically aligned with the prompts and arranged in physically plausible layouts.
Recent advancements in 3D scene synthesis have sparked significant interest across multiple domains, including gaming [22]. augmented reality [1. 50J, and robotics [6, 69]. Deep learning- -based methods have enabled the automatic generation of 3D scenes; however, they are limited by the insufficient diversity of available datasets. Recent progress in language and image generative models [2, 18, 53- 55, 11, 37] has further expanded the possibilities for synthesizing more diverse and plausible 3D scenes. A crucial step in this progress is the generation of spatially coherent layouts, which serve as the foundational structure for building lifelike 3D environments.
Despite impressive progress, existing methods still face notable limitations in generating real- istic layouts. One promising direction involves leveraging multimodal large language models (MLLMs) [69, 48, 9, 27, 8, 4, 3]. Among them, pure large language models (LLMs), which typically translate linguistic priors into layout constraints, often produce incomplete spatial specifications and treat scene synthesis as an optimization problem, which can result in the loss of spatial common- sense [69]. Vision-language models (VLMs)[18, 30] attempt address this issue by incorporating visual context to enhance spatial reasoning. However, their reliance on fixed image viewpoints limits their ability generalize 3D layout understanding [48, 34]. Alternatively, image-based approaches [60, 15] adopt specific viewpoints and apply recurrent inpainting strategies to directly exploit visual priors for commonsense layout generation. These methods, however, often suffer from hallucinations and spatial inconsistencies due to viewpoint dependency and the iterative nature.
Moreover, existing automated evaluation protocols often overlook spatial inconsistencies and unre- alistic layout distributions. Metrics such as CLIPScore [13] and VQAScore [28], based on VLMs, typically rely a single top-down view [69, 60]. This perspective can obscure important object details and impede accurate semantic interpretation (see Fig. 7), thus hindering effective assessment of layout coherence. Additionally, top-down views are also likely underrepresented in VLM training data, further reducing their ability to interpret such perspectives. Consequently, these metrics alone can not reliably reflect scene generation quality (see Tab. 2).
In this work, we approach the problem from two complementary perspectives: 3D scene synthesis and evaluation, with a focus commonsense reasoning and spatial coherence. For scene synthesis, we introduce Video Perception Models for 3D Scene Synthesis (VIPSCENE), which leverage rich visual priors from video generation models capture coherent scene layouts and object placements across views. As shown in Fig. 1, conditioned on either a text or image prompt or both, VIPSCENE first generates an egocentric video of scene, based on which the feed-forward 3D reconstruction recovers global geometry and open-vocabulary perception segments object instances. The instances enable large-scale asset retriever to substitute editable 3D assets, followed by a global optimization step to refine object poses, ensuring physical plausibility and avoiding collisions. VIPSCENE enables the generation of realistic, semantically faithful, and spatially coherent 3D scenes (see Fig. 2). Complementing this, we also propose a novel evaluation protocol, the First-Person View Score (FPVSCORE), which leverages advanced MLLMs like GPT-4o [18] and Gemini [53, 54] to assess generated scenes. A virtual camera captures 360 first-person views, which are concatenated into visual summaries and analyzed by MILMS through structured prompts. This approach offers a more interpretable and human-aligned evaluation of layout realism, spatial consistency, and semantic fidelity. Experiments show that VIPSCENE surpasses state-of-the-ar baselines across standard metrics, while FPVSCORE outperforms existing metrics and aligns closely with human judgments.

---

In summary, our contributions are threefold: (i) We present VIPSCENE, novel approach for realistic 3D scene synthesis that leverages video-based commonsense layout understanding, enriched with semantic and geometric cues obtained through consistent 3D reconstruction and perception. (ii) We introduce FPVSCORF, first-person view-based evaluation protocol that leverages ML.LMs for more comprehensive, interpretable, and human-aligned assessment of spatial coherence and semantic fidelity. (iit) We show that VIPSCENE outperforms state-of-the baselines across all metrics, demonstrating the effectiveness of video-grounded priors and modular decomposition pipeline for generating physically plausible 3D scenes.
2 Related Work
Indoor 3D Scene Synthesis. Indoor 3D scene synthesis has gained attention for applications in robotics [72, 32] and augmented reality [1. 50]. Existing methods generate scenes from language [51, 69, 48, 39, 10], graph-based instructions [71, 73, 26], or images [17, 60J, producing either object-level layouts [71, 70] or complete mesh scenes [15, 47J. However, models trained from scratch often suffer from dataset biases [71, 73, 67, 51J. Recent work thus leverages foundation models for broader generalization. While large language models (LLMs) can capture inter-object commonsense from text, they often yield incomplete or ambiguous spatial layouts due the lack of visual grounding [9, 69, 4, 3, 68]. In contrast, vision-language models (VLMs) incorporate visual cues to enhance spatial reasoning, but their reliance on fixed viewpoints limits their ability to infer holistic 3D scene layouts [48, 8J. Image- approaches [60, 15, 29] attempt to exploit commonsense visual priors for scene layout generation, but those based on single or multi-view images often struggle with limited viewpoint coverage and inconsistent spatial alignment across views. In this work, we leverage large-scale pre-trained video diffusion models [33] to generate long-how long-horizon clips from textual or image conditions, ensuring consistent viewpoints while enriching visual details.
Video Models. Building on the success of image symthesis [38], recent research has shifted toward video generation, which introduces challenges like temporal consistency and dynamic content modeling. Early efforts mainly relied on GAN-based frameworks [56, 43] that, while producing plausible results in controlled settings, often suffered from mode collapse and temporal incoherence. Recent approaches have extended diffusion models to the video domain [14, 65], leveraging their robustness to generate temporally coherent sequences. With the rise of Diffusion Transformers (3iT))35), methods now produce highly consistent photorealistic videos[21, 25, 31, 42, 33, 19], revolutionizing applications film, robotics, and other downstream tasks. In this work, we employ advanced video models to synthesize scenes with consistent views and broad viewpoint coverage, providing a strong foundation for generating 3D scenes with coherent layouts.
3D Geometric Reconstruction. Early approaches primarily focused on Structure from Motion (SfM) techniques, where keypoint detection and matching formed the basis for estimating camera poses and sparse point clouds [44, 45J. More recently, learning-basee approaches [57, 49, 52, 58, 64), including DUSt3R [59] and MASt3R [23], aim to eliminate iterative optimization and complex post-processing by directly performing multiple 3D tasks through a feedforward network. These methods typically integrate data-driven priors to mitigate ambiguities that classical techniques encounter, demonstrating strong generalization capabilities. In this work, we rely on Fast3R[64] for 3D reconstruction, and MASt3R[23] for tracking of objects from the generated video.
3 Method
The goal of VIPSCENE is to generate a realistic and physically plausible 3D scene from a user- specified prompt. Formally, given an image- text-prompt, VIPSCENE generates a 3D scene S = [o ON) ). where each object 02 = (c,, 8;, 4,, 0.) is represented by its category E C, size 8 E 3 position 4 e R3, and orientation o, E R around the gravity axis. Conditioned on the prompt, VIPSCENE first generates a first-person view video of a 3D indoor scene, leveraging commonsense knowledge on scene layout as well as object placements embedded in the video generation model. From this video, we employ recent 3D reconstruction and visual perception models to recover the full 3D scene geometry and decompose into its individual objects (Sec. 3.1). The scene is then recomposed by retrieving the most similar 3D assets from an object database. To ensure physical plausibility and resolve collisions, an additional optimization step refines object placements (Sec. 3.2). The overall framework is illustrated in Fig. 1, and a detailed algorithm is included in Algorithm 1. In summary, VIPSCENE effectively integrates commonsense knowledge into generated 3D scenes, capturing both the input prompt semantics and the physical plausibility of spatial layouts.
3

---

3.1 Scene Understanding
Scene Reconstruction. Given input prompt, a conditional video generator (Cosmos [33] produces a high-fidelity video. For 3D reconstruction, we first sample frames at 2 fps, yielding {I,...., Ir), where each frame I E R3xHxW for = ,,...,, T, representing diverse views of the scene. Trained on web-scale videos, the generator captures commonsense layout priors and spatial relationships, naturally extending the perceptual field beyond image-based methods like Architect [60]. Next, we process all unposed frames in parallel using a multi-view 3D reconstruction method to produce the 3D scene reconstruction R. implemented with Fast3R [64]. For metric 3D reconstruction, we estimate metric depth for each frame using the monocular predictor UniDepth [36], and rescale the reconstructed scene accordingly.
Object Detection. Next, we aim to detect all 3D objects in the reconstructed scene R. While off the-hhe-shef 3D object detectors are natural choice, we found them perform poorly the noisy reconstructed point cloud R, leading to inaccurate object categorization and size estimation, as shown adopt image-based approach. Specifically, we apply Grounded-SAM [41] to
in Fig. 8. Instead, detect and segment objects of interest independently in each frame, and then use MASt3R [23] to track and associate 2D detections across frames using its strong multi-view pixel-correspondence estimation capabilities. In this way, we can assign a unique identifier i to each object in the 31) scene. Specifically, for object i in frame we store its binary 2D object mask Mt We then use the masks across all views, Mi.... Mi, to extract the corresponding points from the reconstructed point cloud R. Given the high degree of noise in R. we propose an adaptive erosion scheme that filters out artifacts while preserving the object geometry. Specifically, we apply morphological erosion to each binary object mask M to suppress edge noise, with the erosion strength scaled by object size: larger objects undergo more aggressive denoising, while smaller objects are subject to gentler erosion. This allows us obtain a clean and accurate point cloud P, for each object i.
3.2 Scene Assembly
3D Asset Retrieval. In this stage, the goal is to replace the object point clouds P. with actual 3D assets. Towards that end, we pick the most similar asset from large-scale object database based on estimated object properties. Specifically, we extend beyond prior approaches (e.g.. Holodeck [69]) that rely primarily on visual similarity, textual relevance, and size similarity. Instead, we additionally adopt a point cloud registration-base based retrieval strategy to identify the most suitable asset candidates. Given an object point cloud P,. we first estimate its orientation 6;. We apply Principal Component Analysis (PCA) to compute the principal axes and use the direction of greatest variance to approximate the orientation. A tight bounding box is then aligned with 0,, from which position 1; and size are derived. As PCA cannot distinguish between O each candidate asset, we compute rigid transformation T [R] with rotation R E RR33 and translation E R3, by minimizing:
and n, each object yields two symmetric poses. For
R t*] argmin
Σ gEQ,
min RRq + t t- t---tt))))))))) -11)) + [So (R): PEP.
R.t
where p is a point in the object's point cloud P. and q is the closest point in the candidate asset Qj- The term Iso(3) (R) ensures that the rotation matrix R remains within the special orthogonal group SO(3) as defined in [74]. To solve this optimization problem, we use the Iterative Closest Point (ICP), initialized with the estimated poses. To identify the most suitable asset for each object, we select the candidate with the lowest root mean square etror (RMSE), resulting in scene composed of geometrically well-aligned assets. This step ensures accurate geometric alignment of the assets.
Object Pose Refinement. To address potential collisions caused by size mismatches between retrieved assets and objects, we introduce an optimization step to refine object placements in a physically plausible manner. This ensures that objects avoid overlap, stay within room boundaries (if specified), and remain close to their initial position. We define the total loss as a weighted sum of three losses, the position loss Lp. the overlap loss L., and the optional boundary loss Ls:
Lical = Cp + XoCo + ApLa.
where Xo and As weighting parameters, and the individual loss terms are defined as:
Cp
11
1
2 2. Co
Σ Area (BBox; O BBox,), Lo
Area (BBox, \ Room).
i=1

---

The position loss Lp encourages minimal deviation from the objects' original locations 7. The overlap loss C, penalizes intersecting object pairs based on the area their bounding boxes. The boundary loss Ls penalizes any part of an object that lies outside the room bounds, if provided. During optimization, object positions are iteratively updated along the gradient of Lioaal until convergence. The process terminates once overlap and boundary violations are eliminated or when no significant improvements are observed. This results collision-free, spatially coherent scene layout.
4 First-Person View Score - FPVSCORE
0
30
60
330
Instruction Template
Seene
Deseription Prompt
Our task here to compare three methods' room layout rationality, all generated from
description (lost description).
Scene 2
Criteria Focus
Semantic Correctness:
Layout Correctness:
3. Overall Preforence:
Room Analysis
provide answer, please deliver detailed analysis for each of the specified criteria. The analyais should he precise
Scene 3
the point.
Output Format
For each criterion, you need to make decision and rank the thrce scenes, Then, in the last row, summarize your final decision by "cranke".
Figure 3: Illustration of First-Person View Score. Rather than relying on a single top-down view, our metric uses sequence of first pest-peroon view images for each generated scene (left). A multimodal language model (MILMM) then analyzes and ranks the sequences hased multiple evaluation criteria (right).
User studies remain the gold standard for evaluating the quality of generated 3D scenes, they accurately reflect human judgment. However, they are expensive, tine-consuming, and difficult to scale. Recent works [69, 60] have explored automated evaluation using vision-language models (VLMs) like CLIP [40] and BLIP [24], which assess how well top-down rendering aligns with a given text prompt. For example, CLIPScore[ I3] computes the cosine similarity between the text encoding of the prompt and the image encoding of the rendered image.
Yet, top-down views may be underrepresented in the training distributions of VLMs, potentially limiting their ability to interpret such inputs accurately. Capturing the full scene layout, including fine-grained geometry and object semantics, in a single image embedding is inherently difficult. These views may also obscure key visual details, further reducing alignment fidelity. As result, the scores produced by these metrics are often unreliable and hard to interpret meaningfully.
Instead, we introduce an alternative evaluation protocol, called FPVSCORE, that uses multiple first- person views, which better reflect the training distributions of foundation models and offer improved scene coverage. Crucially, rather than using a single similarity metric, we exploit the perceptual and reasoning capabilities of multimodal models such as GPT-4o [18] and Gemini [53, 54] to rank the outputs of 3D scene generation models and to explain the rationale behind each ranking.
Metric Details and Prompt Design. Fig. 3 illustrates first-person view metric. For each scene, a virtual camera is placed at the center and simulates a 360-degree rotation, rendering frames at 30-degree intervals ensure comprehensive coverage and sufficient overlap. These frames are horizontally concatenated to form a compact visual summary. To enable consistent comparison across methods, we stack the summaries for all scenes and input them into the multimodal large language models (MLLMs) simultaneously, avoiding inconsistencies that could arise when evaluating them in isolation. Building on prior work in multimodal model-based 3D object assessment [61], we design a structured prompt to enable comparative evaluation of scene generations. As illustrated in Fig. 3. the prompt comprises three key elements: (1) task-specific instructions defining the multi- scene comparison goal, (2) clear list of evaluation criteria, and (3) formatting guidelines to ensure consistent output. The prompt guides the model to assess scenes along dimensions such as semantic correctness, layout accuracy, and overall coherence. We further instruct the model to justify its ratings with brief explanations, enabling verification of its reasoning and increasing trust in the evaluation. This design facilitates more holistic, interpretable, and scalable evaluation protocol for 3D scene generation, addressing the limitations of traditional top-down metrics by aligning more closely with human-like reasoning patterns (see Tab. 2).
5

---

5 Experiments
Experimental Details. We utilize Cosmos [33] for video generation and adopt Fast3R [64] for 3D reconstruction. For open-vocabulary segmentation, we employ Grounded-SAM [41], and UniDepth [36] is applied for monocular depth estimation. For comparison with baselines, we follow prior work [69] and evaluate on four types of scenes living room, bedroom, bathroom, and kitchen. We ask GPT-4o [18] to produce 25 text prompts for each room type. Each prompt consists of a description of room type and the desired items. Based on these prompts, we generate 100 rooms using each method under evaluation. We set A = λ = 10. Consistent with prior work, Holodeck [69], we retrieve 3D models from a high-quality subset of Objaverse [7] to ensure realistic and diverse object representations in the scene. Please refer to the appendix for more details.
Baselines. We compare our method with the most recent state-of-the-art approaches for 3D scene synthesis: Holodeck [69] is a comprehensive system that integrates LLM-based scene generation with optimization steps jointly produce room layouts and object placements. Architect [60] is a generative framework that creates interactive 3D scenes through diffusion-based 2D inpainting, relying on visual priors extracted from single images.
Metrics. To evaluate the quality of generated scenes, we report scores using the proposed first- person view metric FPVSCORE (Sec.4) and other automatic metrics used in prior work [60]] (1) CLIPScore [13], which measures image-text similarity via CLIP embeddings:
(2) BLIPScore, which
evaluates image-caption alignment using the matching head of BLIPv [24]; (3) VQAScore [28], which uses a visual question answering model to score how likely an image depicts the given caption; and (4) GPT-4o Ranking [18], which prompts GPT-4o to rank top-down rendered views.
User Study, We conduct a user study to compare scenes generated by our method against baseline approaches. Participants are shown a 360-degree video captured from the center of each scene, along with top-down rendered image, allowing them to assess both global structure and fine details. Thirty graduate students rated the scenes a 3-point scale (1 = lowest, 3 highest) across three criteria: Prompt Adherence (PA), "To what extent does the generated scene align with the inpus prompt?" Layout Correctness (LC), "Are the object placements physically plausible and functionally sensible?" and Overall Preference (OP).
5.1 Quantitative Results
Tab. presents quantitative results comparing our VIPSCENE to the Holodeck[69] and Architect [60] baselines. We report 2D image-based metrics using both first-person and top-down views, along with user study outcomes. VIPSCENE outperforms both baselines across all metrics. The user study, our most reliable evaluation, indicates that VIPSCENE better captures prompt semanties, produces more realistic scene layouts, and is the overall preferred method. This trend is also reflected in our proposed first-person view metric across three different VLMS (Gemini 2.0, GPT-4o, GPT 4.1). Top-down view scores offer much less clear picture. According to these metrics, performance across methods nearly indistinguisaabl, finding not supported by the more reliable user study. This supports our intuition that top-down view metrics are poorly suited for evaluating 3D scene generation; examine this further in Sec. 5.3. Another notable finding from our study is that Architect generally underperforms compared to Holodeck on the 100 generated scenes. This contrasts with the results reported in [60], but is partially supported by the qualitative examples in Fig. 4, which illustrate that Architect often produces unusual and sometimes impractical object placements.
Method
First-Person View Scores
User Study
Top-Down View Scores
Gemini 2.0 GPT-4ot GPT 4.1
PAt LCt OPt (LIPP BLIPt VQAScore GPT-4o1
Holodeck [69]
1.92
2.02
1.94
2.31 2.06 2.05
29.17 51.27
81.43
1.98
Architect [60]
1.77
1.62
1.76
2.05 1.94 1.98
29.95
49.72
78.34
1.90
VIPSCENE (Ours)
2.32
2.45
2.43
2.52 2.51 2.39
29.98 54.36
82.13
2.12
Table 1: First-Person View, User Study, and Top-Down View Scores. We report scores across different evaluation metrics. For First-Person View Scores and GPT-4o-based metrics, we report average ranking where 3 is the best and is the worst. Prompt Adherence (PA), Layout Correctness (LC), Overall Preference (OP).

---

VIPSCENE (Ours)
Holodeck [69]
Architect [60]
Living Room
Bedroom
Bathroom
Kitchen
Figure 4: Qualitative Results. We present top-down and close-up views of our VIPSCENE, comparing it against Holodeck [69] Architect [60] (columns) across four room types (rows). In the figures, Holodeck clearly leaves large areas unused while over-cluttering others, whereas Architect produces implausible arrangements that are impractical and rarely seen in real environments. VIPSCENE generates room layouts that are overall more realistic and natural.
VIPSCENE Image Prompt
Holodeck [69]
VIPSCENE
Figure 5: Image Prompting and Scene Realism. Left: Examples of image- based prompting: given an input image. VIPSCENE generates video and reconstructs full 3D scene. Note that based on the information from the first frame, the generated video can plausibly hallucinate objects beyond white sofa near the observer in the living room or the side window the bedroom. Right: Additional detailed comparison of scene layouts. Unlike Holodeck [69] (top), which handles object placement and window layout separately, method (bottom) jointly reasons about their spatial relationships, avoiding window occlusions and yiclding more coherent, realistic arrangements.
original field view, such as

---

Figure 6: Results from Complex Inputs. Left: Our model effectively generate coherent vidcos and corresponding 3D scenes from detailed spatial prompts, such "a desk placed under window with bookshelf to its left" and "a modern living room with a red sofa facing a fireplace, a coffee table between, and floor lamp placed diagonally behind the sofa." Right: To further enhance realism, we utilize CLIP features from cropped object regions during retrieval. This improves the aligument of textures and materials with input prompts like "a yellow sofa in the center of the room, with a dark rectangular coffee table in front and metal shelf next the sofa" and "the room has two light-colored sofas, an upholstered armchair nearby, modern brown coffee table in the center, and a wooden bookshelf against the wall." "
5.2 Qualitative Results
We present qualitative results in Fig. 4 to 6. For additional results, including those from multimodal inputs, please refer to the appendix. Fig. 4 specifically compares our VIPSCENE with Holodeck [69] and Architect [60], displaying randomly selected outputs from text prompts for living room, bedroom, bathroom, and kitchen scenes. Holodeck, relying solely on LLMs for spatial constraints and object relationships, often produces implausible absolute placements despite semantically correct relative pairings (e.g., chairs around a table). This is because LLMs exhibit limited 3D spatial reasoning, leading to objects being too close for passage while large room areas remain unused. Architect employs multi-view image diffusion and inpainting to hierarchically populate corner views, yielding a more natural mix of object scales. However, its resulting layouts are often incoherent, with issues like multiple sofas facing the same direction and implausible furniture placements. This incoherence likely stems from view inconsistencies and inpainting limitations.
Fig. 5 (left) shows scenes generated from image inputs by replacing the text-to-video model with an image-to-video model [21], while keeping the rest of the pipeline unchanged. The generated scenes demonstrate strong realism and coherence, with objects arranged in manner consistent with the overall video context. It is worth noting that, relying solely on the first frame, the generated video can reasonably infer objects beyond the original field of view, such the white sofa near the observer in the living room and the side window in the bedroom. Fig. 5 (right) showcases how existing methods overlook the joint spatial relationship between objects and windows, whereas our method explicitly models their relative placement. By leveraging spatial patterns in video data, our approach achieves greater logical consistency and enhanced visual plausibility.
We further evaluate our model using more complex textual descriptions, as illustrated in Fig. 6. The video generation model successfully interprets these detailed spatial instructions, producing coherent and realistic layouts that reflect the specified relationships among objects. These generated videos are subsequently used to reconstruct the corresponding 3D scenes, as shown in Fig. 6 (left). Our initial retrieval strategy primarily focused on object category and point cloud geometry to ensure overall scene plausibility. To better align retrieved assets with the textures and materials implied by the input prompt, we conduct additional experiments leveraging CLIP image features extracted from cropped object regions in the video frames. This enhancement enables the system retrieve assets that more accurately reflect the semantic and visual cues specified in the text, as demonstrated in Fig. (right).
5.3 Metrics Evaluation
A key challenge in evaluating generated 3D scenes the lack of suitable metric. Prior work [69, 60] typically uses 2D image- based metrics such as CLIPScore [13], BLIP- Score [24], or VQAScore [28]; however, moti- vated by the surprisingly similar scores across methods observed in Sec. 5.1 (Tab. 1), we inves- tigate how well these metrics actally align with human preferences. Specifically, we compute Kendall's T correlation [20] between metric- generated scores and reference scores from hu-
Metrics
7 (t)
CLIPScore (single top-down) [13]
0.06
BLIPScore (single top-down) [24]
0.07
VQAScore (single top-down) [28]
0.13
GPT-4o (single top-down)
0.27
GPT-4o (FPVSCORE, first-person views)
0.39
Table 2: Metrics Evaluation. How well do automated metrics agree with human ratings? Scores are Kendall's T correlations. Perfect agreement is r = 1, no association is 00 perfect disagreement is -
8

---

Top-Down View
First-Person View
Architect
Correctorss:
VIPSCINI Holsseck Amchtiect
10
bookshelf also includes
desk and computer...
The second one, clear bookshelr visuble..
one inclodes a bookstelf...
Holodeck
Correctness:
The finst placed reasurshlly.
finctional living room byyout...
The third une, pulbacays clea...
Semantic Correctness
Correctness
Overall Preference
VIPSCENE
The finst one, highly funetionnal and compact....
Overall Preferenee:
one cuffee tibli, but missinga a booksholf...
sofi,
and dysfucctional layout with the
very
meay...
fins looks
soda scatured and multiple tables placed awlowardly..
The socond ene looks like
The sccand resembles
The sccend one contiins soß,
functional living room....
comfortible living rooes....
culllee tubll bookshelvas...
reasonable laycut... somewhat cxppty...
second
third
The third me. looks like plausible living roam...
thind coffee tuble, and bookshelf...
The third one has a good layour..
Input Tex:: "A living room with coffee table, sofa, and bookskelf."
Figure 7: GPT-4o Output Analysis, Exemplary GPT-4o output: top-down view (left) and first-person view (right). Red text highlights implausible results; blue text marks reasonable ones.
evaluators. Tab. 2 reports the correlations, showing how closely each metric's predictions align with human judgments. Both CLIPScore and BLIPScore exhibit almost no association with human preference, suggesting that these metrics are suitable for automated evaluation. Although overall correlations are relatively low (reflecting the subjectivity of the task and variability in human judgments), our proposed first-person view metric, FPVSCORE, shows the strongest alignment with human preferences, suggesting that first-person views provide richer semantic cues leading to more reliable evaluations. Fig. illustrates comparison of the same model's reasoning for top-down and first-person views. Top-down views frequently obscure key object details, hindering semantic understanding and impairing layout evaluation.
Model
Model
T
Prompt
Gemini 2.0
0.61
Gemini 2.0 GPT-4o
0.54
w/o criteria 0.31
GPT-4o 0.74
Gemini 2.0 GPT-4.1
0.47
and analysis
GPT-4.1
0.71
GPT-4.1 & GPT-4o
0.60
Ours
0.39
(a) MLLMs
(b) Agreement across MLLMs
(c) Prompt design
Table 3: FPVScore Consistency Evaluation using Kendall's r correlation: Single-model stability, inter-model agreement, and prompt variants.
We further analyze the consistency of FPVSCORE to validate FPVScore's reliability, focusing on three aspects: (i) Consistency Across Tries. We repeatedly query each model and compute the average Kendall's correlation. Results (Tab. 3a) show MLLMs yield stable outputs, with GPT-4o demonstrating the highest consistency. (ii) Consistency Across Models. We assess agreement among different MLLMs by measuring pairwise Kendall's 7 correlations of their scene rankings. Results (Tab. 3b) indicate models exhibit similar relative judgments. (iii) Prompi Design. Removing specific prompt instructions (Tab. 3c) reduces Kendall's 's 7 correlation, suggesting structured prompts enhance human alignment beyond MLLMs' inherent reasoning.
5.4 Ablation Study
In this section, we present ablation studies on randomly selected subset of scenes to evaluate the contribution of individual model components. For each variant with one component removed, compare it against the full VIPSCENE model user preference test. Fig, reports the percentage of times VIPSCENE was favored over the model variant.
2D Vs. 3D Perception Models. After reconstructing the scene's point cloud, one straightforward approach is directly apply 3D perception model for scene decomposition. However, in practice, we observe that using 3D model, such as Mask3D [46], yields suboptimal results. The model suffers from significant classification emror across various object categories, and its segmentation masks are often inaccurate, leading to poor scene composition. We believe this issue stems from the higher levels of noise present in the point cloud generated by the video generation model, which differs significantly from the distribution of the
Frames
FPVSCORE
5
2.02
10
2.50
20
2.48
Table 4: Ablation on Frame
Quantity in Perception.

---

Prompt Adherence
Layout Correctness
Overall Preference
100%
10%
15%
20%
25%
100%
10%
15%
10%
75%
40%
40%
75%
75%
50%
90%
50%
85%
80%
508
90%
85%
90%
25%
55%
60%
25%
25%
08
08
08
web NR
OPR
3D
NR
w/o OFR
w/
web NR
OPR
w/
Figure 8: Ablation Study. Win ratio of VIPSCENE versus variants mcasured by prompt adherence (PA), layout correctness (LC), and overall performance (OP). Variants include the original model without noise reduction (NR), without object pose refinement (OPR), and using 3D instead of 2D object detector. A 50% win ratio indicates equal performance, while 100% means the full model always outperforms the variant.
training data used for the 3D perception model. In contrast, our approach begins with 2D perception models, which are more robust in accurately identifying object categories in images. This is further enhanced by point cloud denoising techniques, which reduce the noise in the reconstructed point cloud. As result, our method achieves more reliable and precise scene decompositions, ultimately leading to higher-quality scene synthesis. To assess 2D perception efficiency, we analyze the trade-off' between frame count and accuracy. As shown in Tab. 4, enlarging the temporal window improves performance but quickly saturates. In practice, sampling fis and using 10 frames strikes favorable balance between cost and accuracy.
Noise Reduction (NR). Due to potential motion blur and other artifacts in the generated video, well as the 3D reconstruction model itself, the reconstructed point clouds may exhibit significant noise, especially the bor- der of object masks, i..., at depth discontinuities. The adaptive erosion method we introduce effectively filters out these artifacts while preserving the integrity of the target objects, facilitating more precise scene decomposition and improving the asset retrieval process. Beyond the compare fixed-kernel variants. The adaptive setting consistently outperforms others (Tab. 5).
Erosion Method
FPVSCORE
Small Kernel
2.30
Large Kernel Adaptive
2.05
2.67
Table 5: Ablation on Adaptive Erosion.
ablation. we also
Object Pose Refinement (OPR). As demonstrated in Fig. 8, this refinement step effectively mitigates object collisions that may arise from size mismatches between the retrieved assets and the target objects. Following this step, users observe enhanced scene coherence, where objects are maintained close to their original placements while avoiding collisions with each other. This results in a more realistic and visually pleasing scene. We further conduct additional ablation experiments to better isolate the impact of each component the object pose refinement loss. The results are summarized in Tab. 6. It demonstrates that each loss term makes a meaningful contribution to enhancing the physical realism and stability the final scenes.
Variant
ratio (%)
w/o position loss C,
60
w/o overlap loss f.
85
w/o boundary loss Cs
75
Table Ablation on Refinement Loss.
6 Conclusion
In this work, we present VIPSCENE, a novel framework that leverages video perception models for 3D scene synthesis. By integrating video generation, 3D reconstruction, open-vocabulary object detection and tracking, as well as 3D asset retrieval, VIPSCENE bridges the gap between multimodal prompts and coherent, editable 3D scenes. Our method addresses existing challenges in spatial reasoning and multi-view consistency that limit current approaches based on language and image generation models. Furthermore, we introduce a new evaluation metric FPVSCORE that aligns better with human judgment of 3D scenes, offering a more reliable measure of semantic and spatial correctness in generated scenes. Through extensive experiments and user studies, VIPSCENE demonstrates superior performance across diverse scene types, both qualitatively and quantitatively. The obtained results highlight the importance of commonsense knowledge from video data and point toward a promising direction for future research in realistic and interpretable 3D scene generation.
Acknowledgment This work is supported in part by the National Key R&D Program of China under Grant 2024YFB4408200, the National Natural Science Foundation of China under Grants U24B20173 and 42327901, and the Scientific Research Innovation Capability Support Project for Young Faculty under Grant ZYCXQNUSKYCXX
10

---

References
[1] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes. International Journal on Computer Vision (IJCV). 2018.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 21XXx:::3330077744 2023.
[3] Rio Aguina-Kang. Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Ganeshan, R Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, and Daniel Ritchie. Open-universe indoor scene generation using Ilm program synthesis and uncurated object databases. arXiv preprint arXiv:2403.0000 09967,, 2024.
[4] Ata Çelen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, and Xi Wang. I-design: Personalized Ilm interior designer. In European Conference on Computer Vision (ECCV), 2025.
[5] Zilong Chen, Yikai Wang. Wenqiang Sun, Feng Wang. Yiwen Chen, and Huaping Liu. Meshgen: Generating pbr textured mesh with render-enhanced auto-encoder and generative data augmentation. In International Conference on Computer Vision and Puttern Recognition (CVPR), 2025.
[6] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luea Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. International Conference Neural Information Processing Systems (NeurlPS), 2022.
[7] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[8] Wei Deng, Mengshi Qi, and Huadong Ma. Global-local tree search vlms for 3d indoor scene generation. arXiv preprint arXiv:2503. 18476, 2025.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. International Conference on Newral Information Processing Systems (NeurIPS), 2023.
[10] Rao Fu, Zehao Wen, Zichen Liu, and Srinath Sridhar. Anyhome: Open-vocabulary generation of structured and textured homes. In European Conference on Computer Vision (ECCV), 2024.
[1I] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[12] Zebin He, Mingxin Yang. Shuhui Yang. Yixuan Tang, Tao Wang. Kaihao Zhang, Guanying Chen, Yuhong Liu, Jie Jiang, Chunchao Guo, al. Materialmyp: Illuiinatooniivaaii material generation via multi-view pbr diffusion. arXiv preprint arXiv:2503. 10289, 2025.
[13] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, arXiv preprint arXiv:2104.08) 18, 2021.
[14] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. International Conference on Neural Information Processing Systems (NeurIPS), 2022.
[15] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias NieBner. Tex12room: Extracting textured 3d meshes from 2d lext-to-image models. In International Conference Computer Vision (ICCV), 2023.
[16] Xin Huang, Tengfei Wang, Ziwei Liu, and Qing Wang. Material anything: Generating materials for any 3d object via diffusion. In International Conference Computer Vision and Puttern Recognition (CVPR). 2025.
[17] Zehuan Huang, Yuan-Chen Guo, Xingqiao An. Yunhan Yang, Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu. Yan-Pei Cao, and Lu Sheng. Midi: Multi-instance diffusion for single image to 3d scene generation. In International Conference Computer Vision and Pattern Recognition (CVPR), 2025.
[18] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprint 2410:2240.. 2024.
11

---

[19] Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang. Gao Huang, and Jiashi Feng. How far is video generation from world model: physical law perspective. arXiv preprint 1.X2v:2411.02385. 2024.
[20] Maurice G Kendall. A new measure of rank correlation. Biometrika, 1938.
[21]
Kuaishou. Kling, 2025, htttpa//kkiree
com/ globl//
[22] Vikram Kumaran, Jonathan Rowe, Bradford Mott, and James Lester. Sccnecraft: automating interactive narrative seene generation in digital games with large language models. In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
[23] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision (ECCV), 2024.
[24] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. International Conference on Machine Learning (ICML). 2023.
[25] Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi Hc, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.0013. 2024.
[26] Chenguo Lin and MU Yadong. Instructscene: Instruction-driver indoor scene synthesis with semantic graph prior. In International Conference on Learning Representations (ICLR), 2024.
[27] Yiqi Lin, Hao Wu, Ruichen Wang, Haonan Lu, Xiaodong Lin, Hui Xiong, and Lin Wang. Towards language-guides interactive 3d generation: Llms as layout interpreter with generative feedhack. arXiv preprint XxXiv 2023.
[28] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Xide Xia, Graham Neubig, Pengchuan Zhang, Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision (ECCV), 2024.
[29] Ling, Chen-Hsuun Lin, Tsung-Yi Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng. Yunhao Ge, Ming- Yu Liu, Aniket Bera, and Zhaoshuo Li. Scenethesis: A language and vision agentic framework for 3d scene generation. arXiv preprint arXiv:2505.0 02836, 2025.
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, International Conference on Neural Information Processing Systems (NeurIPS), 2023.
[31] Luma. Dream machine, 2024. https:/ / Lumalabs ai htt///reammmaeeeeeeeee htt///rrrmmmmccnnee.....
[32] Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. Conference on Robot Learning (CoRL), 2023.
[33] Nvidia. Cosmos, 2024. https: h/tp:://wwww /wwww nwwdia. com/ us hatss//a/cooooo cosmos/.
[34] Linfei Pan, Dániel Baráth, Mare Pollefeys, and Johannes L Schönberger. Global structure-from-nuee revisited. In European Conference on Computer Vision (ECCV), 2024.
[35] William Peebles and Saining Xie. Scalable diffusion models with transformers. In International Conference on Computer Vision (ICCV), 2023.
[36] Luigi Piccinelli. Yung-Hsu Yang, Christos Sakaridis, Maltia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[37] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXi::2307.0195 2023.
[38] Yifan Pu, Yiming Zhao, Zhicong Tang, Ruihong Yin, Haoxing Ye, Yuhui Yuan, Dong Chen, Jianmin Bao, Sirui Zhang, Yanhin Wang, et al. ART: Anonymous region transformer for variable multi-layer transparent image generation. In CVPR, 2025.
[39] Hou In Derek Pun, Hou In Ivan Tam, Austin T Wang. Xiaoliang Huo, Angel X Chang, and Manolis Savva. Hsm: Hierarchical scene motifs for multi-scale indoor scene generation. arXiv preprint arXiv:2003. 1684,8 2025.
12

---

[40] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
[41] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao. Jiayu Chen, Xinyu Huang. Yukang Chen, Feng Yan, al. Grounded sam: Assembling open- models for diverse visual tasks, arXiv preprint or Xiv:24001 14159, 2024.
[42] Runway. Gen 3, 2024. https:/ http://wwwwmm...oooa/aarrrceeeea
gen- 3- alpha.
[43] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In International Conference Computer Vision (ICCV), 2017.
[44] Johannes Lutz Schönberger and Jan-Michael Frahm. Stracture-fron- revisited. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[45] Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured quiti-view, stereo. In European Conference Computer Vision (ECCV), 2016.
[46] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In International Conference Robotics and Automation (ICRA), 2023.
[47] Jonas Schult, Sam Tsai, Lukas Höllein, Bichen Wu, Jialiang Wang, Chih- Yao Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, al. Controlroom3d: Room generation using semantic proxy rooms. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[48] Fan- Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, and Jiajun Wu. Layoutvim: Differentiable optimization of layout via rision-languag models. arXiv preprint arXiv:2412.02199 2024.
[49] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In International Conference on Computer Vision and Paternn Recognition (CVPR), 2021.
[50] Tomu Tahara, Takashi Seno, Gaku Narita, and Tomoya Ishikawa. Retargetable ar: Context-awarr aug- mented reality in indoor seenes based on 3d scene graph. In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunet (SMAR-Adjunec)) 2020.
[51] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthiss Nießner. Diffuscene: Denoising diffiusion models for generative indoor scene synthesis. In International Conference on Computer Vision and Pottern Recognition (CVPR), 2024.
[52] Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, and Zhicheng Yan. Mv-dust3r+: stagee-ttage scene reconstruction from sparse views in 2 seconds. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[53] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[54] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai. Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
[55] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[56] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[57] Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint arXiv:2408. 16061, 2024.
[58] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In International Conference on Computer Vision and Puttern Recognition (CVPR), 2025.
13

---

[59] Shuzhe Wang, Vincent Leroy. Yohann Cabon. Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In International Conference on Computer Vision and Pattern Recognition (CVPR), pages 20697-20709, 2024.
[60] Yian Wang, Xiaowen Qiu, Jiageng I. Liu, Zhehuan Chen, Jiting Cai, Yufei Wang, Tsun-Hsuan Johnson Wang. Zhou Xian, and Chuang Gan. Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting. International Conference on Neural Information Processing Systems (NeurIPS). 2025.
[61] Tong Wu, Guandao Yang. Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein, Gpt-4v (ision) a human-aligned evaluator for text-to-3d generation. In Proceedings of the IFFF/CVF conference on computer vision and pattern recognition, pages 22227-22238, 2024.
[62] Tianhao Wu, Chuanxia Zheng. Frank Guan, Andrea Vedaldi, and Tat-Jen Cham. Amodal3r: Amodal 3d reconstruction from occluded 2d images. arXiv preprint arXiv:2503. 2025.
[63] Jianfeng Xiang. Zelong Lv. Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01556 2024.
[64] Jianing Yang. Alexander Sax, Kevin J Liang. Mikael Henaff, Hao Tang. Ang Cao. Joyce Chai, Franziska Meier, and Mau Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in forward pass. In International Conference on Computer Vision and Puttern Recognition (CVPR), 2025.
[65] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion prohabilistic modeling for video genera- tion. Entropy, 2023.
[66] Xianghui Yang. Huiwen Shi, Bowen Zhang, Fan Yang. Jiacheng Wang. Hongxu Zhao, Xinhai Liu, Xinzhou Wang. Qingxiang Lin, Jiaao Yu, et al. Hunyuan3d 1.0: A unified framework for text-to-3d and image-to-3d generation. arXiv preprint arXiv:2411.022
2024.
[67] Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[68] Yixuan Yang, Junru Lu, Zixiang Zhao, Zhen Luo, James JQ Yu, Victor Sanchez, and Feng Zheng. Liplace: The 3d indoor scene layout generation and editing via large language model. arXiv preprint arXiv:2406. 03866, 2024.
[69] Yue Yang, Fun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han. Jiajun Wu. Nick Haber. Ranjay Krishna. Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. International Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[70] Zhifei Yang, Keyang Lu, Chao Zhang, Jiaxing Qi, Hanqi Jiang, Ruifei Ma, Shenglin Yin, Yifan Xu, Mingzhe Xing, Zhen Xiao, Jieyi Long, Xiangde Liu, and Guangyao Zhai. Mmgdreamer: Mixed-modality graph for geometry-controllabe
3d indoor scene generation. In Association for the Advancement of Artificial Intelligence (AAAI). 2025.
[71] Guangyao Zhai, Evin Pinar Örekk, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes with scene graphs. In International Conference on Neural Information Processing Systems (NeurlPS), 2023.
[72] Guangyao Zhai, Xiaoni Cai,, Dianye Huang, Yan Di, Fabian Manhardt, Federico Tombari, Nassir Navab, and Benjamin Busam. Sg- Object rearrangement via coarse-to-fin robotic imagination on scene graphs. In International Conference on Robotics and Automation (ICRA), 2024.
[73] Guangyao Zhai, Evin Pinar Örnek, Dave Zhenyu Chen, Ruotong Liao, Yan Di, Nassir Navab, Federico Tombari, and Benjamin Busam. Echoscene: Indoor scene generation via information echo over seene graph diffusion. In European Conference on Computer Vision (ECCV), 2024.
[74] Juyong Zhang, Yuxin Yao, and Bailin Deng. Fast and robust iterative closest point. Transactions on Puttern Analysis and Machine Intelligence (PAMI), 2021.
[75] Yuqing Zhang, Yuan Liu, Zhiyu Xie, Lei Yang, Zhongyuan Liu, Mengzhou Yang. Runze Zhang, Qilong Kou, Cheng Lin, Wenping Wang, et al. Dreammat: High-quality pbr material generation with geometry-an laget- diffusion models. ACM Transactions on Graphics (TOG), 2024.
[76] Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang. Yifei Feng. Mingxin Yang, Sheng Zhang, Xianghui Yang, et al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv preprint arXiv:2501. 2501 2025.
14

---

Appendix
A Detailed Algorithm
This section provides comprehensive breakdown of the proposed algorithm, outlining its modules, their interactions, and the data representations used throughout the process.
Algorithm 1: VIPSCENE: Prompt-to-Scent Generation
Input: Prompt (text / image), video generator G, reconstructor R (Fast3R), monocular depth D (UniDepth), detector S (Grounded-SAMM. tracker T (MASt3R), assets A, weights Ao: Ab Output: Final metrically scaled collision-free free scene S for x==
N where each object 0g = (Ca, $4, 1g 8g) is represented by its category ç₁ E C, size 8; E R position 4 R.
and orientation 0, IR around the gravity axis.
(A) Prompt Video frames
2 V < ((T))
3 {111=1 + -Samle frames from V with fps 2
4 (B) Video Metric 3D reconstruction
5 R + R((1))T=))
// Globally consistent 3D (unposed inputs) // Metric depths
[Dr]_=1 + -((())))
R Rescale R with metrie
// Enforce metric scale
(C) Scene decomposition & object extraction
do
10
f(c. )e < S(T)
1/ Per-frame 2D instance masks with
categories
(M, ls AdaptiveErodo ( denoising
// Size-aware morphological
12
for ( "h - T(H (M, (M, Ja)
[Mi)
[=1
N
// Temporal association / IDs
i=1
13
N do
P. - Segment points from by
// Per-object point cloud // Object category from detections
G Majority label( ((Mi)T=)
15
16
(D) 3D asset retrieval & alignment
for i to N do (s;, 1init ginitt t PCAInit(P.)
18
19
C, Retrieve candidates from according to c; best +- Ø, rmsemin f- +00
20
foreach Q E C, do
foreach E { ginir + ] do (R* t*), rmse ICPAlignn Q; pinit
21
22
11 Eq. (I), R SO(3)
23
if rmse rmsemin then rmse rmse.
25
best + (Q,R*, R', t²)
26
(Q., R4, t. 0.) best, + t;
27
(E) Final scene refinement
jorig 4- 4 Vi
// 1, denctes position variables
29
repeat
// Grraddent-based optimization
30
C,
14
pari ||
C.
Area BBox (1,, s;) n BBox (4, sy))
La D Area(BBox, si) Room)
33
Liotal + E, XoCo + Xgh,
(4) + Update((14, -n (1,) Croa)) if (Overlap((BBox.) 0 and AConal then
35
break
until converged
return S {(c;, St, 4, 1
15

---

B Computational Complexity
We have profiled both the inference latency and peak GPU memory consumption of each major stage in our pipeline in Tab. Among all stages, video generation is the most computationally expensive. requiring around 380s per video and 74GB GPU memory on an H100. 3D reconstruction, object detection and tracking, as well as asset retrieval, are significantly more efficient, each taking only few seconds per scene with much lower memory usage. The total latency for generating a complete scene is 400s, compared to 200s in Holodeck. Although our pipeline is more computationally intensive in the generation stage, it yields significantly higher scene quality, which we believe justifies the trade-off.
Table 7: Runtime and Memory Footprint per Stage.
Stage
Time Cost
Memory Usage
Video Generation
380
~7 GB
3D Reconstruction
~2s
~ GB
Object Detection & Tracking
~ s
~8 GB
3D Asset Retrieval
~ 8
Pose Refinement
~ 1 8
C Additional Qualitative Results
We provide additional qualitative results, including experiments on scene synthesis from multimodal and text-only inputs, shown in Fig. 9 and Fig. 10. Specifically, our video generation model utilizes both image and text inputs as complementary modalities for 3D scene generation. The input image defines the original field of view, while the accompanying text guides the model to plausibly infer and complete scene elements beyond the visible region. This multimodal setting enhances the flcxibility and robustness of the generation process, resulting in more accurate and diverse 3D scenes.
D Adaptive Erosion Method
We apply an adaptive erosion strategy to remove noise and retain high-contidence points in the reconstructed point clouds. As shown in Fig. 11, this approach effectively reduces artifacts in the initial pbj-ct point clouds, resulting cleaner and more coherent geometries. These improvements further benefit the subsequent asset retrieval process and enhance the overall scene quality.
Figure 11: Before-and-Afee Visualization of the Adaptive Erosion Strategy. The method effectively removes noisy points and preserves high-confidence regions, producing cleaner and more consistent geometries.
E Failure Cases
We show representative failure cases in Fig. 12. Typical errors include incorrect category recognition by the detector and missing detections caused by occlusion. When objects are dense, the layout optimization is not realistic enough. In the left example, a misclassified fireplace results in duplicated instances, whereas in the right example, the placement of the wooden table is not reasonable enough.
16

---

Image Input
Complementary
Text
modern living room with coffee table, bookshelf, armchair, TV stand.
3D Scene
Input
Image Input
Complementary
Text
cozy bedroom with bed, nightstand, dressing table, chair and wardrobe.
3D Scene
Input
Image
Input
Complementary
Text Input
A cozy kitchen with a kitchen island, refrigerator, dining table, chairs.
3D Scene
Figure 9: Qualitative Results from Multimodal Inputs. The generated video respects original field view provided by the input image while leveraging the accompanying text to plausibly infer and complete scene elements beyond the visible area.
17

---

VIPSCENE (Ours)
Holodeck [69]
Architect [60]
Living Room
Bedroom
Kitehen
Bathroom
Additional Qualitative Results. We present results of VIPŠCENE, comparing it against Holodeck [69] and Architect [60] (columns) across four room types (rows). For better visibility, ceilings and walls are removed.
Figure 10:
Figure 12: Failure Cases. Left: Misclassification by the detector results in duplicated fireplaces. Right: Inaccurate placement of wooden table due to layout optimization issues in dense scenes.
18

---

Implementation Details
Rescaling the Scene. To refine the overall scale of the reconstructed scene, we estimate depth maps for each view using UniDepth [36], and compare them against the corresponding reconstructed point clouds. For each point, we compute the ratio between the estimated depth and the depth derived from the reconstructed geometry. The global scale factor is then determined as the median of these per-point ratios across all views, providing a robust estimate that mitigates the influence of outliers. This median based scaling approach ensures consistency across views and improves the alignment of the reconstructed scene with real-world metric dimensions.
Orientation Estimation of the Object. Without loss of generality, we assume that object bounding boxes aligned with the ground plane. To estimate their orientation, first determine the ground plane equation. This process begins by extracting the ground point cloud, using method analogous object extraction. Specifically, scenes or "floor" for indoor scenes to generate ground masks. These masks are then used to extract the corresponding ground points from the reconstructed scene, and least-squares fitting is applied to estimate the ground plane. With the ground plane established, each object's 's point cloud P. is transformed into new coordinate system that retains the origin of the original camera coordinate system C, but aligns its horizontal plane with the estimated ground plane. The transformed point cloud is then projected onto the ground plane, and Principal Component Analysis (PCA) is applied to identify the principal axes of the point distribution. The direction of greatest variance is taken as an approximation of the object's orientation 0,. tight bounding box subsequently aligned with this estimated direction.
prompt Grounded-SAM [41] with the label "ground" for outdoor
G Prompts Details
Prompts for Scene Synthesis. We utilize GPT-4o [18] to generate text prompts for four types of living room, bedroom, kitchen, and bathroom. Each prompt specifies the room type along with the objects intended furnish the space. For example, "A bedroom with large bed, two nightstands, floor lamp, wardrobe, and a big window."
indoor scenes:
I'm working on an interior design project and would like to generate video scenes of a [room type] using a text-to-video model. Please help me create detailed prompts to feed into the model.
Guidelines:
1. Based on the typical function and layout of (room type], list the furniture, appliances, decorations, and other items commonly found in the space. Prompts should describe the room's contents clearly and in detail. Example: "A bedroom with a large bed, two nightstands, floor lamp, a wardrobe, and a big window."
Prompts for FPVSCORE. To facilitate consistent and goal-driven evaluations in FPVSCORE, we design structured prompts that include: (1) task-specifie instructions for multi-scene comparison, (2) clearly defined evaluation criteria, and (3) standardized formatting requirements. These prompts guide the model to assess each scene in terms of semantic fidelity, spatial layout accuracy, and overall coherence, while also requiring concise justifications to support its ratings and enhance transparency.
Task: Compare the room layout rationality of three methods, all generated from the same text description. From top to bottom, the video sequences display 360-degree view of each method's generated scene. Decide which method performs best according to the criteria below.
Text Description: [text_ [ecct_description
Instructions:
1. Semantic Correctness
Does the generated layout accurately reflect the text description? Check whether all described objects are present and correctly represented.
2. Layout Correctness
the room design physically plausible and functional?
19

---

Evaluate if the layout supports practical use, space efficiency, and proper object functionality, Consider object positions, orientations, and user convenience.
3. Overall Preference
Does the room layout look realistic and natural?
Consider the visual coherence and harmony of the scene.
Evaluation process:
Carefully examine the multi-view images of all three 3D scenes. Focus on one criterion at time and make independent judgments for each.
Output format:
Provide clear, concise analysis for each criterion. Avoid vague terms like "realistic"
"spacious." Instead, specify exact issues or strengths. For example:
For Semantic Correctness, indicate which objects missing inaccurately depicted. For Layout Correctness, specify which objects are misplaced or poorly oriented, and explain how this impacts usability or functionality.
After the analyses, assign ranks (1-3) to each method per criterion (1 = best, 3 = worst).
Summarize your final ranking in the format: <rank for criterion 1> <rank for criterion 2> <rank for criterion 3> for each method.
Example:
Analysis:
1. Semantic Correctness: The first one ...: The second one ...: The third one
2. Layout Correctness: The first one ...: The second one The third one
3. Overall Preference: The first one The second one The third one
Final answer:
The first one: x x
The second one:
The third one: X x x
(where denotes ranks 1-3)
(Please strictly follow the format above. Do not include extra symbols like quotation marks, or bullet points.)
Prompts for Top-Down View Scores. Following the approach of Architect [60], design targeted prompts to guide GPT-4o in evaluating room layouts based solely on top-down views. To ensure fair comparison, the prompts also emphasize spatial structure, semantic fidelity, and functional usability, consistent with our own.
Task: Compare the room layout rationality of three methods, all generated from the same text description. The top-down views of the scenes produced by the three methods presented from left to right. Identify which method performs best based on the criteria below. Text Description: ftext description)
Instructions:
1. Semantic Correctness
Does the generated layout accurately reflect the text description?
Check whether all described objects present and correctly represented.
2. Layout Correctness
Is the room design physically plausible and functional?
Evaluate if the layout supports practical use, space efficiency, and proper object functionality, Consider object positions, orientations, and user convenience.
3. Overall Preference
Does the room layout look realistic and natural? Consider the visual coherence and harmony of the scene.
Provide only your final ranking of the three methods in the format below:
20

---

Final answer:
X
(where denotes ranks from 3)
H
User Study Details
We conducted a thorough user study to evaluate the quality of the generated scenes, involving thirty participants. All participants took part voluntarily and received no compensation. At the start of the study, participants given five minutes read through the instructions, illustrated in Fig. 13. An example evaluation page presented to the participants is shown in Fig. 14.
Text-to-Room Layout Algorithm Evaluation
to layout algoritims, by different algonitimee and compare them based on the follwing three metrica:
study, you
layout images generated
Prompt Adherence (PA)
"To what extent does the generated scane wign with the input prompt??
Check the generated content represents the room description.
Layout
the object placements make both physically and functionally?"
Evaluate It the layout meets practical requirements, optimizee space utage, and soures abjects can used Intended.
Overall Preference (OP)
scene?"
Rate ovenall satisfaction with the design. Does the room layout realiatic and natural?
Evaluation Process
Cltck the 'Start Evalustion' button to enter the main evaluation page. background description and comespanding g videos below, rate the different metrica. The higher the score, the more it meets the requirements. the "Previous" and "Next" buttons continue evalusting the On the summary page, click the "Dowmoood Results" button to download a JSON and send us.
page.
your
images
Key Tips
computer torminal, This webpage supports resuming the evalustion from where you left If you accidentally diese the page, don't worry, just reopen the came browaer to continue.
strongly
the page
scenes generated Thank you very much for your support of this experiment!
randomizedd
P Pleaso enter your name
User Study Instructions. This page shown participants at the beginning of the study explain the task, interface, and evaluation criteria.
Figure 13:
I Limitations
VIPSCENE currently generates spatially coherent scene layouts, retrieving furniture from Obja- verse [7]. Although objects are richly annotated, some textures lack photorealistic quality. Future work will improve object quality by adopting advanced 3D generation techniques like text-to-3D and image-to-3l methods [62, 63, 66, 76], and by incorporating state-of-thed physically- rendering (PBR) techniques [5, 12, 16, 75] for realistic material representations and lighting. These improvements aim enhance both the diversity and realism of generated scenes.
21

---

A bedroom with bed, wardrobe, chair, dreeeing table, and armchair.
3011
9:11
Method
Method
Method 3
Prompt Adherencee: To whet extant does the genersted seene align with the input prompt?
Merthd
Merhod
Method
Layout Correctnes:: room design both physically plausible and
Method
Marhod
Method
Preference: Does the room layout appear realistic natural?
Method
Marhhd
Method
Provious
Hame
Next
Figure 14: Example Page. Participants were shown 360-degree video captured from the center of each scene, along with top-lown rendered image. This setup allowed them to evaluate both the global structure and fine details. Each rated 3-point scale lowest, highest) across three criteria.
22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We have included our claims and contributions in the introduutioon/asstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations throughout the paper when necessary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification:
Our paper does not include theoretical results.
23

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have disclosed the implementation information necessary for reproducibil- ity in the main paper and the appendix as thoroughly as possible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: Code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /tap:s://nii.... cc/
pub bublic// guides Stt//CodeSSuaa
ssionP ceiioooooiyyy olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (ht tpst
/nips. cc/ htpppuuliii iollic//ae c/ gu des CooeSSbmm ss i onPol mollicy icy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided experimental details in the experiments section of our paper. More details can be found in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The available resources limit our ability to achieve statistical significance in the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have indicated the compute resources required for this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //nuuiirssss. http://plccc//ccccccccttaae weeeeuuiieeiin iees?
Answer:
[Yes]
Justification:
We understand and conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the detailed broader impact in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe no such risk is involved.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have credited and referenced the papers of the codes and models used in producing this work. We have read and followed the licenses of the works.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thoode com/ dat aset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release well-documented assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
We conducted user study to evaluate our method. The full text of instructions given to participants included in the appendix. All participants were volunteers, and no monetary or other compensation was provided.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We adhere to the NeurIPS Code of Ethics and the guidelines. No IRB approvals are needed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
28

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We do involve LLM in the core of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non oon-standard components.
Please refer to our LLM policy (https: : / //nurriss.... ct/ Conffeeenceee / 2025/
LLM) for what should should not be described.
29


Reviews:
[{"review_id": "L3wHO48dS7", "reviewer": "Reviewer_5DAg", "summary": "The paper introduces VIPSCENE, a novel framework for 3D scene synthesis that leverages video perception models. Specifically, by using video generation models, 3D reconstruction and perception models, it aims to generate realistic and structurally coherent 3D environments from text or image prompts. Additionally, a new evaluation protocol, FPVSCORE, is proposed to better align scene quality assessment with human perceptions, especially focusing on first-person views. Extensive experimental results demonstrate VIPSCENE's superior performance over existing methods such as Holodeck and Architect.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. This paper is well-written and easy to follow. The proposed synthesis framework seems to be novel by integrating video generation and perception models. VIPSCENE effectively leverages the commonsense priors embedded in video diffusion models, addressing a key limitation of previous methods relying solely on language or visual priors.\n2. VIPSCENE ensures physical plausibility and avoides collisions by object detection and pose refinement. \n3. FPVSCORE offers a more human-aligned metric through first-person views and multimodal reasoning, outperforming traditional top-down view-based metrics like CLIPScore.\n4. Finally, the extensive experiments demonstrate superior performance quantitatively and qualitatively across multiple scene types; and the ablation studies validate the contribution of each component.\n\nWeaknesses\n1. Computational Complexity: The pipeline involves several resource-intensive steps (video generation, 3D reconstruction, asset retrieval), could you provide the memory consumption and inference latency?\n2. The generation of small objects. Since the synthesis in this paper depends on the 3D reconstruction and detection, however, the small objects are not easy to detect. Thus, is it possible for FPVSCORE to generate small objects in the scene?   \n3. Limited by the video generation and the perception models. Currently, the video generation model struggles to predict videos with temporal and spatial consistency, how do you address the problem caused by the unrealistic video? Could you show some failure cases where the VIPSCENE fails? Meanwhile, the synthesis ability is influenced by perception models as well, could you provide more analysis?", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and positive feedback. We're glad they recognized our method as a **novel integration** of video generation and perception models that **addresses key limitations** of prior work. We also appreciate their acknowledgment of FPVScore as a more **human-aligned metric** than CLIP-based alternatives, and their appreciation of our **comprehensive experiments**.\n\nBelow, we address each concern in detail. We believe these clarifications improve the clarity of our work and look forward to further discussion. Thank you.\n\n---\n\n###  1. **Computational Complexity.**\n\nThe reviewer asks about memory consumption and inference latency of the pipeline components. We have profiled both the inference latency and peak GPU memory consumption of each major stage in our pipeline. The results are summarized in the table below and have been included in the revised paper.\n\nAmong all stages, video generation is the most computationally expensive, requiring ~380s per video and ~74GB GPU memory on an H100. 3D reconstruction, object detection and tracking, as well as asset retrieval, are significantly more efficient, each taking only a few seconds per scene with much lower memory usage.\n\nThe total latency for generating a complete scene is ~400s, compared to ~200s in Holodeck. Although our pipeline is more computationally intensive in the generation stage, it yields significantly higher scene quality, which we believe justifies the trade-off.\n\n| Stage | Time Cost | Memory Usage |\n| --- | --- | --- |\n| Video Generation | ~380s | ~74GB |\n| 3D Reconstruction | ~2s | ~8GB |\n| Object Detection & Tracking | ~15s | ~8GB |\n| 3D Asset Retrieval | ~5s | — |\n| Pose Refinement | ~1s | — |\n\n---\n\n###  2. **Small Object Generation.**\n\nThe reviewer asks if our approach can generate small objects. Yes, **our method is capable of generating small objects**, thanks to the strategic use of the state-of-the-art 2D open-vocabulary detector, Grounded-SAM, applied directly to video frames. Compared to 3D detectors, 2D detectors are generally more robust and benefit from training on large-scale image datasets, which enhances their ability to detect objects of various sizes, including small ones.\n\nAs a result, our pipeline can effectively capture fine-grained scene details. We demonstrate successful generation of small objects such as a glass on a bedside table, a PC on a desk, a fruit bowl on a coffee table, and a vase on a bookshelf (see Fig. 3).\n\n---\n\n###  3. **Video Consistency.**\n\nThe reviewer is curious about potential temporal and spatial inconsistencies in the generated video. While earlier video generation models often struggled with temporal and spatial consistency, recent advancements have made substantial progress in generating longer and more coherent videos. In our work, we **adopt** **state-of-the-art video generation models such as Cosmos and Kling**, which have demonstrated improved consistency.\n\nTo further mitigate residual inconsistencies, our framework employs **two key strategies**. First, we utilize a **globally consistent 3D reconstruction model, Fast3R**, which jointly optimizes over all video frames. This global optimization enforces **multi-view geometric consistency**, making the final 3D reconstruction robust to minor inconsistencies at the frame level. Second, we introduce an **adaptive erosion technique** designed to **remove noisy points at object boundaries**. These noisy points are often artifacts caused by slight temporal or spatial misalignments in the video frames. Together, these strategies effectively improve the quality and consistency of the reconstructed scenes.\n\n---\n\n###  4. **Perception Analysis & Failure Cases.**\n\nAs the output quality of our model also depends on the performance of perception models, the reviewer wonders how these influence the generation quality. We fully agree that perception models play a critical role in synthesis quality. To provide deeper insight, we have included additional ablation studies and failure case analysis in the revised manuscript.\n\nFirst, we analyze the **adaptive erosion method**, which dynamically adjusts the kernel size based on object masks. This approach effectively removes boundary artifacts without over-eroding small objects. Ablation results show that it consistently outperforms fixed-kernel alternatives.\n\n| Erosion Method | FPVScore |\n| --- | --- |\n| Small Kernel | 2.30 |\n| Large Kernel | 2.05 |\n| Adaptive | 2.67 |\n\nSecond, we examine the impact of the **number of video frames** used for reconstruction. Fewer frames lead to incomplete object point clouds, which can cause inaccuracies in object size estimation and retrieval. Our results show that performance improves with more frames, up to a saturation point. In our experiments, we sampled the video at 2 frames per second, using approximately 10 frames, which provided a good trade-off and satisfactory results.\n\n| Number of Frames | FPVScore |\n| --- | --- |\n| 5 | 2.02 |\n| 10 | 2.50 |\n| 20 | 2.48 |\n\nFinally, we include **representative failure cases** in the revised manuscript, such as missing or duplicated detections. While we cannot provide images here due to rebuttal policies, these cases are clearly illustrated and discussed in the updated version."}, {"review_id": "E2tSxgfrq8", "reviewer": "Reviewer_xyXq", "summary": "This paper points out that current LLMs have limited 3D spatial reasoning, and image-based generation methods often struggle with multi-view consistency. To address this, the authors propose VIPScene, which uses video generation models to better capture 3D world knowledge. They also introduce FPVScore, a new metric for evaluating consistency and generation quality from first-person views. Experimental results show that VIPScene outperforms existing baselines on indoor scene generation tasks.", "questions": "Please refer to the Weaknesses section above. I am open to raising my score if the authors can clearly address the listed questions and concerns.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe motivation is meaningful and relevant. The layout generation is based on videos from strong text-to-video models, which helps produce more realistic and diverse scenes. In addition, evaluating the synthesized scenes from ego-centric views introduces a fresh and practical perspective for 3D scene synthesis.\n\n\n## Weaknesses\n\n### Contribution & Originality\n\nWhile the overall pipeline is interesting, the paper lacks clear technical contributions. Most components—such as scene reconstruction, object detection, and scene assembly—rely on existing off-the-shelf methods. There is no clear innovation or customization that sets the pipeline apart from prior work. As a result, the originality of the approach feels limited.\n\n\n\n### Experiments\n\nSince FPVScore is proposed as a key evaluation metric for indoor scene synthesis, it would be more convincing to include comparisons with more strong baselines—especially methods discussed in the related work section (e.g., [15], [47]). This would better position the proposed method and show whether FPVScore offers meaningful distinctions between methods.", "was_revised": true, "final_justification": "Thanks for the response. Most of my concerns are solved. I raise my score to borderline accept.", "rebuttal": "We sincerely thank the reviewer for their time and thoughtful comments. We're pleased they found our method **interesting** and saw our proposed metric as a **fresh and practical contribution** to 3D scene synthesis.\n\nWe address each point in detail below and believe the revisions significantly improve the clarity and quality of the manuscript. We welcome any further questions or suggestions during the discussion period. Thank you.\n\n---\n\n###  1. **Contribution & Originality.**\n\nThe reviewer would like to let us elaborate more on technical contributions. We agree that our approach builds upon powerful existing models. However, we respectfully emphasize that the originality of our work lies in the **novel synthesis paradigm**, the **careful system design that enables it to succeed**, and the **introduction of a new evaluation metric** tailored to this task.\n\n**First**, to the best of our knowledge, **VIPScene is the first method to synthesize interactable 3D scenes conditioned jointly on image and text prompts**, leveraging video generation models to produce plausible spatial layouts and object relationships. This stands in contrast to previous approaches, which are limited either by the abstract spatial reasoning of LLMs or by the narrow field of view in image-based methods.\n\n**Second**, VIPScene’s strong performance stems from **careful customizations designed to solve critical challenges at model interfaces**.\n\n- We address common spatial inconsistencies in reconstructed scenes by introducing a custom adaptive erosion scheme (Sec. 3.1), which significantly improves object segmentation and downstream asset retrieval (Fig. 6).\n- In the asset retrieval stage, we go beyond standard vision-language similarity by incorporating a geometry-aware retrieval strategy using point cloud registration (Sec. 3.2). This ensures that retrieved assets match the shape and scale of reconstructed objects, which is essential for scene realism.\n- Our multi-objective pose refinement loss (Sec. 3.2) is specifically designed to address object placement challenges after retrieval, ensuring physical plausibility, collision avoidance, and fidelity to the layout derived from the video (Fig. 6).\n\n**Third**, we propose **FPVScore**, a novel evaluation protocol that better aligns with human perceptual judgments. Our experiments validate its effectiveness, showing a substantially higher correlation with human ratings than conventional metrics such as CLIPScore (Tab. 2). We believe that **establishing a more perceptually faithful protocol can drive meaningful progress in this domain**.\n\n---\n\n###  2. **FPVScore Validity & Baseline Comparison.**\n\nThe reviewer suggests using FPVScore to evaluate additional baselines, and we appreciate this valuable suggestion. Regarding the mentioned baselines, GPT-4 System Card [15] (a language model) and NeuralRecon [47] (a 3D reconstruction model), we would like to clarify that while both are cited in our submission paper, they do not directly perform 3D scene generation, which is the primary focus of our work. Though [15] and [47] can be part of a 3D scene generation pipeline to some degree, they are not direct competitors in this domain.\n\nTo further validate the effectiveness of FPVScore, we have included two strong 3D scene generation baselines in our experiments: Text2Room [1] and LayoutVLM [2], the latter of which was recently open-sourced.\n\n| Method | FPVScore |\n| --- | --- |\n| Text2Room | 1.32 |\n| LayoutVLM | 1.80 |\n| Ours | 2.62 |\n\nAs shown in the table, **VIPScene significantly outperforms both baselines** **under FPVScore**. Moreover, FPVScore successfully captures critical failure patterns of existing methods: Text2Room often produces scenes with geometric distortions and duplicated objects. LayoutVLM tends to generate spatially incoherent object layouts.\n\nThese results highlight FPVScore’s **discriminative capacity** and its strong alignment with human perceptual judgments. We have now included the comparisons in the manuscript.\n\n[1] Text2room: Extracting textured 3d meshes from 2d text-to-image models. Höllein et al. ICCV 2023.\n\n[2] Layoutvlm: Differentiable optimization of 3d layout via vision-language models. Sun et al. CVPR 2025."}, {"review_id": "cUyfxam7W1", "reviewer": "Reviewer_5Nqw", "summary": "This paper introduces VIPSCENE, a novel framework for generating realistic 3D scenes by leveraging the commonsense knowledge embedded in large-scale video models. The core idea is to first generate plausible first-person viewpoint videos from a text or image prompt and then reconstruct a coherent 3D scene from these videos. To better evaluate the human-perceived quality of the output, the authors also propose FPVSCORE, a new evaluation protocol based on first-person view analysis. Experiments and user studies show that the method generates more semantically and geometrically coherent scenes than existing techniques.", "questions": "See the weakness section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n*   **More Human-Aligned Evaluation:** The proposed FPVSCORE evaluation metric is a significant contribution. It offers a more intuitive, first-person-based alternative that better correlates with human judgment.\n\n*   **Strong Experimental Results and User Studies:** The method demonstrates clear superiority over existing baselines like Holodeck and Architect. The inclusion of a user study provides strong evidence that the generated scenes are not just better according to the proposed metric but are also genuinely preferred by human observers.\n\n*  **Well-Written :** The paper is exceptionally well-written and easy to follow. \n\n### **Weaknesses**\n\nWhile the overall vision is compelling, the paper could be significantly strengthened by addressing several concerns regarding its technical depth, experimental rigor, and presentation clarity.\n\n* **Technical Novelty:** The core contribution appears to be a cleverly engineered multi-stage pipeline (prompt -> video -> 3D) that connects existing large-scale models. The paper does not seem to introduce a new, fundamental technical component. The framework's success relies on the strength of its off-the-shelf parts rather than a core algorithmic innovation developed in this work.\n\n* **Insufficient Ablation Studies:** The current ablation study is too high-level. For example, for the Object Pose Refinement step, the paper should provide a separate analysis of the different loss components. For the artifact removal process, it would be best to provide visual results (e.g., before-and-after comparisons) to demonstrate its effectiveness.\n* **Clarity of Methodology Presentation:** The current diagram of the method is a high-level flowchart. This is useful for conveying the overall concept but insufficient for understanding the technical execution. The paper would greatly benefit from a more detailed architectural diagram illustrating the precise inputs/outputs of each module, the flow of data representations (e.g., videos, point clouds, meshes), and how the different components interface with one another.\n\n* **Unreported Inference Speed:** The paper provides no analysis of the end-to-end inference time. For a multi-stage pipeline, understanding the total computational cost and latency—from initial prompt to the final rendered 3D scene—is crucial for assessing its practical viability. \n\n* **Potential Limitations of FPVSCORE:** The metric seems to evaluate object co-occurrence and viewpoint plausibility. It is unclear if it can detect more subtle geometric inconsistencies, such as objects floating slightly above a surface or having incorrect physical interactions, which a human observer would notice immediately.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' rebuttal. I appreciate their efforts, as they have successfully addressed my specific concerns regarding methodological details and ablation studies by providing new experiments and committing to revisions. This has certainly improved the paper's quality. However, the rebuttal also reinforces my core assessment of the work's contribution. Its primary novelty lies in the clever integration of existing techniques—a significant engineering achievement, but not a fundamental algorithmic innovation. Furthermore, the high computational cost (~400s per scene) was confirmed, which is a major practical limitation. In balancing these points, while the paper is stronger, my overall evaluation of its impact remains consistent with my initial review. Therefore, I am maintaining my score.", "rebuttal": "We would like to thank the reviewer for their time and encouraging feedback. We're glad they recognized **FPVScore as a more intuitive, human-aligned metric**, and appreciated the **strong experimental results and user studies** confirming human preference for the scenes.\n\nWe address all concerns in detail below and believe the updates have significantly improved the clarity and quality of the manuscript. We welcome any further suggestions during the discussion period. Thank you.\n\n---\n\n###  1. **Technical Novelty.**\n\nThe reviewer would like us to provide further elaboration on the technical contributions. We agree that our method benefits from powerful pre-trained models, but we respectfully underscore that the **novelty and technical contributions** of our approach lie in several key aspects:\n\n**First**, to the best of our knowledge, **VIPScene is the first method capable of generating interactable 3D scenes based jointly on image and text prompts**. By leveraging video generation models, our approach enables the creation of coherent spatial layouts and object relationships, surpassing prior methods that either depend on abstract spatial priors from language models or are constrained by limited visual context from static images.\n\n**Second**, VIPScene’s strong performance stems from **careful design choices** that address specific challenges at module interfaces:\n\n- We propose a custom adaptive erosion technique (Sec. 3.1) to resolve spatial inconsistencies in scene reconstruction, enhancing both segmentation quality and subsequent asset retrieval (Fig. 6).\n- Our geometry-aware retrieval mechanism (Sec. 3.2), based on point cloud registration, ensures that retrieved assets are consistent with the shape and scale of reconstructed objects.\n- We introduce a multi-objective pose refinement loss (Sec. 3.2) that improves placement by enforcing physical plausibility, collision avoidance, and aligning with the scene layout inferred from video (Fig. 6).\n\n**Third**, we propose **FPVScore**, a novel evaluation metric designed specifically for this task. This metric better reflects human perceptual judgments compared to existing scores like CLIPScore. **We believe that prioritizing perceptual fidelity in evaluation will foster meaningful improvements in this area.**\n\n---\n\n###  2. **Ablation Studies.**\n\nThe reviewer asks for more detailed ablation studies for object pose refinement and the artifact removal process. First, we have conducted additional ablation experiments to better isolate the impact of each component in the **object pose refinement loss**. The results are summarized below:\n\n| Variant | Win rate of full model vs. variant |\n| --- | --- |\n| w/o $L_p$ (the position loss) | 60% |\n| w/o $L_o$ (the overlap loss) | 85% |\n| w/o $L_b$ (the boundary loss) | 75% |\n\nIt shows that **each loss term contributes meaningfully** **to improving the** **physical realism and stability** of the final scenes.\n\nIn addition, to demonstrate the effectiveness of our **adaptive erosion strategy**, we now include before-and-after visualizations in the Appendix. They show how our method reduces artifacts in initial per-object point clouds and **leads to** **cleaner, more coherent geometries**, which in turn benefit asset retrieval and scene quality.\n\n---\n\n###  3. **Inference Speed.**\n\nThe reviewer asks about the inference speed of our pipeline. We have measured both the inference time and peak GPU memory usage for each major stage of our method, and we now report these results in the revised manuscript.\n\nAmong all components, video generation is the most resource-intensive, taking ~380s per video and utilizing up to 74 GB of GPU memory on an H100. 3D reconstruction, object segmentation and tracking, and asset retrieval are lightweight, each completing in just a few seconds with significantly lower memory requirements.\n\nThe total end-to-end latency for producing a complete 3D scene is ~400s, compared to ~200s in Holodeck. While our pipeline introduces higher compute cost in the generation stage, we believe the resulting improvements in scene quality and realism offer a meaningful and worthwhile trade-off for many applications.\n\n| Stage | Time Cost | Memory Usage |\n| --- | --- | --- |\n| Video Generation | ~380s | ~74GB |\n| 3D Reconstruction | ~2s | ~8GB |\n| Object Detection & Tracking | ~15s | ~8GB |\n| 3D Asset Retrieval | ~5s | — |\n| Pose Refinement | ~1s | — |\n\n---\n\n###  4. **Discussion on FPVScore's Evaluation Scope.**\n\nThe reviewer asks about FPVScore’s ability to capture subtle geometric details. Indeed, detecting subtle geometric inconsistencies, such as slight floating or minor interpenetrations, remains a challenge for purely image-based evaluation methods, including FPVScore. The current implementation of FPVScore is particularly effective at capturing semantic correctness, layout plausibility, and object co-occurrence, which are aspects that Multimodal Large Language Models (MLLMs) can reliably infer from first-person image sequences.\n\nWhile the first-person perspective offers a more immersive and realistic viewpoint than traditional top-down metrics, even human observers may struggle to detect fine-grained physical inaccuracies from rendered videos alone, especially without access to the underlying 3D geometry.\n\nThis limitation is well acknowledged in the field. We believe a promising path forward is to **augment MLLMs with specialized geometric reasoning tools via function calls**. For example, the evaluator could:\n\n- **Check support surfaces**, verifying whether each object rests on a plausible base such as a floor or another object.\n- **Assess physical stability**, determining whether object placements are physically stable.\n\nWe have included a discussion of this direction in the revised manuscript, highlighting how combining commonsense reasoning with explicit geometric validation could significantly enhance the robustness and physical accuracy of scene evaluation. This extension represents a natural evolution of the FPVScore framework.\n\n---\n\n###  5. **Methodology Presentation.**\n\nThe reviewer suggests adding a more detailed diagram to better illustrate our method’s architecture and data flow. In the revised manuscript, we have **added a detailed architectural diagram** to the Method section, clearly showing the **inputs, outputs, data representations**, and how each module interfaces with the others throughout the pipeline.\n\nWhile figures cannot be added at the rebuttal stage, we provide a comprehensive textual walkthrough of the full pipeline below:\n\n(A). Input Prompt to Video\n\n- Input: A user-provided text or image prompt\n- Process: The prompt is passed through the video generation model, producing a high-fidelity video\n- Output: A sequence of frames {$\\{I_1, ..., I_T\\}$} sampled from the generated video\n\n(B). Video to Metric 3D Scene Reconstruction\n\n- Input: The unposed video frames {$\\{I_t\\}$}\n- Process:\n    - Fast3R processes all frames jointly to produce a globally consistent 3D point cloud\n    - UniDepth estimates metric depth for each frame\n    - The point cloud is rescaled using depth maps to ensure correct metric scale\n- Output: A single metric 3D point cloud of the entire scene $R$\n\n(C). Scene Decomposition and Object Segmentation\n\n- Input: The scene point cloud $R$ and original video frames {$\\{I_t\\}$}\n- Process:\n    - Grounded-SAM generates 2D segmentation masks for each object\n    - We apply adaptive erosion to reduce edge noise\n    - MASt3R tracks the masks across frames, establishing temporal correspondences\n    - Tracked masks are used to extract per-object point clouds from $R$\n- Output: A set of object-level point clouds {$\\{P_1, ..., P_N\\}$}\n\n(D). 3D Asset Retrieval and Alignment\n\n- Input: Each object point cloud $P_i$ and the 3D asset database\n- Process:\n    - Estimate initial pose $(p_i, s_i, θ_i)$ via PCA\n    - For candidate assets, apply ICP to find the best alignment\n    - Select the asset with the lowest RMSE\n- Output: A scene composed of aligned 3D assets\n\n(E). Final Scene Refinement\n\n- Input: The composed scene of aligned assets\n- Process:\n    - Apply gradient-based optimization to object placements\n    - Minimize total loss $L_{total} = L_p + λ_o L_o + λ_b L_b$\n    - Enforce collision avoidance and physical plausibility\n- Output: The final, collision-free, and physically valid 3D scene $S$ = {$\\{o_1, ..., o_N\\}$}\n\nWe are confident that the architectural figure, based on this description, will significantly improve the clarity of the methodology. Additionally, we will release the full codebase, enabling readers to explore the system implementation in detail."}, {"review_id": "QgAzts6m5e", "reviewer": "Reviewer_ccSn", "summary": "This paper proposes VIPScene, a framework that uses video generation, 3D reconstruction followed by segmentation and asset retrieval to create 3D scenes. Specifically, the model can take in text or image as input, then feeds the input through video generation to obtain a video of the scene. The video is then constructed into 3D reconstruction via Fast3r. The paper then uses GroundedSAM and Mast3r to detect and track multiple 3D instances within the scene. Each of them is then used to retrieve an asset from the objaverse dataset, and aligned, to create the final 3D scene.\n\n\nThey further proposed a FPVScore as a metric utilizing VLMS to better evaluate the generated 3D scene. The framework is compared with recent methods like Holodeck.", "questions": "Questions which I think are more important to address:\n\n- Analyses of the FPVScore (as listed in the previous section under Experiments)\n- The generalization capability/limitation of the current asset retrieval method on finding similar textures/material. It would be great to compare with Holodeck regarding this.\n    - Addressing the limitation of image prompting could also be helpful.\n- Can the current refining stage be replaced by simply resizing/minimising each of the objects until they no longer touch one another? If possible a score for comparison regarding this would be great.\n- More results on complex prompts as input text.", "limitations": "Some limitations are addressed in the Appendix. Additional limitations mentioned in the above two sections would be great if included.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Figure 1 is very clear, allowing a very easy understanding of the entire VIPScene\n- Analysis of Section 5.3 and Figure 5 are great additions to the paper. It provides great understanding (both quantitative and qualitative) on how the metric could be helpful compared to previous scores.\n- Ablation (Figure 6) is very helpful in understand the importance of different parts of the component.\n- Writing is clear and easy to follow.\n\n**Weaknesses/Questions**\n\nWhile the method design at a high-level is generally convincing, and I think the paper does include most of the important experiments, I believe there are still a few aspects that can improve the quality of the paper, both in terms of the Method and Experiments sections:\n\nMethod\n\n- Using Video Generation Model: Since text/image are the only inputs to the video generation, there is no constraints where additional subjects appear in the scene or objects within the scene moves around. Is this a potential limitation? Or does this happen very rarely for interior designs which VIPscene focuses on?\n- Object Detection: I enjoy the insights provided on GroundedSAM being better than models like Mask3D. Are there any ablations available on the number of frames required to gain good results using GroundedSAM for this particular setup? Are all frames fed in as inputs?\n- 3D Asset Retrieval: How big is the asset database (the paper suggests a subset of Objaverse)? Is it the same dataset as Holodeck? It would also be great to understand the time it requires for the retrieval and whether this is something scalable as the number of assets in database increase. Also the current metric for asset retrieval seems like only taking into consideration the geometry of the point cloud with the asset in the database, disregarding colour/texture information which the initial video provides. This might lead to the generated scene not adhering to the description of the prompt (which may contain information regarding textures, materials etc).\n- Object Pose Refinement: For pose refinement, and correct me if I am wrong, only the position of the object (i.e, translation) is optimized — the optimization did not take into consideration rotation or resizing. This may still lead to overlapping in cases where the optimal position still causes overlap (e.g, one object around a corner, the other object surrounding it). Why not just reducing the size until no overlap instead of a gradient-descent based approach? I am assuming there might not be much edits necessary given that the objects from the asset library are more or less in reasonable sizes and the prompt isn’t something unreasonable.\n\nExperiments\n\n- Figure 3: It would better to provide the actual prompts used to generate the scenes. I assume each row shares the same prompt. Or is the prompt just the single words used as title, i.e.,  “Kitchen”, “Living Room”, etc?\n- Figure 4: I am hesitant on the results shown here, since the color looks very different to the original image prompt.\n- Most of the prompt presented in the paper are relatively simple (i.e., only explains what is in the scene, but no description regarding the scene). It would be helpful to include additional qualitative results on more complex text prompts to understand the robustness of the current method.\n- FPVScore: How consistent is it across multiple tries of a MLLM model? How consistent is it across different MLLM models? How important is the prompt design? Ablation here would be greatly beneficial for better understanding regarding the metrics.\n- Less important question: Would text-to-image then image-to-video lead to better results compared to directly text-to-video? Text-to-image currently performs better at adhering to the details of the prompt regarding spatial relationship and finegrained information like materials/textures. It would be great (though not necessary) to have a comparison regarding this.\n\nOverall, I think this paper does have its merits, but it might need strengthening on some of its figures and some additional studies listed above to better understand the models generalization capability.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, providing interesting insights to where its needed. I am more confident now that this paper is worth accepting with the extensive experiments.", "rebuttal": "We would like to thank the reviewer for their time and positive feedback. We’re glad they found the **analysis insightful**, the **ablation study helpful**, and the **figures clear and effective**. \n\nWe address all concerns below and believe the revisions significantly improve the manuscript’s clarity and quality. We welcome any further suggestions. Thank you.\n\n---\n\n### 1. **Additional and Dynamic Objects in Video Generation Models.**\n\nThe reviewer wonders whether video generation causes unexpected object appearances or movements in the scenes. We address the concern in two parts: the **additional objects not specified in the input prompt**, and the **movement of objects within the scene**.\n\nFirstly, the generated video may contain objects not explicitly mentioned in the prompt. However, these additional objects generally appear in locations **consistent with commonsense spatial relationships** owing to strong priors encoded in the video model. If certain objects are undesired, they can **be excluded by incorporating them into the video generation model’s negative prompt**.\n\nSecondly, we observe that state-of-the-art video generation models such as Cosmos and Kling **rarely generate dynamic objects in indoor scenes**. This behavior likely results from their training data, which predominantly consists of architectural visualizations and real-estate tours characterized by static, person-free environments. When generating outdoor scenes, dynamic objects (e.g., pedestrians) occasionally appear. While this may pose a limitation, it can be addressed by **incorporating dynamic or 4D scene reconstruction methods** [1,2], which explicitly detect and remove dynamic objects during reconstruction.\n\n---\n\n### 2. **Object Detection.**\n\nThe reviewer is curious about frame usage for object detection. We analyzed the effect of varying the number of video frames used as input to Grounded-SAM. Grounded-SAM achieves accurate detection on each frame. However, the number of frames mainly influences the quality of the extracted object point clouds. When fewer frames are used, the reconstructed point clouds tend to be incomplete, which can result in inaccurate retrieval of object sizes.\n\nOur results show that increasing the number of frames improves performance up to a saturation point. In our experiments, we sampled the video at 2 frames per second, using approximately 10 frames, which provided a good trade-off and satisfactory results.\n\n| Number of Frames | FPVScore |\n| --- | --- |\n| 5 | 2.02 |\n| 10 | 2.50 |\n| 20 | 2.48 |\n\n---\n\n### 3. **3D Asset Retrieval.**\n\nThe reviewer asks about the asset database size, retrieval time, and scalability. We use the same high-quality subset of Objaverse as Holodeck, which contains over 50k 3D assets.\n\nOur retrieval process is highly efficient. We first leverage pre-computed embeddings to compute both text and visual similarity, enabling retrieval over the entire 50K asset database in approximately 0.02s. Even when scaling the database by a factor of 20, retrieval remains fast at around 0.06s. Following this initial step, we perform ICP optimization only on the top 30 candidate assets, keeping the overall process efficient. The full retrieval and refinement pipeline takes approximately 0.1s per object.\n\n---\n\n### 4. **Texture and Material Properties & Image Prompt.**\n\nThe reviewer is curious about how texture and material attributes are handled in asset retrieval, and about the color differences between the image prompt and generated scenes. Indeed, our previous retrieval strategy primarily focuses on **object category and the geometry of the point cloud**, as our main goal has been to **ensure the plausibility of the object layout** in the reconstructed scene. However, we agree that texture and material attributes are also important. To address this, we conducted additional experiments that **leverage CLIP image features extracted from cropped object regions in the video frames**. This approach allows us to retrieve assets that better match the textures and materials specified in the text/image prompt. We have added a visualization of this result in the revised manuscript.\n\nWe also conducted a comparison with Holodeck. In our evaluation, **our method demonstrated better performance in retrieving assets that align with the intended textures and materials**. This is because our approach directly utilizes visual cues from the video frames, whereas Holodeck depends on text-based retrieval, which lacks the same level of visual specificity. Additionally, our framework benefits from vision priors provided by the video model, leading to a more accurate spatial arrangement of objects. In contrast, Holodeck translates visual context into textual constraints, which often results in less precise spatial understanding.\n\n| Method | FPVScore |\n| --- | --- |\n| Holodeck | 1.75 |\n| Ours | 1.93 |\n\n---\n\n### 5. **Object Pose Refinement.**\n\nThe reviewer asks whether our pose refinement method is based on translation and how it compares to resizing-based refinement. Yes, our pose refinement primarily focuses on object translation, preceded by a discretization of yaw rotations to 0°, 90°, 180°, or 270° to enhance the visual plausibility of the final scene. This optimization is typically sufficient to **eliminate overlaps and produce physically reasonable arrangements**, as the **refinement is performed in a** **joint manner across all objects**. \n\nWe fully agree that resizing objects could also be a valid strategy. We performed a comparison between resizing-based refinement and our translation-based optimization, and observed that both approaches yield similarly effective results in terms of overlap removal.\n\n| Pose Refinement | FPVScore |\n| --- | --- |\n| Resizing | 2.42 |\n| Ours | 2.45 |\n\n**Our choice was primarily driven by practical considerations**. The environment we use, AI2-THOR, does not provide a native API for resizing objects. To enable object scaling, we would need to compute the appropriate scaling factors, manually rescale the 3D meshes, and re-import them into AI2-THOR as new assets for rendering. Given these additional steps, our translation-based optimization offers a more efficient and easily integrated solution, while still producing satisfactory results.\n\n---\n\n### 6. **Prompts of Figure 3.**\n\nEach row in Figure 3 was generated using the same text prompt, which specifies the room type and desired objects in the room, across all three methods. We have updated the figure caption in the revised version.\n\n---\n\n### 7. **More Complex Text.**\n\nThe reviewer requests more results with complex prompts. We fully agree that the ability to handle more complex scene descriptions is crucial. We conducted additional experiments using prompts that specify **spatial relationships** and **scene-level attributes**. Our results demonstrate that VIPScene can effectively generate scenes based on these more intricate descriptions (e.g., *\"a desk placed under a window with a bookshelf to its left\"*, or *\"a modern living room with a gray sectional sofa facing a fireplace, a coffee table in between, and a floor lamp placed diagonally behind the sofa\"*). This capability is largely attributed to the commonsense priors learned by the video generation model.\n\nWe believe that as video models continue to advance, our method will be able to handle more sophisticated prompts. These new qualitative results have been added to the experimental section of the paper.\n\n---\n\n### 8. **Analysis of FPVScore.**\n\nThe reviewer asks about a deep analysis of FPVScore’s consistency. We address the three aspects of FPVScore reliability as follows:\n\n**Consistency Across Tries**: We repeatedly queried each model and computed the average Kendall’s Tau across trials. The results show that MLLMs yield stable outputs, with GPT-4o demonstrating the highest consistency:\n\n| Model | $\\tau$ |\n| --- | --- |\n| Gemini 2.0 | 0.61 |\n| GPT-4o | 0.74 |\n| GPT 4.1 | 0.71 |\n\n**Consistency Across Models**: We assessed how different MLLMs agree with each other by measuring pairwise Kendall’s Tau correlations between their scene rankings. The results indicate that the models arrive at similar relative judgments:\n\n| Model | $\\tau$ |\n| --- | --- |\n| Gemini 2.0 & GPT-4o | 0.54 |\n| Gemini 2.0 & GPT 4.1 | 0.47 |\n| GPT 4.1 & GPT-4o | 0.60 |\n\n**Prompt Design**: We conducted an ablation study by removing the criteria focus and the room analysis instructions from the prompt. While modern MLLMs, by their inherent reasoning capabilities, can still produce reasonably accurate judgments, the use of a structured and explicit prompt results in stronger alignment with human preferences, as reflected in higher correlation:\n\n| Prompt | $\\tau$ |\n| --- | --- |\n| Ours | 0.39 |\n| w/o Criteria and Analysis Instruction | 0.31 |\n\n---\n\n### 9. **Text-to-Image then Image-to-Video.**\n\nThe reviewer is curious whether text-to-image followed by image-to-video might outperform direct text-to-video. We conducted a comparison between them. We observed that **the text-to-image approach can produce more accurate details**, particularly in terms of fine-grained attributes of objects. \n\nHowever, we also found that **recent video generation models are increasingly capable of handling fine details directly from text**, reducing the performance gap between the two approaches. In addition, the **text-to-image approach can sometimes introduce suboptimal initial views**, which may negatively impact the generation of subsequent video frames. \n\nWe agree that this is a promising direction for future exploration. One possible approach is to incorporate viewpoint constraints directly into the prompt to guide the image and video generation process more effectively.\n\n[1] Monst3r: A simple approach for estimating geometry in the presence of motion. Zhang et al. ICLR 2025.\n\n[2] Continuous 3d perception model with persistent state. Wang et al. CVPR 2025."}]

Author rebuttal:

Meta-review: This paper introduces VIPSCENE, a novel framework for 3D scene synthesis that leverages video perception models. Specifically, by using video generation models, 3D reconstruction and perception models, it aims to generate realistic and structurally coherent 3D environments from text or image prompts.

The paper makes contributions regarding: (1) A novel paradigm (VIPScene) that leverages commonsense knowledge from video generation models for coherent 3D scene synthesis. (2) FPVScore, a first-person, human-aligned evaluation metric to serve the research community. (3) Strong experimental validation, including comprehensive ablations and user studies.

In the meantime, the reviewers pose questions regarding the computational Complexity of the proposed framework, the generation of small objects that are not easy to detect, and the potential of generating physically implausible objects. Several reviewers believe that the technical contributions can be strengthened.
Decision: Accept (poster)