SceneScape: Text-Driven Consistent Scene Generation
Authors: Rafail Fridman
Amit Abecasis
Yoni Kasten
Tali Dekel
Venue: NeurIPS 2023 poster (2023)
Keywords: Computer Vision
Image & Video Editing
Video Generation
Perpetual View Generation
Texture Synthesis & Inpainting

Abstract: We present a method for text-driven perpetual view generation -- synthesizing long-term videos of various scenes solely, given an input text prompt describing the scene and camera poses. We introduce a novel framework that generates such videos in an online fashion by combining the generative power of a pre-trained text-to-image model with the geometric priors learned by a pre-trained monocular depth prediction model. To tackle the pivotal challenge of achieving 3D consistency, i.e., synthesizing videos that depict geometrically-plausible scenes, we deploy an online test-time training to encourage the predicted depth map of the current frame to be geometrically consistent with the synthesized scene. The depth maps are used to construct a \emph{unified} mesh representation of the scene, which is progressively constructed along the video generation process. In contrast to previous works, which are applicable only to limited domains, our method generates diverse scenes, such as walkthroughs in spaceships, caves, or ice castles.

SceneScape: Text-Driven Consistent Scene Generation
Rafail Fridman"
Amit Abecasis
Weizmann Institute Science rafail.. f zmann. ac. il
Weizmann Institute Science amit abecasis@we izman ac ac.il
Yoni Kasten
Tali Dekel
NVIDIA Research
Weizmann Institute of Science
ykasten@nvidia.cmm com
tali www.edeeeeeeeeeeemee ac i1
"POV, pools, water, dark inside beautsful scenery, best quality" (a) Sample frames from our generated video
(b) Our unifled representation
Figure (a) Given text prompt describing seene, SceneScape generates a long-tern vidco depicting a walkthrough of the scene, adhering to given camera trajectory. We achieve this through a test-time optimization approach that utilizes pre-trained text-to-image generative model and depth prediction model, enabling online 3D-consistent perpetual view generation without the need for training data. (b) To synthesize 3D-consistent video, we construct a unified 3D mesh concurrently with the video. visualization purposes, we remove side part of the mesh.
Abstract
We present method for text-driven perpetual view generation synthesizing long-term videos of various scenes solely from an input text prompt describing the scene and camera poses. We introduce novel framework that generates such videos in an online fashion by combining the generative power of pre- trained text-to-image model with the geometric priors learned by pre-trained monocular depth prediction model. To tackle the pivotal challenge of achieving 3D consistency, i.e., synthesizing videos that depict we deploy an online test-time training to encourage the predicted depth map the current frame be geometrically consistent with the synthesized scene. The depth maps are used to construct a unified mesh representation the scene, which is progressively constructed along the video generation process. In contrast previous works, which are applicable only to limited domains, our method generates diverse scenes, such walkthroughs in spaceships, caves, ice castles, Project page: https://scenescape.ggiie github.io/
scenes,
Introduction
By observing only single photo of a scene, we can imagine what it would be like to explore it, move around, or turn our head and look around we can envision the 3D world captured the image and expand it in mind. However, achieving this capability computationally generating long-term video that depicts plausible visual world from an input image poses several key challenges. One of the main challenges is ensuring that the synthesized content consistent with feasible 3D world. For example, it must correctly depict parallax effects and occlusion relations of different objects in
"qqual contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the scene. Additionally, to synthesize new content, strong prior about the visual world and how it would appear beyond the current field of view is necessary. Finally, the generated content should appear smooth and consistent over time.
Previous methods have tackled the task of perpetual view generation for specific domains, for example, synthesizing landscape flythroughs 5. 28 29 or bedrooms walkthroughs (7] These methods involve large-scale training on videos images from the target domain, which limits their use. In this work, inspired by the transformative progress in text-to-imaae generation, we take different route and propose a new framework for text-driven perpetual view generation - synthesizing long-range videos of scenes solely from rree-occbullary text describing the scene and camera poses. Our method does not require any training data but rather generates the scene zero-shot manner by harnessing the generative prior learned by pre-trained text-to-image diffusion model and the geometric priors learned by pre-trained depth prediction model.
More specifically, given an input text prompt and a camera trajectory, our framework generates the video in an online fashion, one frame at a time. The text-to-image diffusion model is used to synthesize new content revealed as the camera moves, whereas the depth prediction model allows us estimate the geometry of the newly generated content. To ensure that the generated scene adheres to feasible 3D geometry, our framework estimates a unified mesh representation of the scene that constructed progressively as the camera moves in the scene. Since monocular depth predictions tend to be inconsistent across different frames. we finetune the depth prediction model at test time to match the projected depth from our mesh representation for the known, previously synthesized content. This results in a test-time training approach that enables diverse, 3D-consistent perpetual view generation.
To summarize, our key contributions are the following:
1. The first text-driven perpetual view generation method, which can synthesize long-term videos of diverse domains solely from text and camera trajectory.
2. The first prro-shotttes--tmms 3D-consistent perpetual view generation method, which synthe- sizes diverse scenes without large-scale training on a specific target domain.
3. Achieving 3D-consistent generation by progressively estimating a unified 3D representation of the scene.
We thoroughly evaluate and ablate our method, demonstrating significant improvement in quality and 3D consistency over existing methods.
2 Related Work
Perpetual view generation. Exploring an "infinite scene" dates back to the seminal work of Kaneva et al. [24] which proposed an image retrieval- based method to create 2D landscape that follows a desired camera path. This task has been modernized by various works. Different methods proposed to synthesize scenes given as input single image and camera motion [e.g..
21 26 28, 29 45 46, [66] 58 63). For example, synthesizing indoor scenes 266 45
46 [88] or
long-range landscapes flythroughs 28 29]. These methods have demonstrated impressive results and creative use of data and supervuuyy signals 57 12 28, 45). However, none of these methods considered the task of text-driven perceptual view generation and rather tailored model for a specific scene domain. Thus, these methods require extensive training on a large-scale dataset to learn proper generative and geometric priors of specific target scene domain.
3D-Consistent view synthesis from a single image. Various methods have considered the task of novel view synthesis from single input image. A surge of methods has proposed utilize NeRF 35 as a unified 3D representation while training on large dataset from a specific domain to learn a generic scene prior 22
60
62.. Other 3D representations have been used by various methods,
including multi-plane plane images 57 66,, point clouds which hold high-dimensione features [46] (88,, In Jain et al. [22], the task further regularized using the 41 encouraging high similarity between the input image and each novel view in CLIP image embedding space. All these methods require training a model on specific domains and cannot produce perpetual exploration of diverse scenes.
or textured-nen meeeh representation 20
semantic prior learned by CLIP

---

Bepallttng Madi
I+41
"alleberossh musiom Mastirplacs
Monscular depta edimation
Projection Cz-1
Starting Mesh M
D
Di Dest
D
Updated Mesh Mes.
Figure 2: SceneScape pipeline. We represent the generated with unified mesh M, which constructed an online fashion. Given a camera Ct+1, at each synthesis step, a new frame is generated by projecting M, into Ct+1, and synthesizing the newly revealed content by using pre-trained text-to-image diffusion model. To estimate the geometry of the new synthesized content, we leverage a pre-trained depth prediction model; ensure the predicted depth is consistent with the existing scene Mr, deploy test-time training. encouraging the predicted depth by the model to match the projected depth from Mr. We then update our mesh representation to form Mr++ which includes the new scene content.
3D-Aware image generation. Related to task is 3D-aware image synthesize, which typically involves incorporating a 3D representation explicitly into a generative model, e.g., a NeRF repre- sentation [
36 55], neural surface renderer (38). These methods have shown impressive 16
results on structured domains (e.g.. cars or faces), but deploying them for diverse scenes is extremely challenging and often requires additional guidance [13]. or heavy supervision, e.g., large-scale scale data that contains accurate camera trajectories, which can be obtained reliably mainly for synthetic data []. We aim for much more diverse, flexible, and lightweight generation of arbitrary scenes.
Text-fo-video generation and editing, Our work follows a broad line of works in the field of text-to- video synthesis and editing. There has been great progress developing text-to-video generative models by expanding architectures to the temporal domain and learning priors from large-scale video datasets (e.g. 15 32 51] Nevertheless, these models are still in infancy, and are lagging behind image models in terms of quality (resolution and video length). On the other side of the spectrum, surge methods proposes tune or directly leverage 2D text-image model for video editing tasks [e.g.. 40 59 65]. We also follow this approach in conjunction with generating explicit 3D scene representation.
Text-to-3D Generation and Editing. Recently, language- mingugge-vision vision pre- pre-trained models 41
49 and
differentiable renderers (30 35 have been utilized for text-driven 3D content generation. For example, CLiP--ased losses have been used for the text-driven stylization of a given surface mesh 34 for synthesizing a NeRF from an input text 23) Taking this approach further, Poole et al. 39 distills the generative prior of pre-trained text-to-image diffusion model by encouraging the optimized NeRF to render images that are in the distribution of the diffusion model [49]. Singer et al. [52] extends this approach to 4D dynamic NERF and optimizes using pre-trained text-to-video model. Nevertheless, it is limited to object-centric scenes while we generate long-term walkthrough-tye
videos. On the other hand, Chang al. [00 and Ma al. [33 construct the semantic scene graph from natural language and retrieve the required elements from the 3D objects databases.
Finally, in very generate 3D scenes based on textual prompts. However, this work focuses specifically on creating meshes and tailors both the camera trajectory, prompts and mesh rendering for this purpose. In contrast, we use the mesh representation only as a tool to achieve videos of diverse scenes.
concurrent work, Texr2Room 19 proposed similar approach
Method
The input to our method is text prompt P, describing the target scene, and camera trajectory denoted by (C&)²=, where C, R3x1 the camera pose of the 7th frame. Our goal is to synthesize ttrm, high-quality video depicting the desired scene while adhering the camera trajectory.

---

Our framework, illustrated in Fig. generates the video one frame at time, by leveraging a pre- trained text-to-image diffusion model 48] and a pre-trained monocular depth prediction model The role of the diffusion model is to synthesize new content that is revealed as the camera moves, whereas the depth prediction model allows us to estimate the geometry of the newly generated content. We combine these two models through a unified 3D representation of the scene, which ensures geometrically consistent video generation. More specifically, at each time step t, the new synthesized content and its predicted depths are used to update unified mesh representation M, which is constructed progressively as the camera moves. The mesh then projected to the next view Ct+1, and the process is repeated.
42
43
Since neither the depth prediction model nor the inpainting model are guaranteed to produce consistent predictions across time, we finetune them both to match the content of our unified scene representation for the known, previously synthesized content. We next describe each of these components in detail.
3.1 Initialization and Scene Representation
We denote the pre-trained text-to-image inpainting model by Φ, which takes as input a text prompt P. a mask M, and a masked conditioning image I.M
I. ((M, I, P)
(I)
The first frame in our video To is generated by sampling an image from Φ. using P without masking (all-ones mask). The generated frame is then fed to a pre-trained depth prediction model g. which estimates Do the depth map for Lo. i.c., Do = goo)..
With the estimated depth and camera poses, naive approach for rendering Ie+1 is to warp Le to Ct+1. e.g., using 2D splatting (37] However, even with consistent depths, rendering a 3D-consistent scene is challenging. In typical camera motion, multiple pixels from frame are mapped into the same pixel in the next frame. Thus, the aggregated color needs to be selected carefully. Furthermore, frame-to-free framee warping lacks a unified representation, thus, once the scene content gets occluded, it will be generated from scratch when it gets exposed again
To address these challenges. we take a different approach and represent the scene with unified triangle mesh M = (V, F, E)) which represented by a tuple of vertices (3D location and color), faces and edges, respectively. We initialize Mo by unprojecting (No, Do). After the first frame is generated, each synthesis step consists of the following main stages:
3.2 Project and Inpaint
The current scene content, represented by the unified mesh representation Mr, is projected to the next camera C+41:
(1 DM M++1) Proj (Me: C++1)
(2)
producing a mask M4+1 of the visible content in C.+1, masked frame 1. and a masked depth map D Given M,11 and JM we now turn to the task of synthesizing new content. To do so, we apply d using Eq.
3.3 Test-time Depth Finetuning
We would like to estimate the geometry of the new synthesized content and use it to update our unified scene mesh. A naïve approach to do is to estimate the depth directly from the inpainted frame, i.e., De+1 = -g g( ). However, monocular depth predictions tend to be inconsistent, even across nearby video frames (27 31 64]. That is, there is no guarantee the predicted depth D++1 would be well aligned with the current scene geometry Mr. We mitigate this problem by taking a test-time training approach to finetune the depth prediction model to be as consistent as possible with the current scene geometry. This approach is inspired by previous methods (27 31 64 which encourage pairwise consistency w.r.t optical flow observations between the predicted depth and our unified 3D representation in the visible regions in frame t + 1. Formally, our training loss given by:
given video. In contrast, we encourage consistency
Ldisp = D.M t+l g((e+z) Me+1

---

where D 1+1 M6+1 are given by projecting the mesh geometry to the current view (Eq. 2 To avoid catastrophic forgetting of the original prior learned by the depth model, we revert g to its original weights each generation step.
3.4 Test-time Decoder Finetuning
Our framework uses Latent Diffusion inpainting model [48]. in which the input image and mask are first encoded into a latent space ELDM(Mr. " the inpainting operation is then carried out in the latent space, producing 2, and the final output image is given by decoding to an RGB image: (2) This encoding/decoding operation is lossy, and the encoded image is not reconstructed exactly in the unmasked regions. Following Avrahami et al. (2), for each frame, we finetune the decoder as follows:
(4) The first term encourages the decoder to preserve the known content, and the second term is prior preservation loss 2 As in Sec. 3.3 revert the weights of the decoder to its original weights.
de PLDM @ M441 12 + ([ 1
D (2)) @ (1 M
3.5 Online Mesh Update
Given the current mesh M,, and the inpainted frame 4++, we use our depth map and camera pose to update the scene with the newly synthesized content. Note that in order to retain the previously synthesized content, some of which may be occluded in the current view, we consider only the new content, unproject, and mesh it, adding it to Mr. That is,
M UnProj (M+11, I441, D441, C++1)
(5)
Finally, Met1 is merged into the mesh:
Me+1 = M, M (V, J UV F,UFFI1.11 E(+1)
(6)
In practice, we also add to the mesh the triangles used to connect the previous content to the newly synthesized one. See Sec. A.1 of appendix for more details.
3.6 Rendering
Rendering high-quality and temporally consistent video from our unified mesh requires careful han- dling. First, stretched triangles may often be formed along depth discontinuities. These triangles encapsu- late content that is unseen the current view, which we would like to inpaint when it becomes visible other views. To do so, we automatically detect stretched triangles based their normals and remove them from the updated mesh.
Boating
warped. inpainted)
Another issue arises when the content at the border of the current frame is close to the camera relative to the scene geometry. This content is typically mapped towards the interior of the next frame due to parallax. This often results in undesirable "floating artifacts" where the revealed border regions are rendered with the existing distant content (see Fig. 3 We assume that such regions should be inpainted, thus we auto- matically detect and include them in the inpainting mask. See Sec. A.2 of appendix for more details.
Doating
warped, right inpainted)
Figure Handling floating artifacts
Results
We evaluate our method both quantitatively and qual- itatively various scenes generated from diverse of prompts, including photorealistic well

---

dimiy Lit Library, with upon leather-bound books dark wooden shelves"
"A grand, marble staircase spirals up to a vaulted ceiling in grand entrance hall of palace."
"POV, haunted house, dark, wooden door, spider webs, skeletons"
simple Museum, pictures, paintings, artistic, best quality climly Lit"
"inside castle of ice, beautiful photo, masterplece"
walkthrough, scl-ft ship interiors, corridorz, qualiby, masterpiece, beautiful scenery, best qualiby
"walktkrough, inside medieval forge, metal, fire, beautiful photo, masterpiece"
Figure 4: Sample frames from generated videos. Our method generates high-quality and diverse scenes while following desired camera trajectory. All videos included in SM.
as imaginary scenes. We generated 50-frame long videos, with a fast-moving camera motion, which follows a backward motion combined with rotations of the camera (see Sec. A of appendix for more details). Since our method represents the scene via mesh, we focus mainly on indoor scenes. Synthesizing outdoor scenes would require careful handling in cases of dramatic depth discontinuities (as discussed in Sec. 5)
Figure and Fig. 4 show sample frames from our videos. The results demonstrate the effectiveness
scenes, depicting significant parallax, complex structures, and various complex scene properties such as lighting or diverse materi- als ge. ice). Additional results, videos, and output depth maps are included in the Supplementary materials (SM).
of our method in producing high-quality and esometrically-plassii

---

Frame
COLMAP Our Depth
(a) Full method
depth finetuning
decoder finctuning
mesh
Figure 5: Sample ablation results of (a) our full method, (b) w/o depth finetuning (Sec. 3.3 (c) w/o decoder
finetuning (Sec. 3.4 and (d) w/o mesh (Sec. (.1c.3.1). We present sample frames, output depth maps, and COLMAP's depth maps. As seen, all components are essential for ohtaining high-quality and 3D-consistent videos.
Rot. (deg.)
Trans. (%)
Reproj. (pix) SI-RMSE Density (%) f CLIP-AS AMT(S))
Ours w/o depth f.
0.61
0.01
0.33
0.91
5.85
2.24
0.02
0.37
0.41
0.90
5.75
52.7
decoder f.
2.81
0.02
0.82
0.17
0.87
5.74
63.0
w/o mesh
3.55
0.04
0.57
0.18
0.78
5.73
69.1
Warp-inpaint
4.89
0.09
0.97
0.70
5.43
85.6
Table 1: Ablations. For each ablation baseline, report, from left to right: accuracy of estimated camera poses, reprojection error, depth SI-RMSF, and the percentage of reconstructed video pixels. We also report CLIP aestheties score [ and measure visual quality using AMT user study. where we report the percentage of judgments in favor of our full method. See Sec, 4.1 and Sec. 4.2 for more details.
4.1 Metrics
In experiments, we use the following metrics for evaluation:
Depth consistency. We evaluate the depth consistency of video by applying COLMAP, using known camera poses, and measuring the Scale-Invariant Root Mean Square Error (SI-RMSE) 14
SI-RMSE
Σ P.qEZ
[(log (p) log (q)) (log (p) log (q)))
(7)
N
where D and D are our output depth maps and COLMAP depths, respectively. We compute the SI-RMSE for each frame and report the average per video. To assess the completeness of reconstruction, we also measure the percentage of pixels recovered by COLMAP, i.e., passed its geometric consistency test. In addition, report the average reprojection error of COLMAP reconstruction, which indicates the consistency of the video with COLMAP's point cloud.
Pose accuracy. We measure the accuracy of the estimated camera trajectory. Specifically, provide COLMAP only with intrinsic parameters and align the two camera systems by computing a global rotation matrix [61]. We measure the normalized mean accuracy of camera positions (in percentage, relative to the overall camera trajectory), ), and camera rotations (in degrees).
Visual quality. We use Amazon Mechanical Turk (AMT) platform for evaluating the visual quality of our videos compared to the competitors. We adopt the Two-alternative Forced Choice (2AFC) protocol, where participants are presented with two side-by-side videos, ours and the competitor's 's (at random left-right ordering) and are asked: "Choose the videofimage that has better visual quality, e.g., sharper, less artifacts such as holes, stretches, or distorted geometry". Additionally, we compute the CLIP aesthetic score [1] an image aesthetic linear predictor learned on top of CLIP embeddings.
7

---

Vidted
"Medieeal castle
"Candy house"
"Opulent hobel"
Figure 6: Baseline comparison. Sample results from our method (top), GEN-1 15 (middle) and VideoFu- sion 32 (bottom) on 3 different prompts. Other methods exhibit saturation and blurriness artifacts in addition temporally inconsistent frames.
4.2
Ablation
We ablate the key components in our framework: (i) depth finetuning (Sec. (Sec.3.3) (.3), (ii) decoder finetuning (Sec. 3.4), and (iii) our unified mesh representation (Sec. 3.1 For the first two ablations, apply our method without each component, and for the third, we replace our mesh with 2D rendering. i.e., warping the content from the current frame the next, using depth-awan splatting [37. In addition, we consider the baseline of simple (w/o finetuning). We ran these ablations on 20 videos, using set of 10 diverse text prompts. See Sec. for the list of prompts and SM sample vidcos.
i.c. using splatting with the original estimated depth
Table reports the above COLMAP metrics (see Sec. 4.1). Our full method results in notably more accurate camera poses, denser reconstructions, and better-aligned depth maps. Excluding depth finetuning significantly affects the depth consistency (high SI-RMSE). Excluding decoder finetuning leads to high-frequency temporal inconsistency, which leads to higher reprojection error. Without mesh, the reconstruction is significantly sparser and we observe drop in all metrics. These results are illustrated in Fig. for a representative video.
To assess visual quality, we follow the AMT protocol described in Sec. 4.1 where we ran each test w.r.t. one the baselines. We collected 4,000 user judgments from 200 participants over 20 videos. As seen in Table 1 the unified mesh plays major role in producing high-quality videos, where the judgments were ambiguous w.r.t w/o depth finetuning baseline. We hypothesize this is because this baseline still leverages the mesh representation, thus the visual artifacts are temporally consistent, making them more difficult to perceive.
4.3 Comparison to Baselines
To the best of our knowledge, there is no existing method designed for our task text-driven perpetual view generation. We thus consider the most relevant methods our setting:
VideoFusion (VF) (32]. A text-to-video diffusion mode which takes input text prompt and produces a 16-frame long video at 256 x 256 resolution.
GEN-1 15]. A text-driven video-to-video translation model, where the translation is
conditioned on depth maps estimated from an input source video. Note that in our setting the scene geometry is unknown and is constructed with the video, whereas GEN-1 takes the scene geometry input, thus tackling simpler task.
Due to the inherent differences between the above methods, we follow a different evaluation protocol comparison with each one of them.
TTo the best of our knowledge, this is the only publicly available text-to-video model.

---

Rot. (deg.)
Trans. (%) 1
Reproj. (px)
Density (%
CLIP-TS
AMT (%) f
Ours VF
0.29
81.4
0.25
95.97
0.78
46.36
0.22
Ours
1.71
3.94
0.60
91.72
0.26
69.34
GEN-1
2.47
8.09
1.64
53.96
0.26
Table 2: Comparison baselines. (top) VideoFusion [2] and (bottom) GEN-1 [TS]. We evaluate 3D consistency using COLMAP metrics and measure visual quality using AMT user study, where we report percentage of judgments in our favor (see Sec. 4.2). Our method outperforms the baselines on all metrics. Note that the experiential setup for these two baselines is different (see Sec. 4.3
Figure 7: Limitation. Outdoor scenes are poorly represented with our mesh representation (left). Occasionally we may observe quality degradation due to error accumulation (right).
4.3.1 Evaluating 3D-consistency
To compare to VF, we used the same set of prompts as in Sec. Sc2. as input to their model and generated 1000 videos. We evaluate the 3D consistency of VF's videos and ours, using the same metrics in Sec. where we used COLMAP without any camera information, and downsampled our vidcos to match their resolution. As seen in Table 2 our method significantly outperforms VF in all metrics. This demonstrates the effectiveness our mesh representation in producing 3D-consistent videos, in contrast VF, which may learn geometric priors by training large-scal video dataset. Note that since VF does not follow a specific camera path, measuring its camera pose accuracy is not possible.
GEN-1 is conditioned on source video, while our method takes camera trajectories as input. Thus, we used the RealEstate 10K dataset [66]. consisting of curated Internet videos and corresponding camera poses.
We automatically filtered 22 indoor videos that follow smooth temporal backward camera motion to adapt it to our method's 's setting. The camera poses are used as input to our method, whereas the RGB frames are used as input video to GEN-1 while supplying both methods with the same prompt. We generate 1 10 videos" using five different prompts from our ten prompts set that allow reasonable editing of house (such as "library" rather than "cave"). See Sec. of appendix for more details.
We measure the video's 3D consistency metrics, as discussed Sec. 4.2. and report them in Table 2 Our method outperforms GEN-1 in all metrics, even though the geometric structure of the video (depth maps) was given to GEN-1 as input.
4.3.2 Visual Quality and Scene Semanties
We conducted an extensive human perceptual evaluation as described in Sec. 4.1 We collect 3,000 user judgments over 60 video pairs from 50 participants. The results of this survey w.r.t. each method are reported in Table 2 As seen, when compared to VF, our method was preferred in 96% of the cases and in 69% of the cases comparison with GEN-1. Sample results of those comparisons can be seen in Fig. [ Note that since GEN-1, VF, and StableDiffusion do not exhibit the same underlying data distribution, existing visual quality metrics such FID 17 cannot be used.
To measure how well our generated videos adhere to the desired scene, we measure the CLIP text- similarity score mean cosine similarity between the prompt and each of the frames. As seen in T our method follows the scene text prompt more closely than VF and is on par with GEN-1.
'There is only limited UI access GEN-1, which restricts the scale of this experiment compared to VE.

---

Discussion and Conclusion
We posed the task of text-driven 3D-consistent perpetual view generation and proposed test-time optimization approach to generate long-term videos of diverse scenes. Our method demonstrates two key principles: (i) how to harness two powerful pre-trained models, allowing us to generate scenes in zero-shot manner without requiring any training data from specific domain, and (ii) how to combine these models with unified 3D scene representation, which by construction ensures feasible geometry, and enables high quality and efficient rendering. As for limitations, the quality of our results depends on the generative and geometric priors and may sometime decrease over time due to error accumulation (Fig. right). In addition, since we represent the scene with triangular meth, is difficult for our method to represent dramatic depth discontinuities, e.g., sky vs. ground in outdoor scenes (Fig. leftt. Moreover, since our method works in an online fashion and does not change the previously generated frames, the errors that appeared in previous frames can propagate to the next frames. Devising new 3D scene representation tailored for perpetual view generation of arbitrary scenes is an intriguing avenue of future research. We believe that the principles demonstrated in our work hold great promise in developing lightweight, 3D-aware video generation methods.
Acknowledgments
We thank Shai Bagon for his insightful comments. thank Narek Tumanyan for his help with the website creation. We thank GEN-1 authors for their help in conducting comparisons. This project received funding from the Isracli Science Foundation (grant 2303/20).
10

---

References
[1] LAION AL
Clip aesthetic score predictor. aesthetic casthetic-predictorr predictor 2022. Accessed: 2023-01-25.
attps://githul https://ggcccau...cmmoAOooOO Com/LAIIONAA
[2] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. CoRR,
abs/2206.02779. 2022. doi: 10.48550/m2X1..2266077777 URL.https://ai. org/10. 485500
arXiv. 2201..2206.022. 02779
[3] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Tex(2live: Text-driven layered image and video editing. In ECCV, 2022.
[4] Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, et al. Gaudi: A neural architect for immersive 3d scene generation. arXiv preprint arXiv:2207.1375.1 2022.
[5] Sheng Cai, Eric Chan, Songyou Peng, Mohamad Shahbazi, Anton Obukhov, Luc Van Gool, and Gordon Wetzstein. Diffdreamer: Consistent single-view perpetual view generation with conditional diffusion models. ArXiv, 0tss/2211.121311 2022.
[6] Duygu Ceylan, Chun-Hao Huang, and Niloy Mitra. Pix2video: Video editing using image diffusion. arXiv:2303. /2688, 2023.
[7] Lucy Chai, Richard Tucker, Zhengqi Li, Phillip Isola, and Noah Snavely. Persistent nature: A generative model of unbounded 3d worlds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023.
[8] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. CVPR, 2021.
[9] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay. Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022.
[10] Angel X. Chang, Manolis Savva, and Christopher D. Manning, Learning spatial knowledge for text to 3d scene generation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, EMNLP, pages 2028-2033 ACL., 2014.
[11] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Text2light: Zero-shot text-driven HDR panorama generation. ACM Trans. Graph., 41(6):15:11195:16 2022.
[12] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Scenedreamer: Unbounded 3d scene generation from 2d image collections. In arXiv, 2023.
[13] Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 14304-14313, 2021.
[14] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using multi-scale deep network. Advances in neural information processing systems, 2014.
[15] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011, 2023.
[16] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. ArXiv, abs/2110.08985.. 2021.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

---

[18] Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans, Imagen video: High definition video generation with diffusion models. CoRR, abs/2210.02303, 2022.
[19] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. arXiv preprint MrxXiv22303.11990 2023.
[20] Ronghang Hu, Nikhila Ravi, Alexander C. Berg, and Deepak Pathak, Worldsheet: Wrapping the world in a 3d sheet for view synthesis from single image. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
[21] Ronghang Hu, Nikhila Ravi, Alexander C. Berg, and Deepak Pathak. Worldsheet: Wrapping the world in 3d sheet for view synthesis from single image. In ICCV, 2021.
[22] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 5885-5894, October 2021.
[23] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. CVPR, 2022.
[24] Biliana Kaneva, Josef Sivic, Antonio Torralba, Shai Avidan, and William T. Freeman. Infinite images: Creating and exploring large photorealistic virtual space. Proc. IEEE, 2010.
[25] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
[26] Jing Yu Koh, Honglak Lee, Yinfei Yang. Jason Baldridge, and Peter Anderson. Pathdreamer: world model for indoor navigation. ICCV, 2021.
[27] Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[28] Zhengqi Li, Qianqian Wang, Noah Snavely, and Angjoo Kanazawa. Infinitenature-zenc: Learn- ing perpetual view generation of natural scenes from single images. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27 2022, Proceedings, Part I, pages 515-534. Springer, 2022.
[29] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14458-
1 14467, 2021.
[30] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: differentiable renderer for image-based 3d reasoning. The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
[31] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent vidco depth estimation. ACM Transactions on Graphics (ToG), 39(4), 2020.
[32] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[33] Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, Sören Pirk, Binh-Son Hua, Sai-Kit Yeung, Xin Tong, Leonidas J. Guibas, and Hao Zhang. Language-driven synthesis of 3d scenes from scene databases. ACM Trans. Graph., 37(6):112, 2018.
[34] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text- driven neural stylization for meshes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13492-13502, June 2022.
12

---

[35] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, 2020.
[36] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional gen- erative neural feature fields. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14488- 11459, 2020.
[37] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5437-5446, 2020.
[38] Roy Or-EI, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: Resolutns 3D-Consistent Image and Geometry Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13503- 13513 33,3, June 2022.
[39] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
[40] Chenyang Qi, Xiaodong Cun. Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.095: 2023.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
[42] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ICCV, 2021.
[43] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transuctions on Pattern Analysis and Machine Intelligence, 44(3), 2022.
[44] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo. Justin Johnson, and Georgia Gkioxari. Accelcrating 3d deep learning with pytorch3d. orXiv:2007.08501, 2020.
[45] Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing a consistent long-term
3d scene video from a single image. In CVPR, 2022.
[46] Chris Rockwell, David F. Fouhey, and Justin Johnson. Pixelsynth: Generating a 3d-consistent experience from a single image. In ICCV, 2021.
[47] Robin Rombach, Patrick Esser, and Björn Ommer. Geometry-free view synthesis: Transformers and no 3d priors, 2021.
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[49] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.
[50] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33: 20154-20166 2020.
[51] Uriel Singer, Adam Polyak, Thomas Hayes, Xiaoyue Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. ArXiv, 0tt/22099.14992. 2022.
13

---

[52] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman. Text-to-4d dynamic scene generation. arXiv:2301.1 /280, 2023.
[53] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[54] Jiaxiang Tang. Stable-dreamfussinn: Text-to-3d with stable-diffusion, 2022.
[55] Alexandru Cristian Telea. An image inpainting technique based on the fast marching method.
Journal of Graphics Tools, 9:23 - 34, 2004.
[56] Hung- Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. CVPR, 2023.
[57] Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 551-560, 2020.
[58] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from single image. In CVPR, 2020.
[59] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for -o-idoo generation. ArXiv, abs/2212.115655. 2022.
[60] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields on complex scenes from single image. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXII, pages 736-753, Springer, 2022.
[61] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33, 2020.
[62] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[63] Shangzhan Zhang, Sida Peng, Tianrun Chen, Linzhan Mou, Haotong Lin, Kaicheng Yu, Yiyi Liao, and Xiaowei Zhou. Painting 3d nature in 2d: View synthesis of natural scenes from single semantic mask. In CVPR, 2023.
[64] Zhoutong Zhang, Forrester Cole, Richard Tucker, William T Freeman, and Tali Dekel. Con- sistent depth of moving objects in video. ACM Transactions on Graphics (TOG), 1012) 2021.
[65] Yuyang Zhao, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. takke---rotagoniis Generic video editing with an ensemble of experts. arXiv preprini wrXiv:2305.08855 2023.
[66] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
14

---

A Implementation Details
We provide implementation details for our framework and finetuning/generation regime.
Runtime, We use the Stable Diffusion model that was additionally finetuned on the inpainted task to perform inpainting. We use DDIM scheduler [53] with 50 sampling steps for each generated frame. Synthesizing 50 frame-long videos with our full method takes approximately 2.5 hours on an NVIDIA TeslaV100 GPU. Specifically, Table 3 reports runtime required for each frame.
Rendering
Inpainting
Depth model finetuning
Decoder finetuning
-00 sec
-5 sec
-40 sec
~60 sec
Table 3: Runtime per-frame. Number of seconds required for each step of the framework. Note that Rendering includes antialiasing and floating artifacts steps. comparison, VideoFusion [32] takes 15 seconds for 16-frame video of 256#256 resolution we do not have direct access to the GEN-1 [5] model, we are limited assessing its runtime. To get rough estimate, measured their online interface runtime, which takes minute to edit video with 25 frames.
NVIDIA TeslaV 100 GPU. Since
Depth prediction model and LDM decoder finetuning. We use MiDaS-DPT Large 42 as our depth prediction model. For each generated frame, we finetune it for 300 epochs, using Adam optimizer [25] with a learning rate le 7. Additionally, we revert the weights the depth prediction model to the initial state, discussed in Sec. 3.3. We finetune the LDM decoder for 100 epochs on each generation step using Adam optimizer with a learning rate of le
Camera path. Our camera follows a rotational motion combined with translation in the negative depth direction. Starting from simple translation for k frames, every frames we randomly sample a new rotation direction in the z-2 plane (panning), that camera follows for n frames. In our experiments, we set k = 5 and = 5. We use PyTorch3D to render and update our unified 3D) representation.
Mask handling. We observed that the scene's geometry sometimes induces disti--dist inpainting masks for the Stable Diffusion inpainting model. To address this issue, we perform morphological open operation on the inpainting mask: Mo = Open(M) with kernel size Then we inpaint the mask difference M Mo using Telea [55] while the inpainting model operates on Mo afterward.
A.1 Mesh update.
As discussed in Sec. 3.5 in the paper, we update the mesh as follows: given an image It+1, a mask of pixels to unproject M. a corresponding depth map D++1 and camera pose C2+1, we unproject the content in process that denoted the main paper by MM+1 = UnProj (M, It+1, D++1, C++1) First, each pixel center is unprojected by its depth value and camera pose into 3D mesh vertex with the pixel's color as its vertex color. Then, each unprojected four neighboring pixels with coordinates (i, (i + 1,)), (i.j 1). (i 1, 1) used to define two adjacent triangle faces.
To prevent holes in the generated mesh we connect the existing mesh M, to the newly unprojected part M4+1 by inserting additional triangles. To do this, we first extract the boundary pixels surrounding the inpainting mask M, and find the faces from M, that are projected to the current view C++1- For each such face, we select the 3D point closest to the current camera center: pz arg min || cl|², (P(I.PP.PP)
where e the camera center of Cr+11 1. and (pi, PP, P2,P3) are the points a given triangle. Finally, we add the selected points the triangulation scheme described above, which automatically creates the required triangles connecting M, and M,+1-
15

---

(a) antialiasing
With antialiasing
(c)
With removal
Figure 8: Rendering improvements. Left: effect of rendering without antialiasing (a) and with antialiasing (b). Right: effect of rendering without stretched triangles removal (a) and with stretched triangles removal (b).
A.2 Rendering
Antialiasing. Rendering an image from mesh requires projecting points and faces to a 2D plane. The rasterization process often creates aliasing artifacts (Fig. [ left), especially when a high-resolution mesh content from an earlier frame rendered a later frame, resulting in large amount of triangle that needs to be rasterized into low number of pixels. To avoid these artifacts, we apply antialiasing, similar the image resize antialiasing method- we render the mesh x2 higher resolution, apply Gaussian blur, and resize it to the required resolution.
Stretched triangles removal. described in Sec. 3.6, the stretched triangles are forming between close and far away content along regions of depth discontinuities (Fig. 8 right), and we would like remove them. Following Liu al. [29]. we apply the Sobel filter on the depth map D, to detect regions of depth discontinuities and threshold the values below the threshold of 0.3. Then we find triangles that are projected to the selected edge regions and filter them based on their normals. Specifically, we keep only the following triangles: {tra (center(tr.) c)²n < e). where is the threshold, n the normal a triangle, and e the camera center of C.+1. In practice, we set 0.055.
Floating artifacts fix. As described in Sec 3.6, content at the border the current frame can be mapped towards the interior of the next frame due to parallax. This creates "floating artifacts," shown in Fig. the paper. To overcome it, we pad the previous depth map D. with border depth values to 1.5x the rendering resolution, as if content exists beyond the image borders. Then, after warp the padded depth to the next camera location, we get a new mask, Mpodd that contains the content beyond the image borders. We use this mask to mask out the floating regions and thus enable the inpainting model to fill those holes with closer content. We perform this procedure on the image that was already rendered in 2x resolution due the antialiasing step described above.
B Baseline Comparison Details
VideoFusion. Since VideoFusion 33 does not have explicit way control the motion presented
in a video, we append "zoom out video" and "camera moving backward" to the input prompt to encourage the generated videos follow a backward camera motion. We generate 1000 videos of 16 frames each, in 256 resolution. When comparing our method to theirs, we downsample our videos to 256 resolution. In this comparison, we use the full set of 10 prompts:
1. "indoor scene, interior, candy house, fantasy, beautiful, masterpiece, best quality"
2. "POV, haunted house, dark, wooden door, spider webs, skeletons"
3. "walkthrough, an opulent hotel with long, carpeted hallways, beautiful photo, masterpiece, indoor scene"
4. "A dimly lit library, with rows upon rows of leather-bound books and dark wooden shelves"
5. "walkthrough, inside a medieval castle, metal, beautiful photo, masterpiece, indoor scene"
6. "Simple museum, pictures, paintings, artistic, best quality, dimly lit"
7. "walkthrough, sci-fi ship interiors, corridors, amazzing quality, masterpiece, beautiful scenery, best quality"
16

---

"Mecleeal castie"
Opulent kotel
Candy house"
"Library"
Figure 9: StableDreamFusion sample results. We provide it camera trajectory and prompt "a DSLR photo of inside *", Full results can be found the SM.
8. "A grand, marble staircase spirals up to a vaulted ceiling in a grand entrance hall of palace warm glow on the intricately designed floor"
9. "POV, cave, pools, water, durk cuvern, inside a cave, beautiful scenery, best quality"
10. "inside castle made of ice, beautiful photo, masterpiece"
GEN-1 To compare GEN-1 13]. used the RealEstate 10K dataset [66], consisting of curated Internet videos and corresponding camera poses. We filtered 22 indoor videos that follow smooth temporal backward camera motion adapt our method's setting. To do that, we filter videos that adhere to the following constraints:
(c++ - TT)TFr
[c(=-1 - - 2 0.9
Vt = framess
where is the center of the camera frame t, and is the viewing direction (last column of the rotation matrix). We subsample the filtered vidcos to contain 25 frames and reverse the video it had positive average displacement (C++1 c)) in the direction.
Since MiDaS produces depth maps that have a different range compared to the depth assumed in the RealEstaae 10K videos, we can't directly take the camera extrinsics from the RealEstate1000 [66] data. To align those ranges, we need find a scaling factor to multiply the MiDaS predictions with. To do that we run COLMAP and compute depth maps, using its dense reconstruction. Our scaling factor is then the ratio of median depth values of COLMAP and MiDaS:
median (DF)N Jt=1)
(8)
mcdian (DD, MPM 1
where D. and DM the COLMAP and MiDaS depth maps for frame respectively, and N is the number frames in the video.
For comparison to GEN-1 we used prompts #1-#5 from the above prompt list that allow reasonable editing video, depicting an interior of a house.
GtaameDreamFusian In addition, we compare our method to StableDreamfussonn 54 an open- source text-to-3D model capable of generating an implicit function of scene (NcRF 35)) from text prompts. We provide it our camera trajectory and simple text prompt such as "a DSLR photo of the inside cave" (when providing prompts from our usual prompts set, StableDreamFustie to converge to meaningful results). The generated scenes contain a lot of blur and unrealistic artifacts, since achieve good visual quality, NERF requires multiple viewpoints of
was unable
as can seen in Figure 9 a scene from different angles. Full video results of MabbleDremmussee can be found in the SM.
C Broader impact
Our framework is test-time optimization method, which does not require any training data. Never- theless, our framework uses two pre-trained models: depth prediction model and text-to- diffusion model 48), which are susceptible to biases inherited their training data (as discussed in Sec. in [48). In our context, these models are used to generate 3D static scenes. To avoid harmful content, we consider only text prompts that describe general scenery and objects, while avoiding prompts that involve sensitive content such as humans.
17

---

Spiral path
"POV, cave, pools, watee,, cavers, cave, beautiful scenery, qualthy,
Frame
Spiral camera path "Halktkrousk, medieval dungeon with caap, stone corridors and flickering torckes lining the walls"
Frame 15
Frame 25
Frame 45
Frame
Frame 15
Frame
Spiral camera path beautiful photo, masterplece, indoor sceiee"
Frame 45
"walkEkrough, opulent hobel with lomg, carpebed kallways,
Big path qualidy"
scene, interior, candy kouse, fankaay, beaultful, wasterppece,
Frame 2
Frame 15
Frame
Frame 45
Big turns camera path
"inside castle made of ice, beautiful photo, masterpiece"
Frame
Frame 15
Frame 45 "inside medieval forge, mebal, fire"
Frame
Original camera path
Original path
"hausted dark, woodek door, splder webs, skelletoss
Figure 10: Additional results with different camera motion. Left: Representative frames generated by our method with colorful borders corresponding to specific camera locations. Right: The camera path used in the generated video, with cameras color-coded to match the displayed frames' borders, reflecting their respective locations along the motion path.
18


Reviews:
[{"review_id": "ESQaiaR3P0", "reviewer": "Reviewer_R7aK", "summary": "This paper proposes a text-driven perpetual view generation method that synthesizes long-term changing views of a scene and its corresponding 3D mesh, given the free-vocabulary text and camera trajectories. The authors apply an off-the-shelf pre-trained text-to-image diffusion model and a pre-trained depth prediction model to achieve the baseline quality of novel-view synthesis by reprojection and inpainting, and also apply a test-time finetuning scheme to enhance the geometry consistency and textural consistency throughout the view sequence. ", "questions": "1. The footage provided in the supplementary is all about a walk-through towards a straight corridor. Curious about if it can handle more complex situations, e.g., turning left/right/back?\n\n2. How many prompts are used in total in the test stage? Can you give a full list of them? It is unclear if the prompts cover various indoor scenarios and further affect the metrics.\n\n3. It seems time-consuming to synthesize one frame according to the supplementary material. A comparison of the runtime with the reference methods would be helpful. ", "limitations": "The authors have stated limitations and potential negative societal impact in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow and provides many impressive qualitative results.\n2. It is promising that this method can synthesize diverse and zero-shot scenes without large-scale training data on a specific target domain. \n3. The SceneScape pipeline is technically sound to generate consistent views as its novel views are grounded in the scene geometry, where the generation of novel contents and geometrical structures can be readily adapted from off-the-shelf text-to-image diffusion-based inpainting/generation models and depth prediction model.\n3. By maintaining a unified 3D mesh representation and adopting a test-time finetuning technique, this method achieves even more consistent results. Such a technique is meaningful for camera pose-conditioned or other geometry-aware online scene generation pipelines.\n", "weaknesses": "I am almost satisfied with this manuscript. Here just list a few questions other than the limitations discussed in the paper, as follows:\n\n1. The SceneScape seems not to be able to handle dynamic foreground content. If the textual descriptions tell the model that it has to generate ``dancing people'', what will SceneScape do? Moreover, as one possible downstream application, how to edit the perpetual scenes generated by SceneScape?\n\n2. There are still quite a few strange distortions in the synthesized scenes. Can it be further improved by some instance-level priors? For example, knowing that there exists a chair, and we can use the prior knowledge of chairs to rectify the shape distortions? Maybe adding the prediction of instance segmentation maps into the SceneScape pipeline will be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and the feedback from the reviewer.\n\n### Dynamic content generation\nBy design, our method supports only static scenes since we use a shared unified mesh representation for all the frames. Thus, when providing SceneScape with a prompt that generates an object with dynamic nature, it stays fixed throughout the entire video. An example of such behavior is the video depicting a walkthrough inside a medieval forge provided in the SM. As can be seen, the fire in the furnace is static. Adapting our method to support dynamic scenes is an exciting avenue for future research. \n\n\n### Improved geometry\nWe thank the reviewer for the interesting suggestion. Indeed the quality of our results is affected by the accuracy and consistency of the depth predictions, which we improve through depth finetuning, but are still not perfect. Incorporating additional object-level shape priors can potentially further improve the estimated geometry of specific objects, yet it is a non-trivial task since object-level information needs to interact consistently with the rest of the generated scene content. Note that our method is not restricted to a specific depth prediction model, thus we can leverage any future improvement in monocular depth prediction when it becomes available.\n\n### Camera motion\nPlease refer to the global response section, where we elaborate on our camera motion.\n\n### Text prompts\nIn our ablations, we use 20 videos with a set of 10 diverse text prompts, which are provided in section 2 in the Supplementary Materials. We use a diverse set of representative prompts describing real and fantasy indoor scenes, dark and lit scenes, as well as buildings and natural structures (cave). \n\n### Runtime w.r.t. baselines\nVideoFusion takes ~15 seconds for a 16-frame video of 256*256 resolution on a TeslaV100 GPU. Since we do not have direct access to the GEN-1 model, we are limited in assessing its runtime. To get a rough estimate, we measured their online interface runtime, which takes ~1 minute to edit a video with 25 frames. Our runtime, along with the runtime of each of our modules, are included in Section 1 of the SM. Please note that our method employs test-time optimization/training, whereas the baselines are feedforward models, thus the runtimes cannot be directly compared. We will include the runtime comparison in the revised SM version.\n"}, {"review_id": "ssfW6kAopk", "reviewer": "Reviewer_ooiE", "summary": "In this paper, the authors study text-driven perpetual view synthesis with the input of text prompts describing the scene and desirable camera poses. Instead of training models on a specific domain, they propose an online scene generation method with a pretrained text-to-image model and a pretrained monocular depth prediction model. To further improve the video consistency, they utilize a shared mesh to represent the generated scene and deploy online time-time training to finetune the monocular depth prediction model for better mesh construction. The experiments demonstrate that the proposed method can generate geometrically-plausible scenes, and the proposed modules can improve the video quality quantitatively and qualitatively.", "questions": "Please answer the questions mentions in the weakness part. I will adjust my score according to the rebuttal.", "limitations": "The authors have properly addressed the limitation and broader impact in the main paper and supplement.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The authors directly deploy a pretrained text-to-image model and a pretrained monocular depth prediction model as a strong indutive bias, enabling their method to work on diverse scenes in a zero-shot manner and eliminate training datasets.\n- The generated videos are consistent and geometrically plausible. With the text prompts, the users can decide on the generated scenes.\n- To further improve the online generated scenes, the authors deploy a unified mesh representation and adopt test-time training to finetune the depth prediction model, which are validated to improve the results in the extensive experiments.\n", "weaknesses": "- The main concern of the proposed method is that the presented results are nearly all with camera trajectory moving back. Can the proposed method work on trajectories similar to those in Text2Room? Or does some inherited property limit it to work on those camera trajectories?\n- I have a question on the need for the decoder finetuning. When we want to apply the text2image diffusion model for inpainting, we can replace the existing part with the input though the decoder can not perfectly reconstruct the input. In this case, why do we need decoder finetuning?\n- When watching the visual ablation study, it is actually hard to distinguish the improvement of the proposed modules. The demonstrated videos are nearly all moving backward and corridor scenarios. In this case, only the warp-inpaint baseline will work well.  \n- In Line 222, you mention the depth-aware splatting. Can you explain how you elaborate more on this baseline?\n- Why is the mesh so blurry in Figure 7 (left)? Worldsheet also uses the mesh representation, but their boundary is not that bad. Have you tried to split the mesh into foreground and background to help the discontinuity of the scene?\n- In the supplement video results, the warp-inpaint baseline is missed for visual ablation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and the feedback from the reviewer. \n\n### Camera Motion\nPlease refer to the global response section, where we elaborate more on our camera motion. \nText2Room (concurrent work) focuses on generating a mesh rather than synthesizing a coherent video as we do. Thus, they tailor their camera poses to increase the quality of the generated mesh (e.g., covering holes). Specifically, they consider many short, segmented, and distant cameras. This choice is unsuitable for representing a coherent video captured by a single camera smoothly moving in space.\n\n### Decoder finetuning\nAs discussed in Sec. 3.4 (L156), directly combining the known content (rendered from the mesh) and the new inpainted content results in a visible seam at the inpainting border. This is caused due to imperfect encoding/decoding of the Latent Diffusion Model. By finetuning the decoder to accurately reconstruct the input content at the known pixel locations, we resolve this issue.\n\n\n### Visual ablation study\nThe warp-inpaint baseline is provided in the Ablations section in the HTML SM file, in the rightmost column under the name “naive warp-inpaint”.\n\nWe agree that the differences between some of the ablation videos, when all videos are played side-by-side, may be difficult to perceive due to the fast camera motion and overall similarity of the generated content. Note that in our user study, only two videos are played simultaneously, which makes the visual comparison easier. As seen in Tables 1 and 2 in the paper, most users preferred our full method. This is further supported by the CLIP aesthetic score.\n\n### Depth-aware splatting\nThe depth-aware splatting baseline is part of the ablation study and evaluates the contribution of having a global unified representation. As an alternative way to generate geometrically consistent video without any unified representation, this baseline generates new frames using a splatting warp [33] of the previous frame content into the next frame. This method uses the depth and the camera motion from the previous known frame to define the warping field. As can be seen, our method outperforms this baseline, indicating the importance of having a unified representation. \n\n### Outdoors scenes\nFigure 7 shows the result of our method without any special handling for outdoor scenes. Splitting the mesh into foreground and background layers, as the reviewer suggests, could help handle the scene's discontinuity. However, when we tried such an approach, we noticed that two layers are usually not enough to support challenging cases of diverse content with multiple depth discontinuities, e.g., a scene of rocks, trees, distant mountains, and cloudy skies. Furthermore, It requires handling segmentation inaccuracies. Therefore, we think that outdoor scenes require special handling and would be exciting future work. \n"}, {"review_id": "Hu6BzTTjyL", "reviewer": "Reviewer_9ntp", "summary": "1. This paper proposes a method for test-driven consistent scene generation using a pretrained text-to-image model and depth prediction models.\n2. The updating of mesh representation and fine-tuning of the decoder and depth prediction module enable continuous scene generation.", "questions": "1. The paper could provide additional analysis on cumulative errors and failure cases to further demonstrate the robustness of the proposed method. ", "limitations": "The paper does address the limitations of the proposed method. ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The continuous scene generation demonstrated in both the main text and supplementary materials proves the effectiveness of the method. The proposed method generates more continuous results than the baseline method, given the same initialization and camera trajectory.\n2. It is interesting to see the fancy effects achieved by utilizing pretrained inpainting and depth prediction models.", "weaknesses": "1. There is a lack of analysis on cumulative errors. \n2. The efficiency analysis provided in the supplementary materials could benefit from further optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and the feedback from the reviewer. \n\n### Error accumulation analysis\n\nFollowing the reviewer’s comment, we measured our error accumulation for our video evaluation set. To do so, we measured: (i) visual quality by measuring the CLIP aesthetic score as a function of frame time step and (ii) geometric consistency by evaluating COLMAP on the first $n$ frames ($n$=10,20,30,40,50) and computed the corresponding SI-RMSE as a representative metric for the geometric consistency. \n\nAs can be seen in the attached PDF, our method exhibits slight degeneration in visual quality, as expected and discussed in Section 5 in the paper, yet results in a significant improvement over the warp-inpaint baseline. Regarding geometry consistency, our SI-RMSE remains steady or even slightly improves with the length of the video (since COLMAP works better when more correspondences can be established). \n\n"}, {"review_id": "1ipMbqSRuU", "reviewer": "Reviewer_Zj8B", "summary": "This paper approaches the task of text-conditioned generation of a walkthrough video upon (mostly indoor) scenes. It proposes to leverage pretrained text-to-image and pretrained depth prediction models, to generate these videos without finetuning. Implementation generates videos in an online manner by repeated warping, inpainting and mesh aggregation. Contributions include test-time overfitting of depth and inpainting, and experiments show these are important to geometric  and semantic performance.", "questions": "-\tMotivating the intro with single-image novel view synthesis is a strange choice, and is a bit confusing, when the focus of this work is text-driven scene synthesis\n-\tMissing period L126\n-\tSlightly strange writing to see a list (L130) enumerated into subsections. It would feel more typical to change this sentence to finish with a period, and overview the subsections, or to simply make subsections as paragraphs.\n-\tOne addition metric that could be useful is CLIP R-Precision, which is used by prior work e.g. DreamFusion. By selecting some random frames in each video and ranking CLIP distance to all text prompts, one can compute precision. This could be helpful as automated visual semantic metrics can be limited in a setting such as this.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper uses a cool insight to generate 3D walkthroughs using 2D pretrained models\n\n-\tThe paper leverages the power of large-pretrained models for online generation of many images in a consistent manner, without the need for finetuning. Careful implementation enables these images to create a consistent scene\n\n-\tVideos are long (50 frames with high movement), showing impressive consistency. They are also typically visually appealing (especially compared to a naïve implementation)\n\nSeveral contributions are executed to make performance high-quality\n\n-\tFinetuning depth and decoder modules with previously generated content meaningfully improves consistency\n\n-\tDealing with floating artifacts is a clear challenge that the proposed method deals with effectively (Fig 3)\n\n-\tThe method performs a union operation on mesh. This is not novel alone, but (correct) implementation is important to performance.\n\n-\tIn total, the proposed method improves COLMAP fitting errors by 3-6x over naïve method", "weaknesses": "The method appears to be limited to backward movement, with little rotation. This trajectory setup makes the task easier and less interesting, as it does not handle side-to-side movement, or “looking around” which are important components of walkthroughs e.g. see RealEstate10K\n\n-\tConsider the motivation for this paper in L15-16 “By observing only a single photo of a scene, we can imagine what it would be like to explore it, move around, or turn our head and look around”. Unfortunately, this work does not answer this problem satisfactorily. Instead, it produces nice-looking and consistent results as we move backwards through an infinitely-long corridor, where we only slightly turn our head periodically.\n\n-\tMore “looking around” or freedom of movement would better test mesh is consistent across dramatically different views and does not have holes. As a viewer, it is also closer to the desired goal of exploring the generated scene.\n\n-\tIf the paper were instead focused on the infinite-generation direction, perhaps a more appropriate set of experiments and direction would be those relating to Infinite Nature?\n\nGenerated videos are of houses with unrealistic structure, breaking the desired illusion\n\n-\tThe trajectory of cameras means the video moves backward with a zig-zag type pattern. Walls tend to move correspondingly, and are typically equally at both edges of the video.\n\n-\tThis results in sequences where the viewer moves down an infinitely long hallway that goes back and forth as the camera does. This feels unrealistic; I can’t imagine an indoor structure that looks like any of the videos.\n\n-\tI feel prior work (e.g. GEN-1 or VideoFusion) actually may generate houses that are in some ways more realistic, as they do not always have walls at the edge of the video frame (although the difference in movement between each frame makes the comparison difficult to judge)\n\nExperiments feel somewhat ad-hoc. The small size and lack of consistency / justification for prompts make it harder to draw conclusions from results\n\n-\tAblations take place on 20 videos using 10 text prompts (small set). It is also surprising to use only 10 text prompts when generating 1000 videos for the VF experiment; or 5 text prompts on 110 videos against GEN-1. Results would be much more compelling if using a different prompt for each video. I’m curious if there is a limitation I am missing that could explain this.\n\n-\tPrompts are not programmatically generated, and I did not find a justification for the selected prompts (lack of consistency; do not reflect some true data distribution). Programmatic generation of prompts would also more easily enable scaling to larger sets.\n\n-\tPrompts contain significant unique detail e.g. “indoor scene, interior, candy house, fantasy, beautiful, masterpiece, best quality” and “POV, haunted house, dark, wooden door, spider webs, skeletons” (feels as if it could be prompt engineered, doesn’t feel reflective of some true distribution). \n\n\nEdit: Thanks for the rebuttal. I think the results with more significant turns are better, and showing some examples without prompt engineering is helpful.\n\nI feel after reading the rebuttal and other reviews the paper is still a 3 - reject. I stand by my original comment the lack of systematic experiments in the paper make it hard to draw clear scientific conclusions from this work. In additon, while results are consistent across frames, the completely unrealistic nature of generated houses mean this contribution is not impressive enough for acceptance in my mind.\n\nThe reviews of this paper are very mixed. Despite my negative review, accepting it would not be the end of the world if other reviewers end up feeling very positive given (1) the generated frame quality and cross-frame consistency and (2) a few technical contributions e.g. floaters, finetuning depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and feedback from the reviewer. \n\n### Camera motion and scope\nPlease refer to our global response regarding our choice of default camera trajectory. Please also refer to the attached PDF and the provided video for sample extended results, including large camera turns and circular motion. Indeed, as an inspirational example, we describe in the introduction the ability of humans to envision and expand a scene in our mind by observing only a single photo of it. We do not claim that the task is yet solved computationally but believe that our work takes a significant first step in this direction. We are the first to show how to harness the generative prior of a text-to-image model for the task of perpetual view generation and thus focus on the fundamental challenges involved in achieving a 3D-consistent video. We believe that our work will trigger future research in this direction, e.g., enabling arbitrary camera motion, as suggested by the reviewer. Finally, we would like to note that our primary goal is not to create a perfect mesh but rather to use its geometric properties to synthesize a 3D-consistent video. Thus, our default camera trajectory and evaluation metrics were selected accordingly. \n\n### Metrics \nWe indeed considered the series of InfiniteNature papers. However, their evaluation metrics are not suitable for our case. Since these methods are GAN-based models trained on specific domains (e.g., landscape videos), they use FID and other reconstruction metrics to evaluate their visual quality. These metrics are not applicable in our case since we do not have ground truth training videos or specific underlying data distribution to match (L273 in the paper). \n\n\n### Generated content quality\nThe quality of the generated content of our method mostly depends on the generative prior and the depth estimation models working together. We believe that our method of combining those priors is a major step towards text-driven perpetual view generation.\nAs described in section 4.3.2 in the paper, to assess the visual quality of our outputs and to compare with the existing works, we performed an AMT user study against prior works -  VideoFusion and GEN-1. As can be seen in Table 2, our method was preferred in 96% and 69% of the cases for VideoFusion and GEN-1, respectively. \nAdditionally, in the case of GEN-1, it takes the scene geometry as input, thus tackling a simpler task.\n\n### Prompt selection\nWe focused on synthesizing static indoor scenes, thus any prompt that describes such scenes can be inputted into our method. We appended a few words such as “masterpiece, best quality” to improve the visual quality and the fidelity to the text prompt, which is common practice in numerous papers  [61,62,63]. Following the reviewer’s comments, we omitted these words and re-run our method. As seen in the attached PDF and in the third row in the provided video, our method is agnostic to these specific words and works well with very simple scene descriptions. We will clarify this in the paper.\n\nRegarding evaluation, our goal was to base our quantitative evaluation on prompts that we could also qualitatively show and evaluate with a user study. Thus, we selected a set of diverse, representative prompts (see SM for the full list).\n\nFinally, as a proof of concept, we also verified that our method works well on ten prompts generated by ChatGPT when it was asked to generate prompts describing indoor scenes. We will include these results in our revised version. \n\n[61] Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall: DreamFusion: Text-to-3D using 2D Diffusion. ICLR 2023\n\n[62] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen: GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. ICML 2022\n\n[63] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen: Hierarchical Text-Conditional Image Generation with CLIP Latents. CoRR abs/2204.06125\n\n### CLIP R-precision\n\nFollowing the reviewer's suggestion, we evaluated the CLIP-R precision. We measured the accuracy with which CLIP retrieves the caption that was used to generate a frame among all the prompts used in our evaluation (10 prompts in total). Our method achieved a score of 0.92, while the naive warp-inpaint baseline scored 0.82. We will add this metric to our revised paper. \n"}, {"review_id": "pgfaoo8gR0", "reviewer": "Reviewer_GkJU", "summary": "The paper present a method for text-driven perpetual view generation. It takes as input a text-prompt and carema trajectory as input, optimizing a unified mesh representation for rendering coherent video sequence accordingly. The key idea is straight-forward by connecting text-driven inpainting and depth warping with mesh-based representation to ensure 3D consistency.", "questions": "- Why finetune depth prediction model at test time? Finetuning may overfit/forgetting when the generated video is too long. Plus, the error would accumulate along with time steps. When I come to the ablation study of depth finetuning, I found that the RGB texture dose not jittering across video frame. In that case, does the depth map finetuning matter?\n- I’m not clear how depth finetuning really work as g is reverted at each time step.\n- The initialization of mesh M_0 is not clear to me. Does the number of point cloud is consistent with the nubmer of pixels? Moreover, how to get faces and edges of mesh from point cloud?\n- Online mesh update: the merging in Eq. 6 would lead to conflict. How to solve conflict is not introduced.\n- Figure 7 shows the failure cases resulted by error accumulation. Here comes the question, can authors evaluate the robustness and error rate of the method? For example, given a fixed set of text prompts, how many scenes can be successfully generated?\n- Moreover, I would like to see experiments or discussion of sensitivity to hyperparameters.", "limitations": "Limitations and negative societal impact are discussed. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes the first method for text-driven perpetual view generation with promising results, i.e., diverse scene textures and inter-frame coherence.\n- As an emerging direction with limited open-source work for comparisons, the authors have tried their best to compare the proposed method with others from different perspective, i.e., video generation, text-drive vid2vid, text-to-3d, for extensive evaluations.\n- The paper is well-organized and easy to follow.", "weaknesses": "- Only bounded indoor scene as the model is constrained by accuracy of depth map. However, outdoor unbounded scene is difficult to generate since depths has infinite ranges and discontinuities.\n- Moreover, the proposed method has the strong tendency to generate tunnel-specific scenes, i.e., a narrow hallway or tunnel through the space. It might be caused by the inpainting+depth continuity assumption. Because the surrounding walls acts as strong geometry cues for view consistency as camera moving backward.\n- Prompt engineering is required. Keywords like “POV” and “walkthrough” is critical for the generation results. The author should discuss their effects and design choices carefully as the paper claims using “free-vocabulary text”.\n- I may question the second contribution in the introduction, which claims the proposed method as the first zero-shot scene generation method. Unlike zero-shot text-to-3D object generation (DreamField, Dreamfusion, etc.) where a 3D representation is generated and camera can freely move, the proposed method binds the camera movement with the optimization process, which limits the camera trajectory after the optimization done. Therefore, it is only a 3D-aware video generation instead of 3D scene.\n- This is a personal one. The general idea (text-to-image + depth inpainting) of this method is not completely novel, with lots of engineering tricks to make it work. And I believe some details are missed to evaluate other technical parts like depth & decoder finetuning. Please check the Question section.\n- I believe the work miss some important literature for reference, in terms of zero-shot text-driven synthesis and indoor scene synthesis.\n    - zero-shot text-driven synthesis:\n        \n        For optimization-based one, there are StyleCLIP and etc. For training without text-image pair, there are LAFITE.\n        \n        StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery, Patashnik et al, ICCV 2021.\n        \n        LAFITE: Towards Language-Free Training for Text-to-Image Generation, Zhou et al, CVPR 2022.\n        \n    - scene synthesis / language-driven scene synthesis:\n        \n        Learning spatial knowledge for text to 3D scene generation, Chang et al, EMNLP 2014\n        \n        Language-driven synthesis of 3d scenes from scene databases, Ma et al, SIGGRAPH Asia 2018.\n        \n        Text2Light: Zero-shot Text-driven HDR Panorama Generation, Chen et al, SIGGRAPH Asia 2022.\n- typos:\n    - lacks a full stop in #L126\n    - Figure or Fig. is not consistent in the paper. For example, #L195.\n    - ‘mesh’ instead of ‘meth’ in #L285", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and feedback from the reviewer. \n\n### Outdoor scenes\nExtending our method to handle outdoor scenes is an interesting research direction that indeed poses several challenges, including the accuracy of the estimated depth, as pointed out by the reviewer, as well as handling dramatic depth discontinuities. We discuss this in Section 5 of the paper. \n\n### Camera Motion\nPlease refer to our global response, where we elaborate on our camera motion configuration and include additional results.\n\n### Prompt selection\nOur method does not require careful prompt engineering and can work with input prompts that describe general, static indoor scenes. We appended a few words such as “masterpiece, best quality” to improve the visual quality and the fidelity to the text prompt, which is common practice in numerous papers [61,62,63]. We omitted these words and re-run our method. As seen in the attached PDF and in the third row of the provided video, our method is agnostic to these specific words and works well with very simple scene descriptions. We will clarify this in the paper.\n\n[61] Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall: DreamFusion: Text-to-3D using 2D Diffusion. ICLR 2023\n\n[62] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen: GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. ICML 2022\n\n[63] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen: Hierarchical Text-Conditional Image Generation with CLIP Latents. CoRR abs/2204.06125\n\n\n### 3D-Consistent video generation\nZero-shot text-to-3D object generation methods such as DreamFusion are not suitable for the task of perpetual view generation. This is demonstrated in Figure 9 in the SM, where we provided DreamFusion with our camera poses and input text prompt. As seen, this method completely fails to optimize a high-quality NeRF. Generally, such methods require multiple observations of the same scene point to coverage, which are not available in our setting. \n\n### Literature reference\nWe thank the reviewer for the references. Although these works are not directly aimed at the task of video generation, we will include them in the context of zero-shot text-driven content manipulation/generation.\n\n### Necessity of depth model finetuning\nAs described in Section 3.3 in the paper and demonstrated in our ablation study, per-frame monocular depth predictions tend to be temporally inconsistent, even across nearby video frames. Finetuning the depth model to align the depth predictions across frames (Eq. 3) mitigates this issue. Note that the finetuned depth prediction model can still retain its prior since (i) the finetuning is done only for 300 steps, and (ii) we revert the model weights to its original non-tuned ones for each frame. Thus, we do not deviate significantly from its prior, thus avoiding catastrophic forgetting. We refer the reviewer to previous works [24,64,65] in the context of video depth estimation for a more elaborated discussion about test-time depth finetuning. \n\n[64] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, Johannes Kopf: Consistent video depth estimation. ACM Transactions in Graphics. 39(4)\n\n[65] Zhoutong Zhang, Forrester Cole, Richard Tucker, William T. Freeman, Tali Dekel: Consistent depth of moving objects in video. ACM Transactions in Graphics 40(4)\n\n### Mesh initialization and update\nAs described in Section 3.1 of the paper, we initialize the mesh by unprojecting each pixel in the first frame into a corresponding 3D point. Then, each unprojected four neighboring pixels are used to define two adjacent triangle faces. We refer the reviewer to Section 1.1 in the SM for details about the mesh update. Note that at each update step (Section 3.5 in the paper), we only add the newly generated content while keeping the existing content in the mesh, thus no conflict is introduced.\n\n### Error accumulation\n\nFollowing the reviewer’s comment, we measured our error accumulation for our video evaluation set. To do so, we measured: (i) visual quality by measuring the CLIP aesthetic score as a function of frame time step and (ii) geometric consistency by evaluating COLMAP on the first $n$ frames ($n$=10,20,30,40,50) and computed the corresponding SI-RMSE as a representative metric for the geometric consistency. \n\nAs can be seen in the attached PDF, our method exhibits slight degeneration in visual quality, as expected and discussed in Section 5 in the paper, yet results in a significant improvement over the warp-inpaint baseline. In terms of geometry consistency, our SI-RMSE remains steady or even slightly improves with the length of the video (since COLMAP works better when more correspondences can be established).\n\n\n### Hyperparameters sensitivity\n\nSince we did not perform any extensive hyperparameters search and used the same hyperparameters to generate all of our results, we omitted any analysis of the hyperparameters' sensitivity. \n"}]

Author rebuttal:
We would like to thank the reviewers for their thorough and valuable feedback. Please note that a link to the video file has been sent to the AC with additional results. Please see the per-reviewer responses for details. 

### Camera motion 
The reason we focused on backward rotational camera motion is twofold: (i) our primary objective is synthesizing long coherent videos. Thus, the camera trajectory has to follow a smooth path that complies with the generated scene, i.e., avoid crashing into walls or going out of the scene's bounds. (ii)  synthesizing the scene while zooming in / moving forward into the scene would require adding new details to existing content, i.e., would require solving the challenging task of super-resolution. Empirically, we found that backward rotational camera motion allows us to bypass these challenges without any user input or manual control while demonstrating non-trivial parallax effects, occlusion/disocclusion handling, and complex scene geometry. 

Following the reviews, we expanded our results to include camera paths that depict sharp turns and backward spiral motion. As can be seen in the attached PDF and the provided video (first and second rows), our method performs well under these challenging camera paths. We will include these results and discussion in the revised paper. 

We believe that a more sophisticated, scene-aware camera path may be possible to design yet out-of-scope of our paper that considers and tackles the task of text-driven perpetual view generation for the first time. 


Meta-review: The paper has mixed reviews. On the positive end, the paper is one of the first explorations toward the text-driven perpetual view generation using pretrained text2image model, and shows reasonably well quality and consistency. On the negative end, there are some concerns regarding camera trajectory, visual quality in some scenarios, insufficient evaluations, and inaccurate claims. I believe most of the concerns are addressed after the rebuttal. I would like to recommend the acceptance of the paper, and suggest the authors revise the manuscript accordingly.
Decision: Accept (poster)