SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rafail Fridman*
Weizmann Institute of Science rafail.fridman@weizmann.ac.il
Amit Abecasis*
Weizmann Institute of Science amit.abecasis@weizmann.ac.il
Yoni Kasten
ykasten@nvidia.com
Weizmann Institute of Science tali.dekel@weizmann.ac.il
Figure 1: (a) Given a text prompt describing a scene, SceneScape generates a long-term video depicting a walkthrough of the scene, adhering to a given camera trajectory. We achieve this through a test-time optimization approach that utilizes a pre-trained text-to-image generative model and a depth prediction model, enabling online 3D-consistent perpetual view generation without the need for training data. (b) To synthesize a 3D-consistent video, we construct a unified 3D mesh concurrently with the video. For visualization purposes, we remove the side part of the mesh.
"POV, cave, pools, water, dark cavern, inside a cave, beautiful scenery, best quality" (a) Sample frames from our generated video
(b) Our unified representation
We present a method for text-driven perpetual view generation – synthesizing long-term videos of various scenes solely from an input text prompt describing the scene and camera poses. We introduce a novel framework that generates such videos in an online fashion by combining the generative power of a pretrained text-to-image model with the geometric priors learned by a pre-trained monocular depth prediction model. To tackle the pivotal challenge of achieving 3D consistency, i.e., synthesizing videos that depict geometrically-plausible scenes, we deploy an online test-time training to encourage the predicted depth map of the current frame to be geometrically consistent with the synthesized scene. The depth maps are used to construct a unified mesh representation of the scene, which is progressively constructed along the video generation process. In contrast to previous works, which are applicable only to limited domains, our method generates diverse scenes, such as walkthroughs in spaceships, caves, or ice castles. Project page: https://scenescape.github.io/
By observing only a single photo of a scene, we can imagine what it would be like to explore it, move around, or turn our head and look around – we can envision the 3D world captured in the image and expand it in our mind. However, achieving this capability computationally – generating a long-term video that depicts a plausible visual world from an input image – poses several key challenges. One of the main challenges is ensuring that the synthesized content is consistent with a feasible 3D world. For example, it must correctly depict parallax effects and occlusion relations of different objects in
the scene. Additionally, to synthesize new content, a strong prior about the visual world and how it would appear beyond the current field of view is necessary. Finally, the generated content should appear smooth and consistent over time.
Previous methods have tackled the task of perpetual view generation for specific domains, for example, synthesizing landscape flythroughs [e.g., 5, 7, 28, 29], or bedrooms walkthroughs [e.g., 45, 47]. These methods involve large-scale training on videos or images from the target domain, which limits their use. In this work, inspired by the transformative progress in text-to-image generation, we take a different route and propose a new framework for text-driven perpetual view generation – synthesizing long-range videos of scenes solely from free-vocabulary text describing the scene and camera poses. Our method does not require any training data but rather generates the scene in a zero-shot manner by harnessing the generative prior learned by a pre-trained text-to-image diffusion model and the geometric priors learned by a pre-trained depth prediction model.
More specifically, given an input text prompt and a camera trajectory, our framework generates the video in an online fashion, one frame at a time. The text-to-image diffusion model is used to synthesize new content revealed as the camera moves, whereas the depth prediction model allows us to estimate the geometry of the newly generated content. To ensure that the generated scene adheres to a feasible 3D geometry, our framework estimates a unified mesh representation of the scene that is constructed progressively as the camera moves in the scene. Since monocular depth predictions tend to be inconsistent across different frames, we finetune the depth prediction model at test time to match the projected depth from our mesh representation for the known, previously synthesized content. This results in a test-time training approach that enables diverse, 3D-consistent perpetual view generation.
To summarize, our key contributions are the following:
1. The first text-driven perpetual view generation method, which can synthesize long-term videos of diverse domains solely from text and a camera trajectory.
2. The first zero-shot/test-time 3D-consistent perpetual view generation method, which synthesizes diverse scenes without large-scale training on a specific target domain.
3. Achieving 3D-consistent generation by progressively estimating a unified 3D representation of the scene.
We thoroughly evaluate and ablate our method, demonstrating a significant improvement in quality and 3D consistency over existing methods.
Perpetual view generation. Exploring an "infinite scene" dates back to the seminal work of Kaneva et al. [24], which proposed an image retrieval-based method to create a 2D landscape that follows a desired camera path. This task has been modernized by various works. Different methods proposed to synthesize scenes given as input a single image and camera motion [e.g., 21, 26, 28, 29, 45, 46, 56, 58, 63]. For example, synthesizing indoor scenes [26, 45, 46, 58], or long-range landscapes flythroughs [28, 29]. These methods have demonstrated impressive results and creative use of data and self-supervisory signals [e.g., 5, 7, 11, 12, 28, 45]. However, none of these methods considered the task of text-driven perceptual view generation and rather tailored a model for a specific scene domain. Thus, these methods require extensive training on a large-scale dataset to learn proper generative and geometric priors of a specific target scene domain.
3D-Consistent view synthesis from a single image. Various methods have considered the task of novel view synthesis from a single input image. A surge of methods has proposed to utilize a NeRF [35] as a unified 3D representation while training on a large dataset from a specific domain to learn a generic scene prior [22, 60, 62]. Other 3D representations have been used by various methods, including multi-plane images [57, 66], point clouds which hold high-dimensional features [46, 58], or textured-mesh representation [20]. In Jain et al. [22], the task is further regularized using the semantic prior learned by CLIP [41], encouraging high similarity between the input image and each novel view in CLIP image embedding space. All these methods require training a model on specific domains and cannot produce perpetual exploration of diverse scenes.
Figure 2: SceneScape pipeline. We represent the generated scene with a unified mesh  $\mathcal{M}$ , which is constructed in an online fashion. Given a camera at  $C_{t+1}$ , at each synthesis step, a new frame is generated by projecting  $\mathcal{M}_t$  into  $C_{t+1}$ , and synthesizing the newly revealed content by using a pre-trained text-to-image diffusion model. To estimate the geometry of the new synthesized content, we leverage a pre-trained depth prediction model; to ensure the predicted depth is consistent with the existing scene  $\mathcal{M}_t$ , we deploy a test-time training, encouraging the predicted depth by the model to match the projected depth from  $\mathcal{M}_t$ . We then update our mesh representation to form  $\mathcal{M}_{t+1}$  which includes the new scene content.
3D-Aware image generation. Related to our task is 3D-aware image synthesize, which typically involves incorporating a 3D representation explicitly into a generative model, e.g., a NeRF representation [8, 9, 16, 36, 50], or neural surface renderer [38]. These methods have shown impressive results on structured domains (e.g., cars or faces), but deploying them for diverse scenes is extremely challenging and often requires additional guidance [13], or heavy supervision, e.g., large-scale data that contains accurate camera trajectories, which can be obtained reliably mainly for synthetic data [4]. We aim for a much more diverse, flexible, and lightweight generation of arbitrary scenes.
Text-to-video generation and editing. Our work follows a broad line of works in the field of text-to-video synthesis and editing. There has been great progress in developing text-to-video generative models by expanding architectures to the temporal domain and learning priors from large-scale video datasets [e.g. 15, 18, 32, 51]. Nevertheless, these models are still in infancy, and are lagging behind image models in terms of quality (resolution and video length). On the other side of the spectrum, a surge of methods proposes to tune or directly leverage a 2D text-image model for video editing tasks [e.g., 3, 6, 40, 59, 65]. We also follow this approach in conjunction with generating an explicit 3D scene representation.
Text-to-3D Generation and Editing. Recently, language-vision pre-trained models [41, 49], and differentiable renderers [30, 35] have been utilized for text-driven 3D content generation. For example, CLIP-based losses have been used for the text-driven stylization of a given surface mesh [34] or for synthesizing a NeRF from an input text [23]. Taking this approach further, Poole et al. [39] distills the generative prior of a pre-trained text-to-image diffusion model by encouraging the optimized NeRF to render images that are in the distribution of the diffusion model [49]. Singer et al. [52] extends this approach to a 4D dynamic NERF and optimizes using a pre-trained text-to-video model. Nevertheless, it is limited to object-centric scenes while we generate long-term walkthrough-type videos. On the other hand, Chang et al. [10] and Ma et al. [33] construct the semantic scene graph from natural language and retrieve the required elements from the 3D objects databases.
Finally, in a very recent concurrent work, Text2Room [19] proposed a similar approach to ours to generate 3D scenes based on textual prompts. However, this work focuses specifically on creating room meshes and tailors both the camera trajectory, prompts and mesh rendering for this purpose. In contrast, we use the mesh representation only as a tool to achieve videos of diverse scenes.
The input to our method is a text prompt  $P$ , describing the target scene, and a camera trajectory denoted by  $\{C_i\}_{i=1}^T$ , where  $C_i \in \mathbb{R}^{3 \times 4}$  is the camera pose of the  $i^{th}$  frame. Our goal is to synthesize a long-term, high-quality video depicting the desired scene while adhering to the camera trajectory.
Our framework, illustrated in Fig. 2, generates the video one frame at a time, by leveraging a pretrained text-to-image diffusion model [48], and a pre-trained monocular depth prediction model [42, 43]. The role of the diffusion model is to synthesize new content that is revealed as the camera moves, whereas the depth prediction model allows us to estimate the geometry of the newly generated content. We combine these two models through a unified 3D representation of the scene, which ensures geometrically consistent video generation. More specifically, at each time step  $t$ , the new synthesized content and its predicted depths are used to update a unified mesh representation  $\mathcal{M}$  which is constructed progressively as the camera moves. The mesh is then projected to the next view  $C_{t+1}$ , and the process is repeated.
Since neither the depth prediction model nor the inpainting model are guaranteed to produce consistent predictions across time, we finetune them both to match the content of our unified scene representation for the known, previously synthesized content. We next describe each of these components in detail.
3.1 Initialization and Scene Representation
We denote the pre-trained text-to-image inpainting model by  $\phi$ , which takes as input a text prompt  $P$ , a mask  $M$ , and a masked conditioning image  $I_{t}^{M}$ :
$$
\boldsymbol {I} _ {t} = \phi (\boldsymbol {M}, \boldsymbol {I} _ {t} ^ {M}, \boldsymbol {P}) \tag {1}
$$
The first frame in our video  $I_0$  is generated by sampling an image from  $\phi$ , using  $P$  without masking (all-ones mask). The generated frame is then fed to a pre-trained depth prediction model  $g$ , which estimates  $D_0$  the depth map for  $I_0$ , i.e.,  $D_0 = g(I_0)$ .
With the estimated depth and camera poses, a naive approach for rendering  $I_{t+1}$  is to warp  $I_t$  to  $C_{t+1}$ , e.g., using 2D splatting [37]. However, even with consistent depths, rendering a 3D-consistent scene is challenging. In typical camera motion, multiple pixels from one frame are mapped into the same pixel in the next frame. Thus, the aggregated color needs to be selected carefully. Furthermore, frame-to-frame warping lacks a unified representation, thus, once the scene content gets occluded, it will be generated from scratch when it gets exposed again
To address these challenges, we take a different approach and represent the scene with a unified triangle mesh  $\mathcal{M} = (V, F, E)$  which is represented by a tuple of vertices (3D location and color), faces and edges, respectively. We initialize  $\mathcal{M}_0$  by unprojecting  $(I_0, D_0)$ . After the first frame is generated, each synthesis step consists of the following main stages:
3.2 Project and Inpaint
The current scene content, represented by the unified mesh representation  $\mathcal{M}_t$ , is projected to the next camera  $C_{t + 1}$ :
$$
\left(\boldsymbol {I} _ {t + 1} ^ {M}, \boldsymbol {D} _ {t + 1} ^ {M}, \boldsymbol {M} _ {t + 1}\right) = \operatorname {P r o j} \left(\mathcal {M} _ {t}, \boldsymbol {C} _ {t + 1}\right) \tag {2}
$$
producing a mask  $M_{t+1}$  of the visible content in  $C_{t+1}$ , a masked frame  $I_{t+1}^{M}$ , and a masked depth map  $D_{t+1}^{M}$ . Given  $M_{t+1}$  and  $I_{t+1}^{M}$ , we now turn to the task of synthesizing new content. To do so, we apply  $\phi$  using Eq. 1.
3.3 Test-time Depth Finetuning
We would like to estimate the geometry of the new synthesized content and use it to update our unified scene mesh. A naive approach to do so is to estimate the depth directly from the inpainted frame, i.e.,  $D_{t + 1} = g(I_{t + 1})$ . However, monocular depth predictions tend to be inconsistent, even across nearby video frames [27, 31, 64]. That is, there is no guarantee the predicted depth  $D_{t + 1}$  would be well aligned with the current scene geometry  $\mathcal{M}_t$ . We mitigate this problem by taking a test-time training approach to finetune the depth prediction model to be as consistent as possible with the current scene geometry. This approach is inspired by previous methods [27, 31, 64], which encourage pairwise consistency w.r.t optical flow observations in a given video. In contrast, we encourage consistency between the predicted depth and our unified 3D representation in the visible regions in frame  $t + 1$ . Formally, our training loss is given by:
$$
\mathcal {L} _ {\text {d i s p}} = \left\| \boldsymbol {D} _ {t + 1} ^ {M} - g (\boldsymbol {I} _ {t + 1}) \odot \boldsymbol {M} _ {t + 1} \right\| _ {1} \tag {3}
$$
where  $D_{t+1}^{M}, M_{t+1}$  are given by projecting the mesh geometry to the current view (Eq. 2). To avoid catastrophic forgetting of the original prior learned by the depth model, we revert  $g$  to its original weights at each generation step.
3.4 Test-time Decoder Finetuning
Our framework uses Latent Diffusion inpainting model [48], in which the input image and mask are first encoded into a latent space  $z = E_{\mathrm{LDM}}(M_t, I_t^M)$ ; the inpainting operation is then carried out in the latent space, producing  $\hat{z}$ , and the final output image is given by decoding  $\hat{z}$  to an RGB image:  $D_{\mathrm{LDM}}(\hat{z})$ . This encoding/decoding operation is lossy, and the encoded image is not reconstructed exactly in the unmasked regions. Following Avrahami et al. [2], for each frame, we finetune the decoder as follows:
$$
\mathcal {L} _ {d e c} = \left\| D _ {\mathrm {L D M}} (\hat {z}) \odot \boldsymbol {M} _ {t + 1} - I _ {t + 1} ^ {M} \right\| _ {2} + \left\| \left(D _ {\mathrm {L D M}} (\hat {z}) - D _ {\mathrm {L D M}} ^ {\text {f i x e d}} (\hat {z})\right) \odot (1 - \boldsymbol {M} _ {t + 1}) \right\| _ {2} \tag {4}
$$
The first term encourages the decoder to preserve the known content, and the second term is a prior preservation loss [2]. As in Sec. 3.3, we revert the weights of the decoder to its original weights.
3.5 Online Mesh Update
Given the current mesh  $\mathcal{M}_t$ , and the inpainted frame  $I_{t + 1}$ , we use our depth map and camera pose to update the scene with the newly synthesized content. Note that in order to retain the previously synthesized content, some of which may be occluded in the current view, we consider only the new content, unproject, and mesh it, adding it to  $\mathcal{M}_t$ . That is,
$$
\tilde {\mathcal {M}} _ {t + 1} = \operatorname {U n P r o j} \left(M _ {t + 1}, I _ {t + 1}, D _ {t + 1}, C _ {t + 1}\right) \tag {5}
$$
Finally,  $\tilde{\mathcal{M}}_{t + 1}$  is merged into the mesh:
$$
\mathcal {M} _ {t + 1} = \mathcal {M} _ {t} \cup \tilde {\mathcal {M}} _ {t + 1} = \left(\boldsymbol {V} _ {t} \cup \tilde {\boldsymbol {V}} _ {t + 1}, \boldsymbol {F} _ {t} \cup \tilde {\boldsymbol {F}} _ {t + 1}, \boldsymbol {E} _ {t} \cup \tilde {\boldsymbol {E}} _ {t + 1}\right) \tag {6}
$$
In practice, we also add to the mesh the triangles used to connect the previous content to the newly synthesized one. See Sec. A.1 of appendix for more details.
3.6 Rendering
Rendering a high-quality and temporally consistent video from our unified mesh requires careful handling. First, stretched triangles may often be formed along depth discontinuities. These triangles encapsulate content that is unseen in the current view, which we would like to inpaint when it becomes visible in other views. To do so, we automatically detect stretched triangles based on their normals and remove them from the updated mesh.
Another issue arises when the content at the border of the current frame is close to the camera relative to the scene geometry. This content is typically mapped towards the interior of the next frame due to parallax. This often results in undesirable "floating artifacts" where the revealed border regions are rendered with the existing distant content (see Fig. 3). We assume that such regions should be inpainted, thus we automatically detect and include them in the inpainting mask. See Sec. A.2 of appendix for more details.
(a) No floating artifacts fix (left - warped, right - inpainted)
(b) With floating artifacts fix (left - warped, right - inpainted)
Figure 3: Handling floating artifacts
We evaluate our method both quantitatively and qualitatively on various scenes generated from a diverse set of prompts, including photorealistic as well
"a dimly lit library, with rows upon rows of leather-bound books and dark wooden shelves"
"A grand, marble staircase spirals up to a vaulted ceiling in a grand entrance hall of a palace."
"POV, haunted house, dark, wooden door, spider webs, skeletons"
"inside a castle made of ice, beautiful photo, masterpiece"
"walkthrough, sci-fi ship interiors, corridors, amazing quality, masterpiece, beautiful scenery, best quality"
"walkthrough, inside a medieval forge, metal, fire, beautiful photo, masterpiece"
Figure 4: Sample frames from our generated videos. Our method generates high-quality and diverse scenes while following a desired camera trajectory. All videos are included in SM.
as imaginary scenes. We generated 50-frame long videos, with a fast-moving camera motion, which follows a backward motion combined with rotations of the camera (see Sec. A of appendix for more details). Since our method represents the scene via a mesh, we focus mainly on indoor scenes. Synthesizing outdoor scenes would require careful handling in cases of dramatic depth discontinuities (as discussed in Sec. 5).
Figure 1 and Fig. 4 show sample frames from our videos. The results demonstrate the effectiveness of our method in producing high-quality and geometrically-plausible scenes, depicting significant parallax, complex structures, and various complex scene properties such as lighting or diverse materials (e.g., ice). Additional results, videos, and output depth maps are included in the Supplementary materials (SM).
Figure 5: Sample ablation results of (a) our full method, (b) w/o depth finetuning (Sec. 3.3), (c) w/o decoder finetuning (Sec. 3.4) and (d) w/o mesh (Sec. 3.1). We present sample frames, output depth maps, and COLMAP's depth maps. As seen, all components are essential for obtaining high-quality and 3D-consistent videos.
Rot. (deg.) ↓	Trans. (%) ↓	Reproj. (pix)↓	SI-RMSE ↓	Density (%) ↑	CLIP-AS ↑	AMT(%)↑
Ours	0.61	0.01	0.33	0.17	0.91	5.85	-
w/o depth f.	2.24	0.02	0.37	0.41	0.90	5.75	52.7
w/o decoder f.	2.81	0.02	0.82	0.17	0.87	5.74	63.0
w/o mesh	3.55	0.04	0.57	0.18	0.78	5.73	69.1
Warp-inpaint	4.89	0.09	0.97	0.67	0.70	5.43	85.6
Table 1: Ablations. For each ablation baseline, we report, from left to right: accuracy of estimated camera poses, reprojection error, depth SI-RMSE, and the percentage of reconstructed video pixels. We also report CLIP aesthetics score [1] and measure visual quality using AMT user study, where we report the percentage of judgments in favor of our full method. See Sec. 4.1 and Sec. 4.2 for more details.
4.1 Metrics
In all experiments, we use the following metrics for evaluation:
Depth consistency. We evaluate the depth consistency of a video by applying COLMAP, using known camera poses, and measuring the Scale-Invariant Root Mean Square Error (SI-RMSE) [14]:
$$
\operatorname {S I} - \operatorname {R M S E} = \sqrt {\frac {1}{N ^ {2}} \sum_ {\mathbf {p} , \mathbf {q} \in I} \left[ \left(\log D ^ {o} (\mathbf {p}) - \log D ^ {o} (\mathbf {q})\right) - \left(\log D ^ {c} (\mathbf {p}) - \log D ^ {c} (\mathbf {q})\right) \right] ^ {2}} \tag {7}
$$
where  $D^{o}$  and  $D^{c}$  are our output depth maps and COLMAP depths, respectively. We compute the SI-RMSE for each frame and report the average per video. To assess the completeness of the reconstruction, we also measure the percentage of pixels recovered by COLMAP, i.e., passed its geometric consistency test. In addition, we report the average reprojection error of COLMAP reconstruction, which indicates the consistency of the video with COLMAP's point cloud.
Pose accuracy. We measure the accuracy of the estimated camera trajectory. Specifically, we provide COLMAP only with intrinsic parameters and align the two camera systems by computing a global rotation matrix [61]. We measure the normalized mean accuracy of camera positions (in percentage, relative to the overall camera trajectory), and camera rotations (in degrees).
Visual quality. We use Amazon Mechanical Turk (AMT) platform for evaluating the visual quality of our videos compared to the competitors. We adopt the Two-alternative Forced Choice (2AFC) protocol, where participants are presented with two side-by-side videos, ours and the competitor's (at random left-right ordering) and are asked: "Choose the video/image that has better visual quality, e.g., sharper, less artifacts such as holes, stretches, or distorted geometry". Additionally, we compute the CLIP aesthetic score [1]—an image aesthetic linear predictor learned on top of CLIP embeddings.
Figure 6: Baseline comparison. Sample results from our method (top), GEN-1 [15] (middle) and VideoFusion [32] (bottom) on 3 different prompts. Other methods exhibit saturation and blurriness artifacts in addition to temporally inconsistent frames.
4.2 Ablation