SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We ablate the key components in our framework: (i) depth finetuning (Sec. 3.3), (ii) decoder finetuning (Sec. 3.4), and (iii) our unified mesh representation (Sec. 3.1). For the first two ablations, we apply our method without each component, and for the third, we replace our mesh with 2D rendering, i.e., warping the content from the current frame to the next, using a depth-aware splatting [37]. In addition, we consider the baseline of simple warp-inpaint, i.e., using splatting with the original estimated depth (w/o finetuning). We ran these ablations on 20 videos, using a set of 10 diverse text prompts. See Sec. B for the list of prompts and SM sample videos.
Table 1 reports the above COLMAP metrics (see Sec. 4.1). Our full method results in notably more accurate camera poses, denser reconstructions, and better-aligned depth maps. Excluding depth finetuning significantly affects the depth consistency (high SI-RMSE). Excluding decoder finetuning leads to high-frequency temporal inconsistency, which leads to higher reprojection error. Without a mesh, the reconstruction is significantly sparser and we observe a drop in all metrics. These results are illustrated in Fig. 5 for a representative video.
To assess visual quality, we follow the AMT protocol described in Sec. 4.1, where we ran each test w.r.t. one of the baselines. We collected 4,000 user judgments from 200 participants over 20 videos. As seen in Table 1, the unified mesh plays a major role in producing high-quality videos, where the judgments were ambiguous w.r.t w/o depth finetuning baseline. We hypothesize this is because this baseline still leverages the mesh representation, thus the visual artifacts are temporally consistent, making them more difficult to perceive.
4.3 Comparison to Baselines
To the best of our knowledge, there is no existing method designed for our task - text-driven perpetual view generation. We thus consider the most relevant methods to our setting:
- VideoFusion (VF) [32]. A text-to-video diffusion model², which takes as input a text prompt and produces a 16-frame long video at  $256 \times 256$  resolution.
- GEN-1 [15]. A text-driven video-to-video translation model, where the translation is conditioned on depth maps estimated from an input source video. Note that in our setting the scene geometry is unknown and is constructed with the video, whereas GEN-1 takes the scene geometry as input, thus tackling a simpler task.
Due to the inherent differences between the above methods, we follow a different evaluation protocol in comparison with each one of them.
2To the best of our knowledge, this is the only publicly available text-to-video model.
Rot. (deg.) ↓	Trans. (%) ↓	Reproj. (px) ↓	Density (%) ↑	CLIP-TS ↑	AMT (%) ↑
Ours	-	-	0.29	81.4	0.25	95.97
VF	-	-	0.78	46.36	0.22
Ours	1.71	3.94	0.60	91.72	0.26	69.34
GEN-1	2.47	8.09	1.64	53.96	0.26
Table 2: Comparison to baselines. (top) VideoFusion [32], and (bottom) GEN-1 [15]. We evaluate 3D consistency using COLMAP metrics and measure visual quality using AMT user study, where we report the percentage of judgments in our favor (see Sec. 4.2). Our method outperforms the baselines on all metrics. Note that the experiential setup for these two baselines is different (see Sec. 4.3).
Figure 7: Limitation. Outdoor scenes are poorly represented with our mesh representation (left). Occasionally we may observe quality degradation due to error accumulation (right).
4.3.1 Evaluating 3D-consistency
To compare to VF, we used the same set of prompts as in Sec. 4.2 as input to their model and generated 1000 videos. We evaluate the 3D consistency of VF's videos and ours, using the same metrics as in Sec. 4.2, where we used COLMAP without any camera information, and downsampled our videos to match their resolution. As seen in Table 2, our method significantly outperforms VF in all metrics. This demonstrates the effectiveness of our mesh representation in producing 3D-consistent videos, in contrast to VF, which may learn geometric priors by training on a large-scale video dataset. Note that since VF does not follow a specific camera path, measuring its camera pose accuracy is not possible.
GEN-1 is conditioned on a source video, while our method takes camera trajectories as input. Thus, we used the RealEstate10K dataset [66], consisting of curated Internet videos and corresponding camera poses.
We automatically filtered 22 indoor videos that follow a smooth temporal backward camera motion to adapt it to our method's setting. The camera poses are used as input to our method, whereas the RGB frames are used as input video to GEN-1 while supplying both methods with the same prompt. We generate  $110\mathrm{~video}^3$  using five different prompts from our ten prompts set that allow reasonable editing of a house (such as "library" rather than "cave"). See Sec. B of appendix for more details.
We measure the video's 3D consistency metrics, as discussed in Sec. 4.2, and report them in Table 2. Our method outperforms GEN-1 in all metrics, even though the geometric structure of the video (depth maps) was given to GEN-1 as input.
4.3.2 Visual Quality and Scene Semantics
We conducted an extensive human perceptual evaluation as described in Sec. 4.1. We collect 3,000 user judgments over 60 video pairs from 50 participants. The results of this survey w.r.t. each method are reported in Table 2. As seen, when compared to VF, our method was preferred in  $96\%$  of the cases and in  $69\%$  of the cases in comparison with GEN-1. Sample results of those comparisons can be seen in Fig. 6. Note that since GEN-1, VF, and StableDiffusion do not exhibit the same underlying data distribution, existing visual quality metrics such as FID [17] cannot be used.
To measure how well our generated videos adhere to the desired scene, we measure the CLIP text-similarity score - mean cosine similarity between the prompt and each of the frames. As seen in Table 2, our method follows the scene text prompt more closely than VF and is on par with GEN-1.
There is only a limited UI access to GEN-1, which restricts the scale of this experiment compared to VF.
We posed the task of text-driven 3D-consistent perpetual view generation and proposed a test-time optimization approach to generate long-term videos of diverse scenes. Our method demonstrates two key principles: (i) how to harness two powerful pre-trained models, allowing us to generate scenes in a zero-shot manner without requiring any training data from a specific domain, and (ii) how to combine these models with a unified 3D scene representation, which by construction ensures feasible geometry, and enables high quality and efficient rendering. As for limitations, the quality of our results depends on the generative and geometric priors and may sometime decrease over time due to error accumulation (Fig. 7 right). In addition, since we represent the scene with triangular meth, it is difficult for our method to represent dramatic depth discontinuities, e.g., sky vs. ground in outdoor scenes (Fig. 7 left). Moreover, since our method works in an online fashion and does not change the previously generated frames, the errors that appeared in previous frames can propagate to the next frames. Devising a new 3D scene representation tailored for the perpetual view generation of arbitrary scenes is an intriguing avenue of future research. We believe that the principles demonstrated in our work hold great promise in developing lightweight, 3D-aware video generation methods.
We thank Shai Bagon for his insightful comments. We thank Narek Tumanyan for his help with the website creation. We thank GEN-1 authors for their help in conducting comparisons. This project received funding from the Israeli Science Foundation (grant 2303/20).
[52] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman. Text-to-4d dynamic scene generation. arXiv:2301.11280, 2023.
[53] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[54] Jiaxiang Tang. Stable-dreamfusion: Text-to-3d with stable-diffusion, 2022. https://github.com/ashawkey/stable-dreamfusion.
[55] Alexandru Cristian Telea. An image inpainting technique based on the fast marching method. Journal of Graphics Tools, 9:23 - 34, 2004.
[56] Hung-Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. In CVPR, 2023.
[57] Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 551-560, 2020.
[58] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In CVPR, 2020.
[59] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. ArXiv, abs/2212.11565, 2022.
[60] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields on complex scenes from a single image. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 736-753. Springer, 2022.
[61] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33, 2020.
[62] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[63] Shangzhan Zhang, Sida Peng, Tianrun Chen, Linzhan Mou, Haotong Lin, Kaicheng Yu, Yiyi Liao, and Xiaowei Zhou. Painting 3d nature in 2d: View synthesis of natural scenes from a single semantic mask. In CVPR, 2023.
[64] Zhoutong Zhang, Forrester Cole, Richard Tucker, William T Freeman, and Tali Dekel. Consistent depth of moving objects in video. ACM Transactions on Graphics (TOG), 40(4):1-12, 2021.
[65] Yuyang Zhao, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Make-a-protagonist: Generic video editing with an ensemble of experts. arXiv preprint arXiv:2305.08850, 2023.
[66] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
We provide implementation details for our framework and finetuning/generation regime.
Runtime. We use the Stable Diffusion [48] model that was additionally finetuned on the inpainted task to perform inpainting. We use DDIM scheduler [53] with 50 sampling steps for each generated frame. Synthesizing 50 frame-long videos with our full method takes approximately 2.5 hours on an NVIDIA TeslaV100 GPU. Specifically, Table 3 reports runtime required for each frame.
Rendering	Inpainting	Depth model finetuning	Decoder finetuning
~40 sec	~5 sec	~40 sec	~60 sec
Table 3: Runtime per-frame. Number of seconds required for each step of the framework. Note that Rendering includes antialIASing and floating artifacts fix steps. In comparison, VideoFusion [32] takes 15 seconds for a 16-frame video of  $256^{*}256$  resolution on an NVIDIA TeslaV100 GPU. Since we do not have direct access to the GEN-1 [15] model, we are limited in assessing its runtime. To get a rough estimate, we measured their online interface runtime, which takes 1 minute to edit a video with 25 frames.
Depth prediction model and LDM decoder finetuning. We use MiDaS-DPT Large [42] as our depth prediction model. For each generated frame, we finetune it for 300 epochs, using Adam optimizer [25] with a learning rate of  $1e - 7$ . Additionally, we revert the weights of the depth prediction model to the initial state, as discussed in Sec. 3.3. We finetune the LDM decoder for 100 epochs on each generation step using Adam optimizer with a learning rate of  $1e - 4$ .
Camera path. Our camera follows a rotational motion combined with translation in the negative depth direction. Starting from a simple translation for  $k$  frames, every  $n$  frames we randomly sample a new rotation direction in the  $x-z$  plane (panning), that camera follows for  $n$  frames. In our experiments, we set  $k = 5$  and  $n = 5$ . We use PyTorch3D [44] to render and update our unified 3D representation.
Mask handling. We observed that the scene's geometry sometimes induces out-of-distribution inpainting masks for the Stable Diffusion inpainting model. To address this issue, we perform a morphological open operation on the inpainting mask:  $M_O = \text{Open}(M)$  with kernel size 3. Then we inpaint the mask difference  $M - M_O$  using Telea [55], while the inpainting model operates on  $M_O$  afterward.
A.1 Mesh update.
As discussed in Sec. 3.5 in the paper, we update the mesh as follows: given an image  $I_{t + 1}$ , a mask of pixels to unproject  $M$ , a corresponding depth map  $D_{t + 1}$  and a camera pose  $C_{t + 1}$ , we unproject the content in a process that is denoted in the main paper by  $\tilde{\mathcal{M}}_{t + 1} = \text{UnProj}(M, I_{t + 1}, D_{t + 1}, C_{t + 1})$ . First, each pixel center is unprojected by its depth value and camera pose into a 3D mesh vertex with the pixel's color as its vertex color. Then, each unprojected four neighboring pixels with coordinates  $(i,j)$ ,  $(i + 1,j)$ ,  $(i,j + 1)$ ,  $(i + 1,j + 1)$  are used to define two adjacent triangle faces.
To prevent holes in the generated mesh we connect the existing mesh  $\mathcal{M}_t$  to the newly unprojected part  $\tilde{\mathcal{M}}_{t + 1}$  by inserting additional triangles. To do this, we first extract the boundary pixels surrounding the inpainting mask  $M$ , and find the faces from  $\mathcal{M}_t$  that are projected to the current view  $C_{t + 1}$ . For each such face, we select the 3D point closest to the current camera center:  $p^* = \underset {p\in (p_1,p_2,p_3)}{\arg \min}\| p - c\|^2,$
where  $c$  is the camera center of  $C_{t + 1}$ , and  $(p_1,p_2,p_3)$  are the points of a given triangle. Finally, we add the selected points to the triangulation scheme described above, which automatically creates the required triangles connecting  $\mathcal{M}_t$  and  $\tilde{\mathcal{M}}_{t + 1}$ .
(a) No antialiasing
Figure 8: Rendering improvements. Left: effect of rendering without antialiasing (a) and with antialiasing (b). Right: effect of rendering without stretched triangles removal (a) and with stretched triangles removal (b).
(a) With antialIASing
(c) No stretch removal
(d) With stretch removal
A.2 Rendering
AntialIASing. Rendering an image from a mesh requires projecting points and faces to a 2D plane. The rasterization process often creates aliasing artifacts (Fig. 8, left), especially when a high-resolution mesh content from an earlier frame is rendered to a later frame, resulting in a large amount of triangle that needs to be rasterized into a low number of pixels. To avoid these artifacts, we apply antialiasing, similar to the image resize antialiasing method - we render the mesh at x2 higher resolution, apply Gaussian blur, and resize it to the required resolution.
Stretched triangles removal. As described in Sec. 3.6, the stretched triangles are forming between close and far away content along regions of depth discontinuities (Fig. 8, right), and we would like to remove them. Following Liu et al. [29], we apply the Sobel filter on the depth map  $D_{t}$  to detect regions of depth discontinuities and threshold the values below the threshold of 0.3. Then we find triangles that are projected to the selected edge regions and filter them based on their normals. Specifically, we keep only the following triangles:  $\{tr_i|(\text{center}(tr_i) - c)^T n < \epsilon\}$ , where  $\epsilon$  is the threshold,  $n$  is the normal of a triangle, and  $c$  is the camera center of  $C_{t+1}$ . In practice, we set  $\epsilon = -0.05$ .
Floating artifacts fix. As described in Sec 3.6, content at the border of the current frame can be mapped towards the interior of the next frame due to parallax. This creates "floating artifacts," shown in Fig. 3 in the paper. To overcome it, we pad the previous depth map  $D_{t}$  with border depth values to  $1.5\mathrm{x}$  the rendering resolution, as if content exists beyond the image borders. Then, after we warp the padded depth to the next camera location, we get a new mask,  $M_{pad}$ , that contains the content beyond the image borders. We use this mask to mask out the floating regions and thus enable the inpainting model to fill those holes with closer content. We perform this procedure on the image that was already rendered in  $2\mathrm{x}$  resolution due to the antialIASing step described above.
B Baseline Comparison Details
VideoFusion. Since VideoFusion [32] does not have an explicit way to control the motion presented in a video, we append “zoom out video” and “camera moving backward” to the input prompt to encourage the generated videos to follow a backward camera motion. We generate 1000 videos of 16 frames each, in 256 resolution. When comparing our method to theirs, we downsample our videos to 256 resolution. In this comparison, we use the full set of 10 prompts:
1. "indoor scene, interior, candy house, fantasy, beautiful, masterpiece, best quality"
2. "POV, haunted house, dark, wooden door, spider webs, skeletons"
3. "walkthrough, an opulent hotel with long, carpeted hallways, beautiful photo, masterpiece, indoor scene"
4. "A dimly lit library, with rows upon rows of leather-bound books and dark wooden shelves"
5. "walkthrough, inside a medieval castle, metal, beautiful photo, masterpiece, indoor scene"
6. "Simple museum, pictures, paintings, artistic, best quality, dimly lit"
7. "walkthrough, sci-fi ship interiors, corridors, amazing quality, masterpiece, beautiful scenery, best quality"
Figure 9: StableDreamFusion sample results. We provide it our camera trajectory and a text prompt "a DSLR photo of the inside  $a^*$ ". Full results can be found in the SM.
8. "A grand, marble staircase spirals up to a vaulted ceiling in a grand entrance hall of a palace. A warm glow on the intricately designed floor"
9. "POV, cave, pools, water, dark cavern, inside a cave, beautiful scenery, best quality"
10. "inside a castle made of ice, beautiful photo, masterpiece"
GEN-1 To compare to GEN-1 [15], we used the RealEstate10K dataset [66], consisting of curated Internet videos and corresponding camera poses. We filtered 22 indoor videos that follow a smooth temporal backward camera motion to adapt it to our method's setting. To do that, we filter videos that adhere to the following constraints:
$$
\frac {\left(c _ {t + 1} - c _ {t}\right) ^ {T} v _ {t}}{\| \left(c _ {t + 1} - c _ {t}\right) \| \cdot \| v _ {t} \|} \geq 0. 9 \quad \forall t = 1, \dots , n \text {f r a m e s},
$$
where  $c_{t}$  is the center of the camera at frame  $t$ , and  $v_{t}$  is the viewing direction (last column of the rotation matrix). We subsample the filtered videos to contain 25 frames and reverse the video if it had a positive average displacement  $(c_{t + 1} - c_{t})$  in the  $z$  direction.
Since MiDaS produces depth maps that have a different range compared to the depth assumed in the RealEstate10K videos, we can't directly take the camera extrinsics from the RealEstate10K [66] data. To align those ranges, we need to find a scaling factor to multiply the MiDaS predictions with. To do that we run COLMAP and compute depth maps, using its dense reconstruction. Our scaling factor is then the ratio of median depth values of COLMAP and MiDaS:
$$
r = \frac {\operatorname {m e d i a n} \left(\left\{D _ {t} ^ {C} \right\} _ {t = 1} ^ {N}\right)}{\operatorname {m e d i a n} \left(\left\{D _ {t} ^ {M} \right\} _ {t = 1} ^ {N}\right)}, \tag {8}
$$
where  $D_{t}^{C}$  and  $D_{t}^{M}$  are the COLMAP and MiDaS depth maps for frame  $t$  respectively, and  $N$  is the number of frames in the video.
For comparison to GEN-1 we used prompts #1-#5 from the above prompt list that allow reasonable editing of a video, depicting an interior of a house.
StableDreamFusion. In addition, we compare our method to StableDreamfusion [54], an open-source text-to-3D model capable of generating an implicit function of a scene (NeRF [35]) from text prompts. We provide it our camera trajectory and a simple text prompt such as "a DSLR photo of the inside a cave" (when providing prompts from our usual prompts set, StableDreamFusion was unable to converge to meaningful results). The generated scenes contain a lot of blur and unrealistic artifacts, as can be seen in Figure 9 since to achieve good visual quality, NERF requires multiple viewpoints of a scene from different angles. Full video results of StableDreamFusion can be found in the SM.
C Broader impact
Our framework is a test-time optimization method, which does not require any training data. Nevertheless, our framework uses two pre-trained models: a depth prediction model and a text-to-image diffusion model [48], which are susceptible to biases inherited to their training data (as discussed in Sec. 5 in [48]). In our context, these models are used to generate 3D static scenes. To avoid harmful content, we consider only text prompts that describe general scenery and objects, while avoiding prompts that involve sensitive content such as humans.
Frame 2
Frame 15
Frame 25
Frame 45
Spiral camera path
Big turns camera path
Original camera path