SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Depth. Table 3 reports the results on NYU depth v2 and KITTI (see Section D for more detailed results and Section B for qualitative comparison with DPT on NYU). We achieve a state-of-the-art absolute relative error of 0.074 on NYU depth v2. On KITTI, our method performs competitively with prior work. We report results with averaging depth maps from one or more samples. Note that most prior works use post processing that averages two samples, one from the input image, and the other based on its reflection about the vertical axis.
Flow. Table 1 reports the zero-shot results of our model on Sintel and KITTI Train datasets where ground truth are provided. The model is trained on our newly proposed pre-training mixtures (AutoFlow (AF), FlyingThings (FT), Kubric (KU), and TartanAir (TA)). We report results by averaging 8 samples at a coarse resolution and then refining them to the full resolution as described in Section 3.3. For a fair comparison, we re-train RAFT on this pre-training mixture; this new RAFT model significantly outperforms the original RAFT model. And our diffusion model outperforms the stronger
Table 3: Performance comparison on the NYU-Depth-v2 and KITTI datasets.  $\top$  indicates method uses unsupervised pretraining,  $\dagger$  indicates supervised pretraining and  $\ddagger$  indicates use of auxiliary supervised depth data. Best / second best results are bolded / underlined respectively.  $\downarrow$ : lower is better  $\uparrow$ : higher is better.
Method	Architecture	NYU-Depth-v2	KITTI
δ1↑	REL↓	RMS↓	δ1↑	REL↓	RMS↓
TransDepth [87]	Res-50+ViT-B†	0.900	0.106	0.365	0.956	0.064	2.755
DPT [50]	Res-50+ViT-B†‡	0.904	0.110	0.357	0.959	0.062	2.573
BTS [34]	DenseNet-161†	0.885	0.110	0.392	0.956	0.059	2.756
AdaBins [3]	E-B5+Mini-ViT†	0.903	0.103	0.364	0.964	0.058	2.360
BinsFormer [35]	Swin-Large†	0.925	0.094	0.330	0.974	0.052	2.098
PixelFormer [1]	Swin-Large†	0.929	0.090	0.322	0.976	0.051	2.081
MIM [80]	SwinV2-LT	0.949	0.083	0.287	0.977	0.050	1.966
AiT-P [46]	SwinV2-LT	0.953	0.076	0.279	-	-	-
DDVM	samples=1	Efficient U-NetT‡	0.944	0.075	0.324	0.964	0.056	2.700
samples=2	Efficient U-NetT‡	0.944	0.074	0.319	0.965	0.055	2.660
samples=4	Efficient U-NetT‡	0.946	0.074	0.315	0.965	0.055	2.613
Table 4: Ablation on infilling and step-unrolling. Without either one, performance deteriorates. Without both, optical flow models fail to train on KITTI.
NYU val	KITTI val	KITTI val
REL	RMS	REL	RMS	AEPE	Fl-all
Baseline	0.079	0.331	0.222	3.770	-	-
Step-unroll	0.076	0.324	0.085	2.844	1.84	6.16%
Infill	0.077	0.338	0.057	2.744	1.53	5.24%
Step-unroll & infill	0.075	0.324	0.056	2.700	1.47	4.74%
Table 5: Coarse-to-fine refinement improves zero-shot optical flow estimation results on Sintel and KITTI, along with the qualitative improvements shown in Figure 6.
Coarse-to-fine refinement	Sintel.clean	Sintel.final	KITTI
AEPE	AEPE	AEPE	Fl-all
Without	1.42	2.12	2.35	8.65%
With	1.24	2.00	2.19	7.58%
RAFT baseline. It achieves the state-of-the-art zero-shot results on both the challenging Sintel Final and KITTI datasets.
Figure 4 provides a qualitative comparison of pre-trained models. Our method demonstrates finer details on both object and motion boundaries. Especially on KITTI, our model recovers fine details remarkably well, e.g. on trees and its layered motion between tree and background.
We further finetune our model on the mixture of the following datasets, AutoFlow, FlyingThings, HD1K, KITTI, Sintel, and VIPER. Table 2 reports the comparison to state-of-the-art optical flow methods on public benchmark datasets, Sintel and KITTI. On KITTI, our method outperforms all existing optical flow methods by a substantial margin (even most scene flow methods that use stereo inputs), and sets the new state of the art. On the challenging Sintel final, our method is competitive with other state of the art models. Except for methods using warm-start strategies, our method is only behind FlowFormer which adopts strong domain knowledge on optical flow (e.g. cost volume, iterative refinement, or attention layers for larger context) unlike our generic model. Interestingly, we find that our model outperforms FlowFormer on 11/12 Sintel test sequences and our overall worse performance can be attributed to a much higher AEPE on a single (possibly out-of-distribution) test sequence. We discuss this in more detail in Section 5. On KITTI, our diffusion model outperforms FlowFormer by a large margin  $(30.34\%)$ .
Infilling and step-unrolling. We study the effect of infilling and step-unrolling in Table 4. For depth, we report results for fine-tuning our pre-trained model on the NYU and KITTI datasets with the same resolution and augmentations as our best results. For flow, we fine-tune on the KITTI train set alone (with nearest neighbor resizing to the target resolution being the only augmentation) at a resolution of  $320 \times 448$  and report metrics on the KITTI val set [39]. We report results with a single sample and no coarse-to-fine refinement. We find that training on raw sparse data without infilling and step unrolling leads to poor results, especially on KITTI where the ground truth is quite sparse. Step-unrolling helps to stabilize training without requiring any extra data pre-processing. However, we find that most gains come from interpolating missing values in the sparse labels. Infilling and step-unrolling compose well as our best results use both; infilling (being an approximation) does not completely bridge the training-inference distribution shift of the noisy latent.
Table 6: The addition of optical flow synthetic datasets substantially improves the zero-shot results on Sintel and KITTI.
Dataset	Sintel.clean	Sintel.final	KITTI	KITTI F1-all
AF pretraining	2.04	2.55	4.47	16.59%
AF→AF+FT	1.48	2.22	3.71	14.07%
AF→AF+FT+KU	1.33	2.04	2.82	9.27%
AF→AF+FT+KU+TA	1.24	2.00	2.19	7.58%
Table 7: The addition of synthetic depth data in pre-training substantially improves fine-tuning performance on NYU.
Dataset	REL	RMS
SceneNet RGB-D	0.089	0.362
ScanNet	0.081	0.346
SceneNet RGB-D + ScanNet	0.075	0.324
Figure 6: Visual results with and without coarse-to-fine refinement. For our pretrained model, refinement helps correct wrong flow and adds details to correct flow.
Coarse-to-fine refinement. Figure 6 shows that coarse-to-fine refinement (Section 3.3) substantially improves fine-grained details in estimated optical flow fields. It also improves the metrics for zero-shot optical flow estimation on both KITTI and Sintel, as shown in Table 5.
Datasets. When using different mixtures of datasets for pretraining, we find that diffusion models sometimes capture region boundaries and shape at the expense of local textural variation (eg see Figure 3). The model trained solely on AutoFlow tends to provide very coarse flow, and mimics the object shapes found in AutoFlow. The addition of FlyingThings, Kubric, and TartanAir removes this hallucination and significantly improves the fine details in the flow estimates (eg, shadows, trees, thin structure, and motion boundaries) together with a substantial boost in accuracy (cf. Table 6). Similarly, we find that mixing SceneNet RGB-D [41], a synthetic dataset, along with ScanNet [9] provides a performance boost for fine-tuning results on NYU depth v2, shown in Table 7.
4.3 Interesting properties of diffusion models
Multimodality. One strength of diffusion models is their ability to capture complex multimodal distributions. This can be effective in representing uncertainty, especially where there may exist natural ambiguities and thus multiple predictions, e.g. in cases of transparent, translucent, or reflective cases. Figure 1 presents multiple samples on the NYU, KITTI, and Sintel datasets, showing that our model captures multimodality and provides plausible samples when ambiguities exist. More details and examples are available in Section A.
Imputation of missing labels. A diffusion model trained to model the conditional distribution  $p(y|x)$  can be zero-shot leveraged to sample from  $p(y|x, y_{\text{partial}})$  where  $y_{\text{partial}}$  is the partially known label. One approach for doing this, known as the replacement method for conditional inference [63], is to replace the known portion the latent  $y_t$  at each inference step with the noisy latent built by applying the forward process to the known label. We qualitatively study the results of leveraging replacement guidance for depth completion and find it to be surprisingly effective. We illustrate this by building a pipeline for iteratively generating 3D scenes (conditioned on a text prompt) as shown in Figure 7 by leveraging existing models for text-to-image generation and text-conditional image inpainting. While a more thorough evaluation of depth completion and novel view synthesis against existing methods is warranted, we leave that exploration to future work. (See Section C for more details and examples.)
Latency. We adopt standard practices from image-generation models, leading to larger models and slower running times than RAFT. However, we are excited by the recent progress on progressive distillation [42, 56] and consistency models [64] to improve inference speed in diffusion models.
Figure 7: Application of zero-shot depth completion with our model by incorporating it into an iterative 3D scene generation pipeline. Starting with a initial image (optionally generated from a text-to-image model), we sample an image-only conditioned depth map using our model. The image-depth pair is added to a point cloud. We then iteratively render images and depth maps (with holes) from this point cloud by moving the camera. We then fill image holes using an existing image inpainter (optionally text conditioned), and then use our model with replacement guidance to impute missing depths (conditioned on the filled RGB image and known depth).
Sintel fine-tuning. Under the zero-shot setting, our method achieves state-of-the-art results on both Intel Final and KITTI. Under the fine-tuning setting, ours is state-of-the-art on KITTI but is behind FlowFormer [23] on Intel Final. We discuss several possible reasons for why this may be the case.
- We follow the fine-tuning procedure in [70]. While their zero-shot RAFT results are comparable to FlowFormer on Sintel and KITTI, the fine-tuned RAFT-it is significantly better on KITTI but less accurate on Sintel than FlowFormer. It is possible that the fine-tuning procedure (e.g. dataset mixture or augmentations) developed in [70] is more suited for KITTI than Sintel.
- Another possible reason is that there is substantial domain gap between the training and test data on Sintel than KITTI. On Sintel test, there is a particular sequence "Ambush 1", where the girl's right arm moves out of the image boundary. Our method has an AEPE close to 30 while FlowFormer has lower than 10. It is likely that the attention on the cost volume mechanism by FlowFormer can better reason about the motion globally and handles this particular sequence well. This particular sequence may account for the major difference in the overall results; among 12 available results on the Sintel website, ours has lower AEPE on 11 sequences but a higher AEPE on the "Ambush 1" sequence, as shown in Table 8. Figure 16 in the appendix further provides visualization.
Table 8: Average end-point error (AEPE) on 12 Sintel test sequences available from the public website.
Sequence	Ours	FlowFormer [23]
Perturbed Market 3	0.787	0.869
Perturbed Shaman 1	0.219	0.252
Ambush 1	29.33	8.141
Ambush 3	2.855	2.973
Bamboo 3	0.415	0.577
Cave 3	2.042	2.352
Market 1	0.719	1.174
Market 4	5.517	8.768
Mountain 2	0.176	0.518
Temple 1	0.452	0.612
Tiger	0.413	0.596
Wall	1.639	1.723
We introduced a simple denoising diffusion model for monocular depth and optical flow estimation using an image-to-image translation framework. Our generative approach obtains state-of-the-art results without task-specific architectures or loss functions. In particular, our model achieves an F1-all score of  $3.26\%$  on KITTI, about  $25\%$  better than the best published method [70]. Further, our model captures the multi-modality and uncertainty through multiple samples from the posterior. It also allows imputation of missing values, which enables iterative generation of 3D scenes conditioned on a text prompt. Our work suggests that diffusion models could be a simple and generic framework for dense vision tasks, and we hope to see more work in this direction.
We thank Ting Chen, Daniel Watson, Hugo Larochelle and the rest of Google DeepMind for feedback on this work. Thanks to Klaus Greff and Andrea Tagliasacchi for their help with the Kubric generator, and to Chitwan Sahara for help training the Palette model.
[1] Ashutosh Agarwal and Chetan Arora. Attention Attention Everywhere: Monocular depth prediction with skip attention. In WACV, 2023.
[2] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. NIPS, 2015.
[3] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. AdaBins: Depth estimation using adaptive bins. In CVPR, pages 4009-4018, 2021.
[4] Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael J. Black. A naturalistic open source movie for optical flow evaluation. In ECCV, pages 611-625, 2012.
[5] Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen. Estimating depth from monocular images as classification using deep fully convolutional residual networks. IEEE T-CSVT, 28(11): 3174-3182, 2017.
[6] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An information-rich 3D model repository. arXiv:1512.03012, 2015.
[7] Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J. Fleet. A generalist framework for panoptic segmentation of images and videos. In ICCV, 2023.
[8] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In ICLR, 2023.
[9] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, 2017.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009.
[11] Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2022.
[12] David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In ICCV, pages 2650-2658, 2015.
[13] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS, volume 27, 2014.
[14] Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning optical flow with convolutional networks. In ICCV, pages 2758-2766, 2015.
[15] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, pages 2002–2011, 2018.
[16] Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and Ian Reid. Unsupervised CNN for single view depth estimation: Geometry to the rescue. In ECCV, pages 740-756, 2016.
[17] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets Robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11):1231-1237, 2013.
[18] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth estimation. In ICCV, pages 3828-3838, 2019.
[73] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114, 2019.
[74] Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In ECCV, pages 402-419, 2020.
[75] Su Wang, Chitwan Sahara, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Soricut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan. Imagen Editor and EditBench: Advancing and evaluating text-guided image inpainting. In CVPR, 2023.
[76] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. TartanAir: A dataset to push the limits of visual SLAM. In IROS, 2020.
[77] Philippe Weinzaepfel, Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid. DeepFlow: Large displacement optical flow with deep matching. In ICCV, pages 1385-1392, 2013.
[78] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. SynSin: End-to-end view synthesis from a single image. In CVPR, 2020.
[79] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-280, 1989.
[80] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In CVPR, 2023.
[81] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. GMFlow: Learning optical flow via global matching. In CVPR, pages 8121-8130, 2022.
[82] Li Xu, Jiaya Jia, and Yasuyuki Matsushita. Motion detail preserving optical flow estimation. IEEE T-PAMI, 34(9):1744-1757, 2011.
[83] Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. NeurIPS, 2019.
[84] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
[85] Feihu Zhang, Oliver J. Woodford, Victor Adrian Prisacariu, and Philip H.S. Torr. Separable flow: Learning motion cost volumes for optical flow estimation. In ICCV, pages 10807-10817, 2021.
[86] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, pages 649-666, 2016.
[87] Jiawei Zhao, Ke Yan, Yifan Zhao, Xiaowei Guo, Feiyue Huang, and Jia Li. Transformer-based dual relation graph for multi-label image recognition. In ICCV, pages 163-172, 2021.
[88] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE T-PAMI, 2017.
A Multimodal prediction
We provide more qualitative examples for multimodal prediction. Figures 8 and 9 illustrate multimodal depth predictions on NYU and KITTI respectively. Multimodality of the posterior distribution exists in regions where there are multiple plausible predictions. For example, this includes reflective and transparent surfaces (mirrors and glass surfaces in rows 1 to 5 of Figure 8 and windows of cars in Figure 9). We further find that the model captures uncertainty in depth estimates in the vicinity of object boundaries, some of which arise due to noise in ground truth measurements in the training data. This can be observed at the boundaries of cars in Figure 9 and around edges of objects in Figure 8 (most clearly visible in the last row).
Figure 10 illustrates different samples on KITTI from the optical flow diffusion model, also capturing multiple modes of the predictive posterior. Multimodality exists on transparent surfaces and near occlusions. As shown in Figure 11, on Sintel, multimodality also exists on occluded or out-of-bounds pixels where multiple predictions are plausible.
Figure 8: Qualitative examples of multimodal estimation on the NYU depth dataset. Our model is able to output multiple plausible depth maps where ambiguity exists. Rows 1 to 5 show transparent or reflective surfaces where two answers exist. In all samples (specially the last row) we observe the model's ability to capture uncertainty in depth near object boundaries (see the areas of high variance).
Figure 9: Qualitative examples of multimodal depth estimation on KITTI. Our model is able to predict multimodal samples, especially windows of cars and object boundaries. Please refer to the areas with high variance.
Figure 10: Qualitative examples of multimodal optical flow estimation on KITTI. Multimodality exists on transparent surfaces (e.g., windows of cars) and shadows where our model estimates layered motion in different samples (see the last row).
Input images
Figure 11: Qualitative examples of multimodal optical flow estimation on Sintel. Multimodality also exists on examples with challenging occlusion or out-of-bound cases.
Variance heat map & multimodal samples
B Qualitative comparison of depth estimation with DPT
Figure 12 provides a qualitative comparison of our model with DPT-Hybrid [50] finetuned on the NYU depth v2 [61] dataset. The depth estimates of our diffusion model are more accurate both on coarse-scale scene structure (walls, floors, etc.) and on individual objects.
C More samples for zero-shot imputation of depth
Figure 13 provides samples generated using our iterative text-to-3D pipeline. We note that such pipelines for iteratively generating 3D scenes have been previously proposed in literature [37, 60, 78]. However, these methods explicitly learn networks to refine the color [37, 60, 78] and the depth map [37, 60]. In contrast, we propose leveraging the text-conditioned image prior from existing large scale text-to-image [55] and text-conditional image completion [75] models, and use our depth estimation model zero-shot for depth completion. One caveat with our current approach of using the replacement method for conditional inference [63] for imputing depth, is that it does not enable one to fix errors in the depth predicted in the previous step. One approach to fix artifacts would be by noisig-denoising, like that used for coarse-to-fine refinement. We leave further exploration into this to future work.
D Complete depth results on NYU and KITTI
Tables 9 and 10 provide detailed results on the val set of NYU depth v2 and KITTI depth datasets. We follow the standard evaluation protocol used in prior work [35]. For both the NYU depth v2
Figure 12: Qualitative comparison of our model with DPT-Hybrid [50] (fine-tuned on NYU) on the NYU depth v2 val set. Our method infers better depth for both scene structure (walls, floors, etc.) and individual objects. Specific differences are highlighted with red arrows.
Prompt: A living room.
Prompt: A library.
Prompt: A meeting room.
Prompt: A kitchen.
Prompt: A warehouse.
Prompt: A movie theatre.
Figure 13: Text-to-3D samples. Given a text prompt, an image is first generated using Imagen [55] (first row of first column), after which depth is estimated (second row of first column). Subsequently the camera is moved to reveal new parts of the scene which are infilled using an image completion model and our model (which conditions on both the incomplete depth map and the filled image). At each step, newly generated RGBD points are added to a global point cloud which is visualized in the rightmost column.
Table 9: Comparison of performance on the NYU-Depth-v2 dataset.  $\top$  indicates method uses unsupervised pretraining,  $\dagger$  indicates supervised pretraining and  $\ddagger$  indicates use of auxiliary supervised depth data. Best / second best results are bolded / underlined respectively.  $\downarrow$ : lower is better and  $\uparrow$ : higher is better.
Method	Architecture	δ1↑	δ2↑	δ3↑	REL↓	RMS↓	log10↓
TransDepth [87]	Res-50+ViT-B†	0.900	0.983	0.996	0.106	0.365	0.045
DPT [50]	Res-50+ViT-B†‡	0.904	0.988	0.998	0.110	0.357	0.045
AdaBins [3]	E-B5+Mini-ViT†	0.903	0.984	0.997	0.103	0.364	0.044
BinsFormer [35]	Swin-Large†	0.925	0.989	0.997	0.094	0.330	0.040
PixelFormer [1]	Swin-Large†	0.929	0.991	0.998	0.090	0.322	0.039
MIM [80]	SwinV2-LT	0.949	0.994	0.999	0.083	0.287	0.035
AiT-P [46]	SwinV2-LT	0.953	0.993	0.999	0.076	0.279	0.033
DDVM	samples=1	Efficient U-NetT‡	0.944	0.986	0.995	0.075	0.324
samples=2	Efficient U-NetT‡	0.944	0.987	0.996	0.074	0.319
samples=4	Efficient U-NetT‡	0.946	0.987	0.996	0.074	0.315