SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 10: Comparison of performance on the KITTI dataset.  $\top$  indicates method uses unsupervised pretraining,  $\dagger$  indicates supervised pretraining and  $\ddagger$  indicates use of auxiliary supervised depth data. Best / second best results are bolded / underlined respectively.  $\downarrow$ : lower is better and  $\uparrow$ : higher is better. E-B5: EfficientNet-B5 [73].
Method	Backbone	δ1↑	δ2↑	δ3↑	REL ↓	Sq-rel ↓	RMS ↓	RMS log ↓
BTS [34]	DenseNet-161†	0.956	0.993	0.998	0.059	0.245	2.756	0.096
TransDepth [87]	ResNet-50+ViT-B†	0.956	0.994	0.999	0.064	0.252	2.755	0.098
DPT [50]	ResNet-50+ViT-B†‡	0.959	0.995	0.999	0.062	-	2.573	0.092
AdaBins [3]	E-B5-mini-ViT†	0.964	0.995	0.999	0.058	0.190	2.360	0.088
BinsFormer [35]	Swin-Large†	0.974	0.997	0.999	0.052	0.151	2.098	0.079
PixelFormer [1]	Swin-Large†	0.976	0.997	0.999	0.051	0.149	2.081	0.077
MIM [80]	SwinV2-LT	0.977	0.998	1.000	0.050	0.139	1.966	0.075
DDVM	samples=1	Efficient U-NetT‡	0.964	0.994	0.998	0.056	0.339	2.700
samples=2	Efficient U-NetT‡	0.965	0.994	0.998	0.055	0.325	2.660
samples=4	Efficient U-NetT‡	0.965	0.994	0.998	0.055	0.292	2.613
and KITTI datasets we report the absolute relative error (REL), root mean squared error (RMS) and accuracy metrics  $(\delta_i < 1.25^i$  for  $i\in 1,2,3)$ . For NYU we also report absolute error of log depths  $(\log_{10})$ . For KITTI we additionally report the squared relative error (Sq-rel) and root mean squared error of log depths (RMS log). The predicted depth is up-sampled to the full resolution using bilinear interpolation before evaluation. For the indoor model we evaluate on the cropped region proposed by [13] and for the outdoor model the cropped region proposed by [16] as is standard in prior work.
E Ablations
Tables 11 and 12 show that an  $L_{1}$  loss in training the diffusion model performs much better than an  $L_{2}$  loss for monocular depth estimation on NYU and KITTI. Tables 13 and 14 show the effectiveness of Palette-style [54] self-supervised pretraining for monocular depth estimation on NYU and KITTI respectively. All results use a single sample. Because these findings are reasonable and expected to generalize to other dense vision tasks, we do not further ablate them for optical flow estimation for compute efficiency.
Table 11: Ablation for the choice of loss function on the NYU depth v2 dataset.
δ1↑	δ2↑	δ3↑	REL↓	RMS↓	log10↓
L2	0.932	0.981	0.994	0.085	0.349	0.037
L1	0.944	0.986	0.995	0.075	0.324	0.032
Table 12: Ablation for the choice of loss function on the KITTI dataset.
δ1↑	δ2↑	δ3↑	REL ↓	Sq-rel ↓	RMS ↓	RMS log ↓
L2	0.954	0.993	0.998	0.065	0.321	2.773	0.099
L1	0.964	0.994	0.998	0.056	0.339	2.700	0.091
Table 13: Ablation for self-supervised pretraining on the NYU depth v2 dataset.
δ1↑	δ2↑	δ3↑	REL↓	RMS↓	log10↓
No self-supervised pre-training	0.936	0.980	0.992	0.081	0.352	0.035
With self-supervised pre-training	0.944	0.986	0.995	0.075	0.324	0.032
Table 14: Ablation for self-supervised pretraining on the KITTI depth dataset.
δ1↑	δ2↑	δ3↑	REL ↓	Sq-rel ↓	RMS ↓	RMS log ↓
No self-supervised pre-training	0.952	0.990	0.997	0.064	0.389	2.998	0.104
With self-supervised pre-training	0.965	0.994	0.998	0.055	0.332	2.696	0.091
F Coarse-to-fine refinement for depth
Figure 14 demonstrates performance of coarse-to-fine refinement on the NYU depth v2 dataset. While refinement improves fine-scale details in the estimated depth maps, the qualitative improvements are small and we do not find significant quantitative improvements. Hence the results reported in this work do not use coarse-to-fine refinement for depth estimation. Further work is needed to develop a coarse-to-fine algorithm capable of more robust gains in depth estimation.
Figure 14: Samples with coarse-to-fine refinement on the NYU depth v2 dataset. We find that refinement adds sharpness and detail to the depth estimation but does not provide quantitative improvements.
G Coarse-to-fine optical flow refinement for RAFT
For a fair comparison with optical flow estimation, we also apply our coarse-to-fine refinement scheme to RAFT [74], to determine whether our performance gains translate to RAFT as well. We first estimate flow at a low resolution,  $320 \times 448$ , upsample the low-resolution flow to the original resolution, divide original-resolution input images into  $2 \times 5$  overlapping patches of size  $320 \times 448$ , then estimate flow on the cropped patches using the upsampled flow field as the initial guess for the recurrent refinement (12 steps in total) of RAFT [74]. After estimating flow of each patch, we merge them using weighted masks [28]. Table 15 reports the result. Unlike our diffusion-based method, the coarse-to-fine scheme actually hurts the accuracy of RAFT on Sintel Clean and KITTI and only marginally improves the accuracy on Sintel Final. Further exploration into better approaches for coarse-to-fine refinement for RAFT is warranted. We leave that to future work.
H Training and inference details
H.1 Architecture
UNet. The predominant architecture for diffusion models is the U-Net developed for the DDPM model [21], and later improved in several respects [11, 45, 63]. Here we adapt the Efficient U-Net architecture that was developed forImagen [55]. It is more efficient that the U-Nets used in prior work owing to the use of fewer self-attention layers, fewer parameters and less computation at higher
Table 15: Our coarse-to-fine refinement scheme marginally improves the performance of RAFT on Sintel Final while hurting performance on Sintel Clean and KITTI. We report the EPE on the Sintel and KITTI datasets.
Sintel Clean	Sintel Final	KITTI
RAFT baseline	1.27	2.28	2.71
RAFT with the coarse-to-fine refinement	1.35	2.26	2.85
Figure 15: Overview of the Efficient UNet architecture proposed in [55]. CH_IN and CH_OUT refer to the number of input and output channels respectively.  $t$  refers to the time embedding. FiLM refers to the modulation layers proposed in [47].  $N$  is the number of ResNet + self-attention blocks.
resolutions, along with other adjustments that make it well suited to training medium resolution diffusion models.
Specifically we adopt the configuration for the  $64 \times 64 \rightarrow 256 \times 256$  super-resolution model (see Figure 15 for an overview) with several changes. We drop the text cross-attention layers but preserve the self-attention in the lowest resolution layers dblock4 and ublock4 (see Figure 15). For supervised training for the flow model, we find it beneficial to additionally enable self-attention for the last-but-one layers dblock3 and ublock3. The number of input and output channels differ across self-supervised pre-training and supervised pre-training and are also different for flow and depth models. For self-supervised pre-training  $CH\_IN = 6$  and  $CH\_OUT = 3$  (see Figure 15) since the input consists of a 3-channel source RGB image and a 3-channel noisy target image concatenated along the channel dimension and the output is a RGB image. The supervised depth model has  $CH\_IN = 4$  (RGB image + noisy depth) and  $CH\_OUT = 1$ . The supervised optical flow model has  $CH\_IN = 8$  (2 RGB images + noisy flow along  $x$  and  $y$ ) and  $CH\_OUT = 2$ . Note that this means we need to reinitialize the input and output convolutional kernels and biases before the supervised pretraining stage. All other weights are re-used.
Resolution. Our self-supervised model was trained at a resolution of  $256 \times 256$ . The indoor depth model is trained at  $240 \times 320$ . For Waymo we use  $256 \times 384$  and for KITTI depth  $256 \times 832$ . Flow pretraining is done at a resolution of  $320 \times 448$ , and finetuning at  $320 \times 768$ .
H.2 Datasets and augmentation
For unsupervised pre-training, we use the ImageNet-1K [10] and Places365 [88] datasets and train on the self-supervised tasks of colorization, inpainting, uncropping, and JPEG decompression, following [54]. Throughout, we mix datasets at the batch level.
Flow. For supervised flow pretraining we use a mix of AutoFlow (native resolution  $448 \times 576$ ), FlyingThings  $(540 \times 960)$ , Kubric  $(512 \times 512)$  and TartanAir  $(480 \times 640)$  synthetic datasets. We finetune on the standard mixture consisting of AutoFlow, FlyingThings, Viper  $(540 \times 960)$ , HD1K  $(540 \times 1280)$ , Sintel  $(436 \times 1024)$ , and KITTI  $(375 \times 1242)$ .
We follow the same photometric and geometric augmentation schemes from [70], comprising random affine transformation, flipping, and cropping.
Depth. For supervised pre-training of the indoor model we mix the following datasets. ScanNet [9] is a dataset of 2.5M examples captured using a Kinect v1-like sensor. It provides depth maps at  $480 \times 640$  and RGB images at  $968 \times 1296$ . SceneNet RGB-D [41] is a synthetic dataset of 5M images generated by rendering ShapeNet [6] objects in scenes from SceneNet [20] at a resolution of  $240 \times 320$ .
For the outdoor model training we use the Waymo Open Dataset [71], a large-scale driving dataset consisting of about 200k frames. Each frame provides RGB images from 5 cameras and LiDAR maps. We use the RGB images from the FRONT, FRONT_LEFT and FRONT_RIGHT cameras and the TOP LiDAR only to build about 600k aligned RGB depth maps.
For indoor fine-tuning and evaluation we use NYU depth v2 [61], a commonly used dataset for evaluating indoor depth prediction models. It provides aligned image and depth maps at  $480 \times 640$  resolution. We use the official split comprising 50k images for training and 654 for evaluation.
For outdoor fine-tuning and evaluation, we use KITTI [17], an outdoor driving dataset which provides RGB images and LiDAR scans at resolutions close to  $370 \times 1226$ . We use the training/test split proposed by [13], comprising 26k training images and 652 test images.
We use random horizontal flip data augmentation which is common in prior work. Where needed, images and dense depth maps are resized using bilinear interpolation to the model's resolution for training and nearest neighbor interpolation is used for sparse maps.
H.3 Step-unrolling and interpolation of missing depth and flow
As discussed in Section 3.2 of the main paper, infilling and step-unrolling are used to mitigate distribution shift between training and inference with diffusion models. The problem arises due to the missing data in the training depth maps and flow fields.
Infilling. For indoor depth maps, we use nearest neighbor interpolation during training (see Section 3.2 in the main paper). For the outdoor depth data we use nearest neighbor interpolation except for sky regions, as they are often large and are much further from the camera than adjacent objects in the image. We use an off-the-shelf sky segmenter [36], and then set all sky pixels to be the maximum modeled depth (here,  $80\mathrm{m}$ ). For missing optical flow ground truth we employ a simple sequence of 1D nearest neighbor interpolations first along rows, and then along columns.
Table 16: Ablation of the number of unroll steps for monocular depth estimation. Performs improves up to four steps of unrolling and plateaus thereafter. The models trained without infilling missing depth benefit more from a larger number of unroll steps, which is to be expected. Best and second best results are bolded and underlined respectively.
Unroll steps	NYU	KITTI
No infill	Infill	No infill	Infill
REL	RMS	REL	RMS	REL	RMS	REL	RMS
0	0.079	0.331	0.077	0.338	0.222	3.770	0.057	2.744
1	0.076	0.324	0.075	0.324	0.085	2.844	0.056	2.700
2	0.076	0.317	0.075	0.315	0.068	2.799	0.054	2.629
3	0.075	0.316	0.074	0.317	0.061	2.789	0.054	2.591
4	0.075	0.316	0.074	0.315	0.059	2.739	0.053	2.568
Step-unrolling. By default we use a single unroll step in all results where step-unrolling is enabled. In Table 16, we show that using multiple unroll steps can further improve performance on the task of monocular depth estimation.
Finally, while we use infilling and step-unrolling, there are other ways in which one might try to mitigate the problem. One such approach was taken by [46], which faced a similar problem when training a vector-quantizer on depth data. Their approach was to synthetically add more holes following a carefully chosen masking ratio. We prefer our approach since nearest neighbor infilling
is hyper-parameter free and step-unrolled denoising diffusion could be more generally applicable to other tasks with sparse data.
Table 17: Comparison of step-unrolling and self-conditioning [8]
NYU	KITTI
No infill	Infill	No infill	Infill
REL	RMS	REL	RMS	REL	RMS	REL	RMS
Baseline	0.079	0.331	0.077	0.338	0.222	3.770	0.057	2.744
Self conditioning	0.082	0.335	0.081	0.333	0.242	3.940	0.057	2.761
Step unrolling	0.076	0.324	0.075	0.324	0.085	2.844	0.056	2.700
We also considered the approach of self-conditioning [8] as an alternative to step-unrolling. However, as we show in Table 17, we find that self-conditioning is unable to bridge the train-inference distribution shift of the noisy latent for the task of monocular depth estimation. This is specially apparent in the results for KITTI without infilling where self conditioning leads to no improvement whereas step-unrolling substantially improves performance.
H.4 Hyper-parameters
Self-supervised. The self-supervised model is trained for 2.8M steps with an  $L_{2}$  loss and a mini-batch size of 512. Other hyper-parameters are same as those in the original Palette paper [54].
Supervised. The supervised flow and depth models are trained with  $L_{1}$  loss. Usually a constant learning rate of  $1 \times 10^{-4}$  with a warm-up over 10k steps is used. However, for depth fine-tuning we find that a lower learning rate of  $3 \times 10^{-5}$  achieves slightly better results. All models are trained with a mini-batch size of 64. The indoor depth model is pre-trained for 2M steps and then fine-tuned on NYU for 40k steps. The outdoor depth model is pre-trained for 0.9M steps and fine-tuned on KITTI for 40k steps. For flow, we pretrain for 3.7M steps, followed by finetuning for 50k steps. Other details, like the optimizer and the use of EMA are the same as [54].
H.5 Inference
Sampler. We use the DDPM ancestral sampler [21] with 128 denoising steps for monocular depth models and 64 steps for optical flow models. Increasing the number of denoising steps further did not greatly improve performance.
Coarse-to-fine refinement. We use  $2 \times 5$  overlapping patches (\{top, bottom\}  $\times$  {left, center-left, center, center-right, right}) for coarse-to-fine refinement. For Sintel we use  $t' = 32/64$  and for KITTI  $t' = 8/64$ .
Table 18: Inference speed comparison of our method with DPT [50] on the indoor depth model finetuned on NYU. Diffusion model inference is bottlenecked by the large number of denoising steps. We show that some efficiency gains can be achieved by simply reducing the number of denoising steps. Our model with 24 denoising steps is comparable in performance to DPT while being  $\sim 5\mathrm{x}$  slower (modulo differences in hardware). * we use the step-time reported in the DPT paper at a resolution of  $384\times 384$ , however, the DPT performance metrics on NYU are with a model trained at a resolution of  $480\times 640$ , for which the step time will be higher.
Method	Architecture	Resolution	Total Time [ms]	Inference steps	REL ↓	RMS ↓
DPT-Hybrid	Nvidia RTX 2080	384 × 384*	38*	-	0.110	0.357
DDVM	TPU v4	240 × 320	204	24	0.104	0.378
272	32	0.086	0.342
544	64	0.077	0.324
1089	128	0.075	0.324
Efficiency. Inference speed with diffusion models is a well-known issue, as multiple denoising steps are used to transform noise to a target signal. This can be prohibitive for vision tasks where near
real-time latency is often desired. Table 18 compares the inference speed of our diffusion model for depth against DPT [50]. Despite having an efficient denoiser backbone ( $\sim 8.5$  ms per denoising step on a TPU v4), the diffusion model is considerably slower than DPT in total wall time. The most obvious way to reduce inference latency is to reduce the number of denoising steps. This can be done with only moderate reduction in performance. As shown in Table 18, we perform comparably with DPT with as few as 24 denoising steps. However, a more thorough study into optimizing the inference speed of these models while preserving the generation quality is warranted. With the use of progressive distillation [42, 56] it is likely possible to reduce latency even further, as this approach has been shown to successfully distill generative image models with over 1000 denoising steps into those with just 2-4 steps.
Fine-tuning on Sintel. In Section 5 we discuss possible reasons for why our model's superior zero-shot performance compared to FlowFormer [23] does not transfer to fine-tuning on Sintel. Figure 16 provides qualitative examples to further support the claims.
Figure 16: Visual results on Sintel test. We compare with Flowformer [23] and provide flow visualization and an error map on each scene. On Ambush 1, FlowFormer can better predict the motion of the girl's right arm that moves out of the image boundary, likely due to the global reasoning capability of attention. On Cave 3 and Market 1, our method provides much finer details on motion boundaries with lower end-point error (EPE).
Input image
Figure 17: Qualitative examples of multimodal estimation on the NYU depth dataset showing examples where our model's uncertainty gets captured in the multimodal posterior. In the examples above, the model confuses the play mat (farther away from the viewpoint) for a table (closer to the viewpoint).
Variance heat map
Multimodal prediction samples
Uncertainty in depth estimation. We observe certain cases where the model is uncertain about the depth estimates. Interestingly, this uncertainty appears to be well captured in the predictive posterior, as illustrated in Figure 17.