The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
Authors: Saurabh Saxena
Charles Herrmann
Junhwa Hur
Abhishek Kar
Mohammad Norouzi
Deqing Sun
David J. Fleet
Venue: NeurIPS 2023 oral (2023)
Keywords: Monocular depth
optical flow
diffusion
depth
flow
TL;DR: Advances in denoising diffusion to handle limited, noisy, incomplete labels of dense vision tasks, specifically monocular depth estimation and optical flow, achieving sota results

Abstract: Denoising diffusion probabilistic models have transformed image generation with their impressive fidelity and diversity.
We show that they also excel in estimating optical flow and monocular depth, surprisingly without task-specific architectures and loss functions that are predominant for these tasks. 
Compared to the point estimates of conventional regression-based methods, diffusion models also enable Monte Carlo inference, e.g., capturing uncertainty and ambiguity in flow and depth.
With self-supervised pre-training, the combined use of synthetic and real data for supervised training, and technical innovations (infilling and step-unrolled denoising diffusion training) to handle noisy-incomplete training data, one can train state-of-the-art diffusion models for depth and optical flow estimation, with additional zero-shot coarse-to-fine refinement for high resolution estimates. 
Extensive experiments focus on quantitative performance against benchmarks, ablations, and the model's ability to capture uncertainty and multimodality, and impute missing values. Our model obtains a state-of-the-art relative depth error of 0.074 on the indoor NYU benchmark and an Fl-all score of 3.26\% on the KITTI  optical flow benchmark, about 25\% better than the best published method.

The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar,
Mohammad Norouzi, Deqing Sun, David J. Fleet"
irrnhhrrrinnrrrmmmn, unhwahur, junhhrr abhiskar,
Google DeepMind and Google Research
Abstract
Denoising diffusion probabilistic models have transformed image generation with their impressive fidelity and diversity. We show that they also excel in estimating optical flow and monocular depth, surprisingly, without task-specific architectures and loss functions that are predominant for these tasks. Compared the point estimates of conventional regression-basee methods, diffusion models also enable Monte Carlo inference, e.g., capturing uncertainty and ambiguity in flow and depth. With self-supervised pre-training, the combined use of synthetic and real data for supervised training, and technical innovations (infilling and step-unrolled denoising diffusion training) to handle noisy-incomplete training data, and a simple form of coarse-to- refinement, one can train state-of-the-an diffusion models for depth and optical flow estimation. Extensive experiments focus on quantitative performance against benchmarks, ablations, and the model's ability capture uncertainty and multimodality, and impute missing values. Our model, DDVM (Denoising Diffusion Vision Model), obtains a state-of-the-arr relative depth error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26% the KITTI optical flow benchmark, about 25% better than the best published method. For overview see diffusion- vision. iioio
Introduction
Diffusion models have emerged as powerful generative models for high fidelity image synthesis, capturing rich knowledge about the visual world 21 48, 55] (22.. However, at first glance, it is unclear whether these models be effective many classical computer vision tasks. For example, consider two dense vision estimation tasks, namely, optical flow, which estimates frame-to- frame correspondences, and monocular depth perception, which makes depth predictions based single image. Both tasks are usually treated as regression problems and addressed with specialized architectures and task-specific loss functions, e.g., cost volumes, feature warps, or suitable losses for depth. Without these specialized components or the regression framework, general generative techniques may be ill-equipped and vulnerable both generalization and performance issues.
In this paper, we show that these concerns, while valid, can be addressed and that, surprisingly, a generic, conventional diffusion model for image to image translation works impressively well both tasks, often outperforming the state of the art. In addition, diffusion models provide valuable benefits over networks trained with regression; particular, diffusion allows for approximate inference with multi-modal distributions, capturing uncertainty and ambiguity (e.g. see Figure
One key barrier to training useful diffusion models for monocular depth and optical flow inference concerns the amount and quality of available training data. Given the limited availability of labelled
'DF is also affiliated with the University Toronto and the Vector Institute.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input image(s)
Variance heat map
Multimodal prediction samples
Sinee KITTI DXM
Figure Examples multi-modal prediction depth (NYU) optical flow (Sintel and KITTI). Each row shows an input image (or overlayed image pair for optical flow), a variance heat map from 8 samples, and 3 individual samples. Our model captures multi-modalit in uncertain/aamiiguuus cases, such as reflective (e.g. mirror NYU), transparent (e.g. vehicle window KITTI), and translucent (e.g. fog on Sintel) regions. High variance also
object boundarics, which often challenging cases for optical flow, and also partially originate from noisy ground truth measurements for depth. See Fig.
10 and for more examples.
training data, we propose training pipeline comprising multi-task supe-supe pre-training followed by supervised pre-training using combination of real and synthetic data. Multi-task self-supervised pre-training leverages the strong performance of diffusion models on tasks like colorization and inpainting (e.g., [54] We also find that supervised (pre- teee-ttrainnng with a combination of real and large-scale synthetic data improves performance significantly.
further issue concerns the fact that many existing real datasets for depth and optical flow have noisy and incomplete ground truth annotations. This presents challenge for the conventional training framework and iterative sampling in diffusion models, leading to problematic distribution shift between training and inference. To mitigate these issues we propose the use of an L1 loss for robustness, infilling missing depth values during training, and the introduction of step-unrolled denoising diffusion. These elements of the model are shown through ablations to be important for both depth and flow estimation.
Our contributions as follows:
We formulate optical flow and monocular depth estimation as image to image translation with generative diffusion models, without specialized loss functions and model architectures.
2. We identify and propose solutions to several important issues w.r.t data. For both tasks, to mitigate distribution shift between training and inference with noisy, incomplete data, we propose infilling, step-unrolling, and an L1 loss during training. For flow, to improve generalization, we introduce new dataset mixture for pre-training, yielding RAFT 74 baseline that outperforms all published methods in zero-shot performance the Sintel and KITTI training benchmarks.
3. Our diffusion models is competitive with surpasses SOTA for both tasks. For monocular depth estimation we achieve a SOTA relative error of 0.074 on the NYU dataset and perform competitively KITTI. For flow, diffusion surpasses the stronger RAFT baseline by large
margin in pre-training and our fine-tuned model achieves an Fl-all outlier rate of 3.26% on the public KITTI test benchmark, ~55% lower than the best published method [70].
4. Our diffusion model is also shown to capture flow and depth uncertainty, and the iterative denoising process enables zero-shot, coarse-to-fine refinement, and imputation.
2 Related work
Optical flow and depth estimation have been extensively studied. Here we briefly review only the most relevant work, and refer the interested readers the references cited therein.
Optical flow. The predominant approach to optical flow regression-based, with a focus on
specialized network architectures to exploit domain knowledge, e,g., cost volume construction [14]
22
85 coarse-to-fine estimation [68] 77 82), occlusion handling
or iterative refinement
25 26 74], evidenced by public benchmark datasets
Some

---

with
Approximetely interpooroooao
Stop unroling
Forward diffusion process
(4(0,
Stop grad
Masked demoising
Concatenate noisy more conditioning images
U-Net
Training architecture. Given ground truth flow/depth, we first infill missing values using interpolation. Then, add noise the label map and train neural network to model the conditional distribution of the noise given RGB image(s), noisy label, and time step. One can optionally unroll the denoising step(s) during training (with stop gradient) to bridge distribution gap between training and inference for ye-
Figure 2:
recent work has also advocated for generic architectures: Perceiver IO (28 introduces generic transformer-based model that works for any modality, including optical flow and language modeling. Regression-based methods, however, only give single prediction of the optical flow and do not readily capture uncertainty or ambiguity in the flow. Our work introduces a surprisingly simple, generic architecture for optical flow using denoising diffusion model.
We find that this generic generative model is surprisingly effective for optical flow, recovering fine details on motion boundaries, while capturing multi-modality of the motion distribution.
Monocular depth. Monocular depth estimation has been long-standing problem in computer vision [58] [59 with recent progress focusing on specialized loss functions and architectures [ 15 31 such the use of multi-scale networks [12 [33, adaptive binning 3 35 and weighted scale- shift invariant losses [13. Large-scale in-domain pre- pre- training has also been effective for depth estimation
49 50 52, which we find to be the case here well. We build on this rich literature, but with simple, generic architecture, leveraging recent advances in generative models.
Diffusion models. Diffusion models are fatent-variable generative models trained to transform sample of Gaussian noise into a sample from a data distribution (2] 62). They comprise a forward process that gradually annihilates data by adding noise, 'time' / increases from t I, and learned generative process that reverses the forward process, starting from a sample random noise at = and incrementally adding structure (attenuating noise) decreases model conditions the steps of the reverse process (c.g., on labels, text, or an image).
A conditional diffusion
Central to the model is denoising network fo that is trained to take noisy sample y, at some time-step f, along with a conditioning signal x, and predict less noisy sample. Using Gaussian noise in the forward process, (as slowly decreases) as a sum of non- lonelrnear regression objectives, with the L2 loss (here with the --parameterizationn:
express the training objective over the sequence of transitions
Es 2) e() fo(æ, V + VI
U
where E ~ (N(, t ~ u(00 1). and where 7t > is computed with a pre-determined noise schedule. For inference (i.e., sampling), one draws random noise sample yı, and then iteratively uses fo to estimate the noise, from which one can compute the next latent sample y.. for s < t.
Self-supervised pre-training. Prior work has shown that self-supervised tasks such as coloriza- tion 86) and masked prediction [S) serve as effective pre-training for downstream vision tasks. Our work also confirms the benefit of self-supervised pre-training [54] for diffusion-based image-to- image translation, by establishing new SOTA on optical flow and monocular depth estimation while also representing multi muuti-modality and supporting zero-shot coarse-to- to-ine refinement and imputation.
3

---

First frame
AF+FT
AF+FT+KU
AF+FT+KU+TA
GroundTruth
First frame
AF
AF+FT
AF+FT+KU
AF-FT-KU+TA
GroundTruth
Figure 3: Effects of adding synthetic datasets in pretraining. Diffusion models trained only with AutoFlow (AF) tend provide very coarse flow estimates and can hallucinate shapes. The addition of FlyingThings (FT), Kubric (KU), TartanAir (TA) remove the An-induced bias toward polgonal-shan regions, and significantly improve flow quality on fine detail, e.g. trees, thin structures, and motion boundarics.
3 Model Framework
In contrast to the conventional monocular depth and optical flow methods, with rich usage of spe- cialized domain knowledge on their architecture designs, we introduce simple, generic architectures and loss functions. We replace the inductive bias in state-of-the-an architectures and losses with powerful generative model along with a combination of self-supervised pre-training and supervised training on both real and synthetic data.
The denoising diffusion model (Figure) takes noisy version of the target map (i.e., a depth or flow) as input, along with the conditioning signal (one RGB image for depth and two RGB images for flow). The denoiser effectively provides noise-free estimate of the target map (i.e., ignoring the specific loss parameterization used). The training loss penalizes residual error in the denoised map, which is quite distinct from typical image reconstruction losses used optical flow estimation.
3.1 Synthetic pre-training data and generalization
Given that we train these models with generic denoising objective, without task-specific inductive biases the form of specialized architectures, the choice training data becomes critical. Below we discuss the datasets used and their contributions in detail. Because training data with annotated ground truth limited for many dense vision tasks, here we make extensive use of synthetic data in the hope that the geometric properties acquired from synthetic data during training will transfer to different domains, including natural images.
AutoFlow (69 has recently emerged as a powerful synthetic dataset for training flow models. We were surprised find that training AutoFlow alone is insufficient, the diffusion model appears to devote a significant fraction of its representation capacity to represent the shapes of AutoFlow regions, rather than solving for correspondence. As result, models trained on AutoFlow alone exhibit strong bias to generate flow fields with polygonal shaped regions, much like those in AutoFlow, often ignoring the shapes of boundaries in the two-frame RGB inputs (e.g. see Figure B)
To mitigate bias induced by AutoFlow in training. we further mix in three synthetic datasets during training, namely, "hyingTTinng311 40), Kubric [19] and TartanAir (76). Given model pre-trained on AutoFlow, for compute efficiency, we use a greedy mixing strategy where we fix the relative ratio of the previous mixture and tune the proportion of the newly added dataset. We leave further exploration of an optimal mixing strategy to future work. Zero-shot testing of the model on Sintel and KITTI (see Table I and Fig. 3 shows substantial performance gains with each additional synthetic dataset.
We find that pre-training similarly important for depth estimation (see Table 7). We learn separate indoor and outdoor models. For the indoor model pre-train mix of ScanNet [9] and SceneNet RGB-D 41]. The outdoor model is pre-trained on the Waymo Open Dataset [7].
3.2 Real data: Challenges with noisy, incomplete ground truth
Ground truth annotations for real-world depth or flow data are often sparse and noisy, due to highly reflective surfaces, light absorbing surfaces (65), dynamic objects [43). etc. While regression-bast based

---

Algorithm 1 Denoising diffusion train step with infilling and step unrolling
I: a + conditioning images, y + flow or depth map, mask +- binary mask of known values
2: 1 ~ 1), ((0,
3: y fill holes_wwwwwaae with interpolation(y)
4: Ve = V * -T-
5: if uurrll step then
6: Epred sttppsoopoooee gereggradddinnieee yr. (),
7: Vyred = (yt vI 71 Epreea) / vi
8: yt+ = V/++ * Upred + vI - 7t *
9:
(yr V/u v\/vT= /ITV
70
10: end if
11: Gpred = fa(2, .)
12: loss = seduce__mantee Epread [pred[[mask) [mask])
methods can simply compute the loss on pixels with valid ground truth, corruption of the training data is more challenging for diffusion models. Diffusion models perform inference through iterative refinement of the target map y conditioned on RGB image data æ. It starts with a sample of Gaussian noise y1. and terminates with a sample from the predictive distribution p(yo æ²). A refinement step from time to 8, with << l, proceeds by sampling from the parameterized distribution pa (y(y, yr, æ); i.e., each step operates the output from the previous step. During training, however, the denoising steps are decoupled (see Eqn. [T. where the denoising network operates on noisy version of the ground truth depth map instead of the output of the previous iteration (reminiscent of teaching forcing in RNN training [79). Thus there is a distribution shift between marginals over the noisy target maps during training and inference, because the ground truth maps have missing annotations and heavy- tailed sensor noise while the noisy maps obtained from the previous time step at inference time should not. This distribution shift has very negative impact on model performance. Nevertheless, with the following modifications during training we find that the problems can be mitigated effectively.
Infilling. One way reduce the distribution shift is impute the missing ground truth. We explored several ways to do this, including simple interpolation schemes, and inference using our model (trained with nearest neighbor interpolation). We find that nearest neighbor interpolation is sufficient to impute missing values in the ground truth maps in the depth and flow field training data.
Despite the imputation of missing ground truth depth and flow values, note that the training loss is only computed and backpropagated from pixels with known (not infilled) ground truth depth. We refer to this the masked denoising loss (see Figure 2
Step-unrolled denoising diffusion training. A second way to mitigate distribution shift in the Ye marginals in training and inference, is to construct y/ from model outputs rather than ground truth maps. One can do this by slightly modifying the training procedure (see Algorithm IT to run one forward pass of the model and build y/ by adding noise to the model's output rather than the training map. We do not propagate gradients for this forward pass. This process, called nep-unrolled denoising diffisson,, slows training only marginally (~15% on TPU v4). This step-unrolling is akin to the predictor-correcte sampler of 663 which uses an extra Langevin step to improve the target marginal distribution of yt. Interestingly, the problem of training / inference distribution shift resembles that of exposure bias [51 in autoregressive models, for which the mismatch is caused by teacher forcing during training 779. Several solutions have been proposed for this problem in the
literature 2 32 84). Step-unrolled denoising diffusion also closely resembles the approach in (57 for training denoising autoencoders text.
We only perform step-unrolled denoising diffusion during model fine-tuning. Early in training the denoising predictions are inaccurate, so the latent marginals over the noisy target maps will be closer to the desired true marginals than those produced by adding noise to denoiser network outputs. One might consider the use of a curriculum for gradually introducing step-unrolled denoising difffusion in the later stages of supervised pre- pre-training, but this introduces additional hyper-parameters so we simply invoke step-unrolled denoising diffusion during fine-tuning, and leave an exploration of curricula to future work.
L denoiser loss. While the La loss in Eqn. I is ideal for Gaussian noise and noise-free ground truth maps, in practice, real ground truth depth and flow fields are noisy and heavy tailed; egg., for distant objects, near object boundaries, and near pixels with missing annotations. We hypothesize
5

---

First frame
Ground Truth
RAFT
DDVM (Ours)
Figure 4: Visual results comparing RAFT with method after pretraining. Note that method does much better on fine details and ambiguous regions.
that the robustness afforded by the L1 loss may therefore be useful in training the neural denoising network. (See Tables [I and 12 in the supplementary material for an ablation of the loss function for monocular depth estimation.)
3.3 Coarse-to-fine refinement
Training high resolution diffusion models is often slow and memory intensive but increasing the image resolution of the model has been shown to improve performance on vision tasks [18], One simple solution, yielding high-resolution output without increasing the training cost, is perform inference in a coarse-to-fi manner, first estimating flow over the entire field of view at low resolution, and then refining the estimates in patch-wise manner. For refinement, first up-sample the low-resolution map to the target resolution using bicubic interpolation. Patches are cropped from the up-scaled map, denoted 2, along with the corresponding RGB inputs. Then run diffusion model inference starting at time with a noisy map Ye ~ N (yy: /Tv 2, (1 Ye J). For simplicity, t' a fixed hyper-parametee, set based patches. Following Perceiver IO (28), the patch estimates are merged using weighted masks with lower weight near the patch boundaries since predictions at boundaries more prone errors. (See
validation set. This process is carried out for multiple overlapping
Section H.5 for more details.)
4 Experiments
pretrained with Palette [54] style self-supervised pretraining, and slightly modified to have the appropriate input and output channels for each task. Since diffusion models expect inputs and generate outputs in the range [-],, we normalize depths using max depth of 10 meters and 80 meters respectively for the indoor and outdoor models. We normalize the flow using the height and width of the ground truth. Refer to Section H for more details on the architecture, augmentations and other
As our denoiser backbone, we adopt the Efficient UNet architecture 55
Optical flow.
We pre-train on the mixture described in Section 3.1 at resolution of 340x448 and report zero-shot results on the widely used Sintel 44 and KITTI
datasets. We further fine-tune this model on the standard mixture consisting of AutoFlow (69], FlyingThings [40], VIPER (53] HDIK (30), Sintel and KITTI resolution of 360 and report results on the test set from the public benchmark. We standard average end-point error (AEPE) metric that calculates L2

---

First frame
Ground Truth
RAFT
DDVM (Ours)
Figure 5: Visual results comparing RAFT with our method after finetuning. Ours does much better on fine details and ambiguous regions.
Table 1: Zero-shot optical flow estimation results on Sintel and KITTI. We provide proposed pre-training mixture and substantially improve the accuracy over the original. Our diffusion model outperforms even this much stronger' baseline and achieves state-of-the-art zero-shot results on Sintel. final and KITTI.
Table 2: Optical flow finetuning evaluation public benchmark datasets (AEPF), for Sintel and Fl-all for KITTI). Bold indicates the best underline the 2nd -best. uses ex- tra datasets (AutoFlow and VIPER) on top of defaults (Flying Things, HDIK, K, KITTI, and Sintel). uuses
RAFT baseline using
Sintel.
SinteLelean Sintel.tinal
KITTI
Model
Dataset
AEPE
AEPE
Fl-all
Method
SinteLeclea final KITTI
FlowFoommr
Chairs - Things
1.01
2.40
4.09
14.72%
SKFlow CRAFT
721
1.30
2.26
4.84%
RAFT
Chairs + Things
1.68
2.80
2.42
4.79%
Perceiver 10
AutoFlow
1.81
2.42
4,98
FlowForner
28
1.14
2.18
4.68%
RAFT
AutoFlow
1.74
2.41
4.18
13.41%
RAFT (ours)
AF FT-KFAAA
1.27
2.28
2.71
9.16%
RAFT-it 70
DDVM (ours) AF AFTERRUL
1.24
2.00
2.19
7.58%
DDVM (ours)
1.75
2,48
3.26%
distance between ground truth and prediction. On KITTI, we additionally use the outlier rate, Fl-all, which reports the outlier ratio in % among all pixels with valid ground truth, where an estimate is considered as an outlier if its error exceeds 3 pixels and 5% w.r.f. the ground truth.
Depth. We separately pre-train indoor and outdoor models the respective pre-traie pre-training datasets described in Section 3.1 The indoor depth model is then finetuned and evaluated on the NYU depth v2 dataset [6]] and the outdoor model on the KITTI depth dataset [17]. We follow the standard evaluation protocol used in prior work [35]. For both NYU depth v2 and KITTI, we report the absolute relative error (REL), root squared error (RMS) and accuracy metries (61 1.25).
4.1 Evaluation on benchmark datasets
Depth. Table 3 reports the results on NYU depth v2 and KITTI (see Section for for more detailed results and Section for qualitative comparison with DPT on NYU). We achieve tate-of-the-ws absolute relative error of 0.074 on NYU depth v2. On KITTI, our method performs competitively with prior work. We report results with averaging depth maps from most prior works use post processing that averages two samples, one from the input image, and the other based on its reflection about the vertical axis.
samples. Note that
Flow. Table reports the zero-shot results of our model on Sintel and KITTI Train datasets where ground truth are provided. The model is trained on our newly proposed pre-training mixtures (Aut- oFlow (AF), FlyingThings (FT), Kubric (KU), and TartanAir (TA)). We report results by averaging samples coarse resolution and then refining them to the full resolution as described in Section 3.3 For fair comparison, we re-train RAFT on this pre-training mixture; this new RAFT model signif- icantly outperforms the original RAFT model. And our diffusion model outperforms the stronger
7

---

Table 3: Performance comparison on the NYU-Depth-v2 and KITTI datasets. T indicates method uses unsupervised pretraining. findicates supervised pretraining and I indicates use auxilliary supervised depth data. Best/ / second best results are bolded / underlined respectively. 4: lower is better t: higher is better.
NYU-Depth-22
KITII
Method
Architecture
61 f
REL1
RMSL
61 1
RELI
RMSL
TepnsDepth
87
Res-50~ViT.B'
0.900
0,106
0.365
0.956
0.064
2.755
DPT 50
Vix-5U+ViI--B*
0.904
0.110
0.357
0.959
0.062
2.573
BTS 34
DenseNet. 1617
0.885
0.110
0.392
0.956
0.059
2.756
AdaBins
E-BS+Mini-Vir*
0,903
0.103
0.364
0.964
0,058
2.360
BinsForreer
35
Swin-Large
0.925
0.094
0.330
0.974
0.052
2,098
PixelFormer
Swin-Large!
0.929
0.090
0.322
0.976
0.081
2.081
MIM
SwinV2.1.
0.949
0,083
0.287
0.050
1.966
ArT-P
40
SwinV2-1.
0.953
0.076
0.279
samples=1
Efficient U-Net" T1
0.944
0.075
0,324
0.964
0.056
2,700
DDVM
samples=2
Efficient U-Net" T4
0.944
0.074
0.319
0.965
0.055
2,660
samplese4
Efficient U-Net T4
0.074
0.315
0.055
2.613
Table 4: Ablation infilling and step-unrolling. Without either one, performance deteriorates. Without both, optical flow models fail to train on KITTI.
Table 5: Coarse-to-fiee refinement im-
proves zero-shot optical flow estimation sults on Sintel and KITTI, along with the qualitative improvements shown in Figure 6
NYU val
KITTI val
KITTI val
REL.
RMS
REL.
RMS
AEPE
Flail
Coarse to fine
Sintel.clean Sintel.fial
KITTI
Baseline
0.079
0.331
3.770
refinement
ARPE
AFPE
AEPE
Flall
Step-unroll
0.076
0.324
0.085
2,844
1.84
6. 1.66
Infill
0.077
0.338
0.057
2.744
1.53
5.24%
Without With
1.42
2.12
2.35
8.65%
Step-unroll & infiil
0.075
0.324
0.056
2,700
1.47
4.74%
1.24
2,00
2.19
7.58%
RAFT baseline. It achieves the state-of-the-art zero-ssh results on both the challenging Sintel Final and KITTI datasets.
Figure 4 provides a qualitative comparison of pre-trained models. Our method demonstrates finer details on both object and motion boundaries. Especially on KITTI, our model recovers fine details remarkably well, e.g. on trees and its layered motion between tree and background.
We further finetune our model on the mixture of the following datasets, AutoFlow, FlyingThings. HDIK, KITTI, Sintel, and VIPER. Table reports the comparison to state-of- the---art optical flow methods on public benchmark datasets, Sintel and KITTI. On KITTI, our method outperforms all existing optical flow methods by substantial margin (even most scene flow methods that use stereo inputs), and sets the new state of the art. On the challenging Sintel final, our method is competitive with other state of the art models. Except for methods using warm-start strategies, our method is only behind FlowFormer which adopts strong domain knowledge on optical flow (e.g. cost volume, iterative refinement, or attention layers for larger context) unlike our generic model. Interestingly, we find that our model outperforms FlowFormer on 1 11/12 Sintel test sequences and our overall worse performance can be attributed to much higher AEPE on single (possibly out-of-dine citto---dstibution) test sequence. We discuss this in more detail in Section 5 On KITTI, our diffusion model outperforms FlowFormer by a large margin (30.34%).
4.2 Ablation study
Infilling and step-unrolling, We study the effect of infilling and step-unrolling in Table 4 For depth, we report results for fine-tuning our pre-trained model on the NYU and KITTI datasets with the same resolution and augmentations as our best results. For flow, we fine-tune on the KITTI train set alone (with nearest neighbor resizing the target resolution being the only augmentation) at a resolution of 448 and report metrics on the KITTI val set 39 We report results with single sample and no coarse-to-f to-fine refinement. We find that training on raw sparse data without infilling and step unrolling leads to poor results, especially on KITTI where the ground truth is quite sparse. Step-unrolling helps to stabilize training without requiring any extra data pre-processing. However, we find that most gains come from interpolating missing values in the sparse labels. Infilling and step-unrolling compose well as our best results use both; infilling (heing an approximation) does not completely bridge the training-inference
inference distribution shift of the noisy latent.
8

---

Table 6: The addition of optical flow synthetic datasets sub- stantially improves the zero-shot results Sintel and KITTI.
Table 7: The addition of synthetic depth data in pre-training substantially im- proves fine-tuning performance on NYU.
Dataset
Sintel.clean
Sintel.final
KITTI
KITTI Fl-all
Dataset
REL
RMS
AF peetraining AF++
2.04
2:55
4.47
16.59%
2.22
14.07%
SceneNet RGB-D
AE AF-AFFFT-KU
1.33
2.04
2,82
9.27%
ScanNet
0
0.346
2.00
7.58%
SceneNet ScanNet
First frame
Ground Truth
Base
Refined
Figure 6: Visual results with and without coarse-to-fine refinement. For our pretrained model, refinement helps correct wrong flow and details to correct flow.
substantially improves fine-grained details in estimated optical flow fields. It also improves the metrics for zero-shot optical flow estimation on both KITTI and Sintel, as shown in Table
Coarse-to-fie refinement. F shows that coarse-to-fine fine refinement (Section 3.3
Datasets. When using different mixtures of datasets for pretraining, we find that diffusion models sometimes capture region boundaries and shape at the expense of local textural variation (eg see Figure The model trained solely on AutoFlow tends to provide very coarse flow, and mimics the object shapes found in AutoFlow. The addition of FlyingThinns,, Kubric, and TartanAir removes this hallucination and significantly improves the fine details in the flow estimates (eg, shadows, trees, thin structure, and motion boundaries) together with a substantial boost in accuracy (ef. Table (6) Similarly, we find that mixing SceneNet RGB-D [41, synthetic dataset, along with ScanNet [9] provides a performance boost for fine-tuning results on NYU depth v2, shown in Table
4.3 Interesting properties of difflusion models
Multimodality. One strength of diffusion models is their ability to capture complex multimodal distributions. This can be effective in representing uncertainty, especially where there may exist natural ambiguities and thus multiple predictions, in cases of transparent, translucent, or reflective cases. Figure presents multiple samples on the NYU, KITTI, and Sintel datasets, showing that our model captures multimodality and provides plausible samples when ambiguities exist. More details and examples are available in Section
Imputation of missing labels. A diffusion model trained to model the conditional distribution p(y]z) be zero-shot leveraged to sample from p(y) xr 3pprriia) where Uportial is the partially known label. One approach for doing this, known as the replacement method for conditional inference [63 is to replace the known portion the latent yt at each inference step with the noisy latent built by applying the forward process to the known label. We qualitatively study the results of leveraging replacement guidance for depth completion and find it to be surprisingly effective. We illustrate this by building pipeline for iteratively generating 3D scenes (conditioned on text prompt) as shown in Figure by leveraging existing models for eoxttto-image generation and text-conditional image inpainting. While more thorough evaluation of depth completion and novel view synthesis against existing methods is warranted, we leave that exploration future work. (See Section for more details and examples.)
5 Limitations
Latency. We adopt standard practices from image-generation models, leading to larger models and slower running times than RAFT. However, we are excited by the recent progress on progressive distillation 42 56) and consistency models (64) to improve inference speed in diffusion models.

---

Input Hoxt prompt
Abbrrrom
Inpaitt Image
Move camera
Deciproject point cloud
render
Image and Dopth
Myle
DDVM
Preceted Siph
Update
Precktedinghh
Figure 7: Application of zero-shot depth completion with model by incorporating it into iterative 3D generation pipeline. Starting with initial image (optionally generated from text-to-image model), sample an image-only conditioned depth map using our model. The image-depth pair is added to point cloud. We then iteratively render images and depth maps (with holes) from this point cloud by moving the camera. We then fill image holes using an existing image inpainter (optionally text conditioned), and then use our model with replacement guidance impute missing depths (conditioned on the filled RGB image and known depth).
Sintel fine-tuning. Under the zero-shot setting, method achieves state-of-the-an results both Sintel Final and KITTI Under the fine-tuning setting, ours is state-of-the-ane on KITTI but is behind on Sintel Final. We discuss several possible reasons for why this may be the case.
FlowFormer 23
We follow the fine-tuning procedure 70 While their zero-shot RAFT results are comparable to FlowFormer on Sintel and KITTI, the fine-tuned RAFT-it is significantly better KITTI but less accurate on Sintel than FlowFormer. It is possible that the fine-tuning procedure (e.g. dataset mixture augmentations) developed in [70] is more suited for KITTI than Sintel.
Another possible reason is that there is substantial domain gap between the training and test data on Sintel than KITTI. On Sintel test, there is particular sequence "Ambush 1", where the girl's right arm moves out of the image boundary. Our method has an AEPE close to 30 while FlowFormer has lower than 10. It is likely that the attention on the cost volume mechanism by FlowFormer can better reason about the motion globally and handles this particular sequence well. This particular sequence may account for the major difference in the overall results; among 12 available results on the Sintel website, ours has lower AEPE on sequences but higher AEPE on the "Ambush sequence, shown in Table 8
Figure 16 in the appendix further provides visualization.
Table 8: Average end-point error (AEPE) on 12 Sintel test sequences available from the public website.
Sequence
Ours
FlowFormer
Perturbed Market
0.787
0.869
Perturbed Shuman
0,219
0.252
Ambush
29.33
8.141
Ambush 3
2,855
2.973
Bamboo
0.415
0.577
Cave 3
2.042
2.352
Market
0.719
1.174
Market 4
5.517
8.768
Mounlain
0.176
0.518
Temple
0.452
0.612
Tiger
0.413
0.596
1.723
Conclusion
We introduced simple denoising diffusion model for monocular depth and optical flow estimation using an image-to-inge to-ie translation framework. Our generative approach obtains state-of of-te-of-the--a results without task-specific architectures loss functions. In particular, our model achieves an Fl-all score of 3.26% on KITTI, about 25% better than the best published method [70]. Further, our model captures multi-modality and uncertainty through multiple samples from posterior. also allows imputation of missing values, which enables iterative generation of 3D scenes conditioned on a text prompt. Our work suggests that diffusion models could be simple and generic framework for dense vision tasks, and we hope to more work in this direction.
10

---

Acknowledgements
We thank Ting Chen, Daniel Watson, Hugo Larochelle and the rest of Google DeepMind for feedback on this work. Thanks to Klaus Greff and Andrea Tagliasacchi for their help with the Kubric generator, and to Chitwan Saharia for help training the Palette model.
References
[1] Ashutosh Agarwal and Chetan Arora. Attention Attention Everywhere: Monocular depth prediction with skip attention. In WACV, 2023.
[2] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. NIPS, 2015.
[3] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. AdaBins: Depth estimation using adaptive bins. In CVPR, pages 1009-4418, 2021.
[4] Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael J. Black. A naturalistic open source movie for optical flow evaluation. ECCV, pages 61 1-655, 2012.
[5] Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen. Estimating depth from monocular images as classification using deep fully convolutional residual networks. IEEE T-CSVT, 28(1 1): 3174-3182, 2017.
[6] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An information-rict 3D model repository. arXiv:1512.03012, 2015.
[7] Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J. Fleet. A generalist framework for panoptic segmentation of images and videos. In ICCV, 2023.
[8] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In ICLR, 2023.
[9] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Niefnner ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, 2017,
[10] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li. and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.
[11] Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurlPS, 2022.
[12] David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In ICCV, pages 2650-2658, 2015.
[13] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. In NIPS, volume 27, 2014.
[14] Philipp Fischer, Alexey Dosovitskiy, Eddy llg. Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning optical flow with convolutional networks. In ICCV, pages 2758-2766, 2015.
[15] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, pages 2002-2011, 2018.
[16] Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and Ian Reid. Unsupervised CNN for single view depth estimation: Geometry to the rescue. In ECCV, pages 740-756, 2016.
[17] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets Robotics: The KITTI dataset. The International Journal of Roboties Research, 12111::221111277 2013.
[18] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth estimation. In ICCV, pages 3828-3838, 2019.

---

[19] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. In CVPR, pages 3749-3761, June 2022.
[20] Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, and Roberto Cipolla. Understanding real world indoor scenes with synthetic data. In CVPR, pages 4077-4085, 2016.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS, 2020.
[22] Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz. Fast cost-volume filtering for visual correspondence and beyond. IEEE T-PAMI, 35(2):004-511, 2012.
[23] Zhaoyang Huang. Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. FlowFormer: A transformer architecture for optical flow. In ECCV, pages 668 688.685,220 2022.
[24] Junhwa Hur and Stefan Roth. MirrorFlow: Exploiting symmetries in joint optical flow and occlusion estimation. In ICCV, pages 312-321, 2017.
[25] Junhwa Hur and Stefan Roth. Iterative residual refinement for joint optical flow and occlusion estimation. In CVPR, pages 5754 5763, 2019.
[26] Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. FlowNet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, pages 2462-2470, 2017.
[27] Serdar Ince and Janusz Konrad. Occlusion-aware optical flow estimation. IEEE T-IP, 17(8): 1443-1451, 2008.
[28] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver 10: general architecture for structured inputs & outputs. In ICLR, 2022.
[29] Jisoo Jeong, Jamie Lin, Fatih Porikli, and Nojun Kwak. Imposing consistency for optical flow estimation. In CVPR, 2022.
[30] Daniel Kondermann, Rahul Nair, Katrin Honauer, Karsten Krispin, Jonas Andrulis, Alexander Brock, Burkhard Gussefeld, Mohsen Rahimimoghaddam, Sabine Hofmann. Claus Brenner, et al. The HCI Benchmark Suite: Stereo and flow ground truth with uncertainties for urban autonomous driving. CVPR Workshops, pages 19-28, 2016.
[31] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. 3DV, pages 239-248, 2016.
[32] Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. Professor Forcing: A new algorithm for training recurrent networks. NIPS, 29, 2016.
[33] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In ECCV, pages 577-593, 2016.
[34] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and II Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv: 10326, 2019.
[35] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. BinsFormer: Revisiting adaptive bins for monocular depth estimation. 2204x 2022.
12

---

[36] Orly Liba, Longqi Cai, Yun- Ta Tsai, Elad Eban. Yair MovshovitzAAtias, Yael Pritch, Huizhong Chen, and Jonathan Barron. Sky Optimization: Semantically aware image processing of skies in low-light photography. In CVPR Workshops, June 2020.
[37] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite Nature: Perpetual view generation of natural scenes from a single image. In ICCV, 2021.
[38] Ao Luo, Fan Yang, Kunming Luo, Xin Li, Haoqiang Fan, and Shuaicheng Liu, Learning optical flow with adaptive graph reasoning. In AAAI, pages 1890-1898, 2022.
[39] Zhaoyang Lv, Kihwan Kim, Alejandro Troccoli, Deqing Sun, James M. Rehg, and Jan Kautz. Learning rigidity dynamic scenes with a moving camera for 3D motion field estimation. In ECCV, pages 468 (684484,220 2018.
[40] Nikolaus Mayer, Eddy 1lg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR, 2016.
[41] John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew Davison. SceneNet RGB-D: Can 5M synthetic images beat generic imagenet pre-training on indoor segmentation? In ICCV, 2017.
[42] Chenlin Meng, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In NeurlPS 2022 Workshop Score-Based Methods, 2022.
[43] Moritz Menze, Christian Heipke, and Andreas Geiger. Joint 3D estimation of vehicles and scene flow. In ISPRS Workshop on Image Sequence Analysis (ISA), 2015.
[44] Moritz Menze, Christian Heipke, and Andreas Geiger. Object scene flow. ISPRS Journal of Photogrammetry and Remote Sensing (JPRS), 2018.
[45] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pages 8162-8171, 2021.
[46] Jia Ning, Chen Li, Zheng Zhang, Zigang Geng, Qi Dai, Kun He, and Han Hu. All in Tokens: Unifying output space of visual tasks via soft token. In ICCV, 2023.
[47] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLm: Visual reasoning with general conditioning layer. In AAAI, 2018.
[48] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. crXiv:2204.06115, 2022.
[49] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE T-PAMI, (4(3))462233113333 2020.
[50] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction.
In ICCV, pages 12179-12188, 88, 2021.
[51] Marc Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.
[52] Zhongzheng Ren and Yong Jae Lec. Cross-Domain self-supervised multi-task feature learning using synthetic imagery. In CVPR, 2018.
[53] Stephan R. Richter. Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In ICCV, pages 2213-2222, 2017.
[54] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-Image Diffusion Models. In SIGGRAPH, 2022.
13

---

[55] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan. S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In NeurIPS, 2022.
[56] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
[57] Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. In ICLR, 2022.
[58] Ashutosh Saxena, Sung Chung, and Andrew Ng. Learning depth from single monocular images. NIPS, 2005.
[59] Ashutosh Saxena, Min Sun, and Andrew Y. Ng. Make3D: Learning 3D scene structure from single still image. IEEE T-PAMI, 31(5):824-840) 2009.
[60] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang. 3D photography using context-aware layered depth inpainting. In CVPR, 2020.
[61] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, pages 746-760, 2012.
[62] Jascha fohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. ICML, pages 2256 2265,2265,222 2015.
[63] Yang Song, Jascha Gohl-Dicksteins Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[64] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
[65] Martin Stommel, Michael Beetz, and Weiliang Xu. Inpainting of missing values in the kinect sensor's 's depth maps based on background estimates. IEEE Sensors Journal, 14(4): 107-1116. 2014.
[66] Xiuchao Sui, Shaohua Li. Xue Geng, Yan Wu, Xinxing Xu, Yong Liu, Rick Goh, and Hongyuan Zhu. CRAFT: Cross-attentional flow transformer for robust optical flow. In CVPR, pages 17602-17611, 2022.
[67] Deqing Sun, Erik B. Sudderth, and Michael J. Black. Layered segmentation and optical flow estimation over time. In CVPR, pages 1768-1775, 2012.
[68] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow 2018.
using pyramid, warping, and cost volume. In CVPR, pages
[69] Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T. Freeman, and Ce Liu. AutoFlow: Learning better training set for optical flow. In CVPR, pages 10093-10102, 2021.
[70] Deqing Sun, Charles Herrmann, Fitsum Reda, Michael Rubinstein, David J. Fleet, and William T. Freeman. Disentangling architecture and training for optical flow. In ECCV, pages 165-182, 2022.
[71] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao. Shuyang Cheng, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[72] Shangkun Sun, Yuanqi Chen, Yu Zhu, Guodong Guo, and Ge Li. SKFlow: Learning optical flow with super kernels. In NewrIPS, 2022.
14

---

[73] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. ICML, pages 6105-6114, 2019.
[74] Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In ECCV, pages 402-419, 2020.
[75] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David Fleet, Radu Soricut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan. Imagen Editor and EditBench: Advancing and evaluating text-guided image inpainting, In CVPR, 2023.
[76] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. TartanAir: A dataset to push the limits of visual SLAM. In IROS, 2020.
[77] Philippe Weinzaepfel, Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid. DeepFlow: Large displacement optical flow with deep matching. In ICCV, pages 1385-1392, 2013.
[78] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. SynSin: End-to-end view synthesis from single image. In CVPR, 2020.
[79] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Compulation, 1(2):270-200, 1989.
[80] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In CVPR, 2023.
[81] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. GMFlow: Learning optical flow via global matching. In CVPR, pages 8121-8130, 2022.
[82] Li Xu, Jiaya Jia, and Yasuyuki Matsushita. Motion detail preserving optical flow estimation. IEEE T-PAMI, 34(9):1744-113) 2011.
[83] Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. NeurlPS, 2019.
[84] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
[85] Feihu Zhang, Oliver J. Woodford, Victor Adrian Prisacariu, and Philip H.S. Torr. Separable flow: Learning motion cost volumes for optical flow estimation. ICCV, pages 10807-10817, 2021.
[86] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, pages 649-666, 2016.
[87] Jiawei Zhao, Ke Yan, Yifan Zhao, Xiaowei Guo, Feiyue Huang, and Jia Li. Transformer-based dual relation graph for multi-label image recognition. In ICCV, pages 163-172, 2021.
[88] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE T-PAMI, 2017.
15

---

A Multimodal prediction
We provide more qualitative examples for multimodal prediction. Figures g and Iillustrate multi- modal depth predictions NYU and KITTI respectively. Multimodality of the posterior distribution exists regions where there are multiple plausible predictions. For example, this includes reflective and transparent surfaces (mirrors and glass surfaces in rows to 5 of Figure 8 and windows of cars in Figure 2 We further find that the model captures uncertainty object boundaries, some of which arise due to noise in ground truth measurements in the training data. This can observed at the boundaries cars Figure and around edges of objects in Figure
anndgillustaall
depth estimates in the vicinity of
8 (most clearly visible in the last row).
Figure iillustrats different samples on KITTI from the optical flow diffusion model, also capturing multiple modes of the predictive posterior. Multimodality exists on transparent surfaces and near occlusions. As shown in Figure on Sintel, multimodality also exists on occluded out-of-bounds pixels where multiple predictions are plausible.
Input image
Variance map
Multimodal prediction samples
SUEP
UO
Figure 8: Qualitative examples of multimodal estimation the NYU depth dataset. Our model is able output multiple plausible depth maps where ambiguity exists. Rows to 5 show transparent reflective surfaces where two answers exist. In all samples (specially last row) we observe the model's ability capture uncertainty depth near object boundaries (see the areas of high variance).
16

---

Input image
Variance map
Multimodal prediction samples
Figure 9: Qualitative examples of multimodal depth estimation on KITTI. Our model is able to predict multimodal samples, especially windows of cars and object boundaries. Please refer the areas with high variance.
Overlayed inputs
Variance map
Multimodal prediction samples
Figure 10: Qualitative examples multimodal optical flow estimation on KITTI. Multimodality exists on
transparent surfaces (e.g., windows of cars) and shadows where our model estimates layered motion in different samples (see the last row).
17

---

Input images
Variance heat map & multimodal samples
Qualitative examples of multimodal optical flow estimation Sintel. Multimodality also exists on examples with challenging occlusion or out-of-bound cases.
Figure 11:
B Qualitative comparison of depth estimation with DPT
Figure 12 provides qualitative comparison of our model with DPT-Hybrid [50] finetuned on the NYU depth v2 [61] dataset. The depth estimates of our diffusion model are more accurate both on coarse-scale scene structure (walls, floors, etc.) and on individual objects.
C More samples for zero-shot imputation of depth
Figure 13 provides samples generated using our iterative text-to-3D pipeline. We note that such pipelines for iteratively generating 3D scenes have been previously proposed in literature [37]60] 78) However, these methods explicitly learn networks to refine the color [37 60 78 and the depth map In contrast, we propose leveraging the text-conditioned image prior from existing large scale text-to-image (55) and text-conditional image completion (75) models, and use our depth estimation model zero-shot for depth completion. One caveat with current approach of using the replacement method for conditional inference [63) for imputing depth, is that it does not enable one to fix errors in the depth predicted in the previous step. One approach to fix artifacts would be by noising-demoising, like that used for coarse-to-fine refinement. We leave further exploration into this to future work.
37 60
D Complete depth results on NYU and KITTI
TablesDaand provide detailed results on the val set of NYU depth v2 and KITTI depth datasets. We follow the standard evaluation protocol used in prior work [35). For both the NYU depth v2
18

---

Image
Ground Truth
DPT
Ours
Figure 12: Qualitative comparison of our model with DPT-Hybrid 60 (fine-tuned on NYU) on the NYU depth v2 val set. Our method infers better depth for both scene structure (walls, floors, etc.) and individual objects. Specific differences are highlighted with red arrows.
19

---

Prompt: A living room.
B
Prompt: A library.
Prompt: mecting room.
Prompt: kitchen.
Prompt: warehouse.
WHIZ
AAIIZ
Prompt: A movie theatre.
Figure 13: Text-to-3D samples. Given text prompt, an image is first generated using Imagen [5 (first row of first column), after which depth is estimated (second row of first column). Subsequently the camera is moved reveal new parts of the scene which infilled using conditions on both the incomplete depth map and filled image). each step, newly generated RGBD points are added to a global point cloud which is visualized in the rightmost column.
image completion model and model (which
20

---

Table 9: Comparison of performance the NYU-Depth-v2 dataset. T indicates method uses unsupervised pretraining, indicates supervised pretraining and indicates of auxiliary supervised depth data. Best second best results are bolded underlined respectively. i: lower is better and t: higher better.
Method
Architecture
51 1 82 1 6g
REL
RMS
logio 4
TephsDopth
Res-50+ViT-B
0.900
0,983
0.996
0.106
0.365
0.045
DPT 50
Res-50+ViT-B""
0.904
0,988
0.998
0.110
0.357
0.045
AdaBins
Mini--ii-ViT
0.903
0.984
0.997
0,103
0.364
0.044
BinsForme
35
Swin Large'
0.925
0,989
0.997
0.094
0.330
0.040
PixelFormer
Swin-Large
0.929
0.991
0.998
0.090
0.322
0.039
MIM
8D
SwinV2-L
0.949
0.994
0,999
0.083
0.287
0.035
SwinV2-L
0.953
0,999
0.279
samples=1
Efficient U. N.Nett
0.944
0,986
0.995
0.075
0.324
0.032
DDVM
samples=2
Efficient U-Net"
0.944
0,987
0.996
0.074
0.319
0.032
samplese)
Efficient U-Net"
0.946
0.987
0.996
0.074
0.315
0.032
Table 10: Comparison of performance the KITTI dataset. T indicates method uses unsupervised pretrain- ing. indicates supervised pretraining and indicates use auxiliary supervised depth data. Best second best results are bolded underlined respectively. 4: lower better and t: higher is better. E-B5: EfficientNet B5 773
Method
Backhone
61
62
6g
REI
Sq-rel 4
RMS
RMS log
DenseNet-161"
0.245
0.096
Dephh
RosNoct50-VVT-B'
0.064
0.252
2.755
0.098
DPT
50
SoeNeet-0--ii..
0.959
0.995
0.999
0.062
2.573
0.092
AdaBins
3
F-B5+mini-Vir VIT
0.964
0.995
0.999
0.058
0.190
2,360
0.088
BinsFormee
35
Swin Large
0.974
0.997
0.999
0.052
0.151
2.098
0.079
PixelFormer
Swin-Large
0.976
0.997
0.999
0.051
0.149
2.081
0.077
MIM
80
SwinV2-L
0.977
0.998
1.000
0,050
0.139
1,966
0.075
samples=1
Ellicient U-Net
0.964
0.994
0.998
0.056
0.339
2,700
0.091
DDVM
samples=2
Efficient Net
0.965
0.994
0.998
0.055
0.325
2,660
0.090
samples=4
Efficient U-Net
0.965
0.994
0.998
0.055
0.292
2.613
0.089
and KITTI datasets we report the absolute relative error (REL), root mean squared error (RMS) and accuracy metries (ō, 1.25' for i 1,2. For NYU we also report absolute error of log depths (log10). For KITTI we additionally report the squared relative error (Sq-rel) and root mean squared error of log depths (RMS log). The predicted depth is up-sampled to the full resolution using bilinear interpolation before evaluation. For the indoor model we evaluate on the cropped region proposed by and for the outdoor model the cropped region proposed by [6 as is standard in prior work.
13
E
Ablations
Tables TT and 12 show that L loss in training the diffusion model performs much better than Le loss for monocular depth estimation on NYU and KITTI. Tables 13 and [4 show the effectiveness of Palette-style (54) self-supervised pretraining for monocular depth estimation respectively. All results use single sample, Because these findings are reasonable and expected to generalize to other dense vision tasks, do not further ablate them for optical flow estimation for compute efficiency.
NYU and KITTI
Table 11: Ablation for the choice of loss function on the NYU depth v2 dataset.
52 1
83
REL
RMS..
logno 4
L
0.932
0.981
0.994
0.085
0.349
0.037
L
0.944
0.986
0.995
0.075
0.324
0.032
Table 12:
Ablation for the choice loss function on the KITTI dataset.
611
8zt
REL
Sq-rel
RMS
RMS log
La
0.954
0.993
0.998
0.065 0.056
0.321 0.339
2.773 2.700
0.099
L
0.964
0.994
0.998
0.091
21

---

Table 13: Ablation for self-supervised pretraining on the NYU depth v2 dataset.
81 t
82 f
83 f
REL.
RMS!.
log1o +
No self-suuervisee pre- pre-training With self-supervised pre-training
0.936
0.980
0.992
0.081
0.352
0.035
0.944
0.986
0.995
0.075
0.324
0.032
Table 14: Ablation for self-supervised pretraining on the KITTI depth dataset.
611
82
ost
REL 4 Sq-rel & RMS 4 RMS log +
No sulf-supe pre pre-training
0.952
0.990
0.997
0.064
0.389
2.998
0.104
With self-supervised pre-tre vre-tranning
0.965
0.994
0.998
0.055
0.332
2.696
0.091
F Coarse-to-fine refinement for depth
Figure 14 demonstrates performance of coarse-to- to-fine refinement on the NYU depth v2 dataset. While refinement improves fine-scale details in the estimated depth maps, the qualitative improvements are small and we do not find significant quantitative improvements. Hence the results reported in this work do not use coarse-to-fine refinement for depth estimation. Further work is needed to develop coarse-to-fine fonfine algorithm capable of more robust gains in depth estimation.
Image
Ground Truth
refinement
With refinement
Figure 14: Samples with coarse-to fine refinement on the NYU depth v2 dataset. We find that refinement adds sharpness and detail the depth estimation but does not provide quantitative improvements.
G Coarse-to-fine optical flow refinement for RAFT
For a fair comparison with optical flow estimation, we also apply our coarse-to-fine refinement scheme to RAFT 774, to determine whether our performance gains translate to RAFT well. We first estimate flow at low resolution, 320 x 448, upsample the low-resolution flow to the original resolution, divide original-resolution input images into 2 5 overlapping patches of size 320 x 448, then estimate flow on the cropped patches using the upsampled flow field as the initial guess for recurrent refinement (12 steps in total) of RAFT (74). After estimating flow of each patch, we merge them using weighted masks (28). Table reports the result. Unlike our diffusion based method, the coarse-to-fine scheme actually hurts the accuracy of RAFT Sintel Clean and KITTI and only marginally improves the accuracy on Sintel Final. Further exploration into better approaches for coarse-to- tonfine refinement for RAFT is warranted. We leave that to future work.
H Training and inference details
H.1 Architecture
UNet. The predominant architecture for diffusion models is the U-Net developed for the DDPM Here we adapt the Efficient U- Net architecture that was developed for Imagen [55). It is more efficient that the U-Nets used in prior work owing the use of fewer self-attention layers, fewer parameters and less computation at higher
model (21]. and later improved in several respects [1] 45 63
22

---

Table 15: Our coarse-to-fine refinement scheme marginally improves the performance of RAFT on Sintel Final while hurting performance on Sintel Clean and KTTTI. We report the EPE the Sintel and KITTI datasets.
Sintel Clean
Sintel Final
KITTI
RAFT baseline
1.27 1.35
2.28 2.26
2.71
RAFT with the coarse-to- to---tt-iiiooeet to-seet--fineeeeeee refinement
2.85
HxWxCH_N
HxWxCHLoun
Pravious dblock
Conv 3 2trides: 2
Coru kernet 3
inputcom
outgut conV
Depti to space
32
dblock1
ublodk1
FLM
Self attention [optiona]
N
H/2x W/2 123
H/2x 62
ResNet block
ResNet
ublock2
XN
x 256
Self attantion loptional)
FILM
dblock3
Jblock3
HI8 x W88 x 512
H/8 Wi8 x 256
Skip connection
H1T6
ub/ocku4
Previous ublock
Efficient UNat
DBlock
UBlock
Figure 15: Overview of the Effficinnt UNet architecture proposed in 55 CH and CH_OUT refer to the number of input and output channels respectively. refers to the time embedding. FilM refers the modulation layers proposed [47]. N the number ResNet self-attention blocks.
resolutions, along with other adjustments that make well suited to training medium resolution diffusion models.
Specifically we adopt the configuration for the x4 + 256 super-resolution model (see Figure [5 an overview) with several changes. We drop the text cross-attention layers but preserve the self- attention the lowest resolution layers dblock4 and ublock4 (see Figure 15). For supervised training for the flow model, we find it beneficial to additionally enable self-attention for the last-but-ong layers dblock3 and ublock3. The number of input and output channels differ across self-supervised pre- vre-training and supervised pre-training and are also different for flow and depth models. For self- supervised pre-training CH_ CH_IN=6 and CH_OCTH 3_TT=3=3 (see Figure 15p since the input consists of a 3-channel source RGB image and 3-channel noisy target image concatenated along the channel dimension and the output a RGB image. The supervised depth model has CH IN= (RGB image + noisy depth) and CH- OUT=1. The supervised optical flow model has CH = (2 RGB images + noisy flow along x and y) and CH_ OUT=2. Note that this means we need to reinitialize the input and output convolutional kernels and biases before the supervised pretraining stage. All other weights are re-used.
Resolution. Our self-supervised model was trained at resolution of 256 x 256. The indoor depth model is trained at 240x320. For Waymo we use 256x384 and for KITTI depth 256x832. Flow pretraining is done at resolution of 320x448, and finetuning at 368.
H.2 Datasets and augmentation
For unsupervised pre-training, we use the ImageNet-IK [10] and P6aces365 88 datasets and train on the self-supervised tasks of colorization, inpainting, uncropping, and JPEG decompression, following (5 Throughout, mix datasets at the batch level.
Flow. For supervised flow pretraining we use a mix of AutoFlow (native resolution 448x576), FlyingThings (540x960) Kubric (112x512) and TirtanAir (480x640) synthetic datasets. We finetune on the standard mixture consisting of AutoFlow, FlyingThing Things, Viper (540x960), HD1K (540x 1280), Sintel (436x 1024), and KITTI (375 x 1242).
23

---

(0) at resolution of We follow the same photometric and geometric augmentation schemes from 770. comprising random affine transformation, flipping, and cropping.
Depth. For supervised pre-training of the indoor model we mix the following datasets. ScanNer [9] is a dataset of 2.5M examples captured using Kinect v1-like sensor. It provides depth maps at x 640 and RGB images at 968 x 1296. SceneNet RGB-D 41 is a synthetic dataset of 5M images generated by rendering ShapeNet [6] objects in scenes from SceneNet x20
For the outdoor model training consisting of about 200k frames. Each frame provides RGB images from cameras and LiDAR maps. We use the RGB images from the FRONT, FRONT LEFT and FRONT RIGHT cameras and the TOP LiDAR only to build about 600k aligned RGB depth maps.
the Waymo Open Dataset 771 large-scale driving dataset
For indoor fine-tuning and evaluation we use NYU depth v2 [61], commonly used dataset for evaluating indoor depth prediction models. provides aligned image and depth maps at x80 resolution. We use the official split comprising 50k images for training and 654 for evaluation.
For outdoor fine-tuning and evaluation, use KITTI 17,, an outdoor driving dataset which provides RGB images and LiDAR scans at resolutions close to x 1226. We use the training/test split proposed by [33] comprising 26k training images and 652 test images.
We use random horizontal flip data augmentation which is common in prior work. Where needed, images and dense depth maps resized using bilinear interpolation the model's resolution for training and nearest neighbor interpolation is used for sparse maps.
H.3 Step-unrolling and interpolation of missing depth and flow
As discussed in Section 3.2 of the main paper, infilling and step-um rep-unrolling are used to mitigate distribution shift between training and inference with diffusion models. The problem arises due to the missing data the training depth maps and flow fields.
Infilling. For indoor depth maps, we use nearest neighbor interpolation during training (see Section 3.2 the main paper). For the outdoor depth data we use nearest neighbor interpolation except for sky regions, as they are often large and are much further from the camera than adjacent objects in the image. We use an aff-the-shellf sky segmenter (36), and then set all sky pixels be the maximum modeled depth (here, 80m). For missing optical flow ground truth we employ a simple sequence of ID nearest neighbor interpolations first along rows, and then along columns.
Table 16: Ablation of the number of unroll steps for monocular depth estimation. Performs improves up to four steps of unrolling and plateaus thereafter. The models trained without infilling missing depth benefit more from a larger number of unroll steps, which to be expected. Best and second best results are bolded and underlined respectively.
NYU
KITTI
No infill
Infil
No infill
Infill
Unroll steps
REL
RMS
REL
RMS
REL
RMS
REL
RMS
0
0.079
0.331
0.077
0.338
0.222
3.770
0.057
2.744
1
0.076
0.324
0.075
0.324
0.085
2.844
0.056
2.700
2
0.076
0.317
0.075
0.315
0.068
2.799
0.054
2.629
3
0.075
0.316
0.074
0.317
0.061
2.789
0.054
2.591
4
0.075
0.316
0.074
0.315
0.059
2.739
0.053
2.568
Step-unrolling. By default we use single unroll step in all results where step-unrolling is enabled. In Table 16] we show that using multiple unroll steps can further improve performance on the task of monocular depth estimation.
Finally, while we use infilling and step-unrolling, there are other ways in which one might try to mitigate the problem. One such approach was taken by (46), which faced a similar problem when training vector-quantizer on depth data. Their approach was to synthetically add more holes following carefully chosen masking ratio. We prefer our approach since nearest neighbor infilling
24

---

is parameerreeeer free and step step-lnrolled denoising diffusion could be more generally applicable to other tasks with sparse data.
Table 17: Comparison of step-unrolling and self-conditioning g
NYU
KITTI
No infill
Infill
No infill
Infill
REL
RMS
REL
RMS
REL
RMS
REL
RMS
Baseline
0,079
0.331
0.077
0.338
0.222
3.770
0.057
2.744
Self conditioning
0.082
0.335
0.081
0.333
0.242
3.940
0.057
2.761
Step unrolling
0.076
0.324
0.075
0.324
0.085
2.844
0.056
2.700
We also considered the approach of self-conditioning 8 as an alternative to step-unrolling. However, show in Table 17 find that self-conditioning is unable bridge the train-inference distribution shift of the noisy latent for the task monocular depth estimation. This is specially apparent in the results for KITTI without infilling where self conditioning leads to improvement whereas step-unrole unrolling substantially improves performance.
H.4 Hyper-parameters
Self-supervised. The self-supervised model is trained for 2.8M steps with L2 loss and mini-batch size of 512. Other hyper-parameters are same as those in the original Palette paper (54)
Supervised. The supervised flow and depth models are trained with L1 loss. Usually constant learning rate of 10 with a warm-up over 10k steps is used. However, for depth fine-tuning we find that a lower learning rate of 3 x 10-5 achieves slightly better results. All models are trained with mini-batch size of 64. The indoor depth model pre-trained for 2M steps and then fine-tuned on NYU for 40k steps. The outdoor depth model is pre-trained for 0.9M steps and fine-tuned on KITTI for 40k steps. For flow, we pretrain for 3.7M steps, followed by finetuning for 50k steps. Other details, like the optimizer and the use of EMA are the same as [54].
H.S Inference
Sampler. We use the DDPM ancestral sampler 21 with 128 denoising steps for monocular depth models and 64 steps for optical flow models. Increasing the number of denoising steps further did not greatly improve performance.
Coarse-to refinement, We use 2x5 overlapping patches (/top, bottom] x (left, center- center, center-right, right)) for coarse-to-fie refinement. For Sintel we use t' 32/64 64 and for KITTI ===a 8/64.
Limitations
Table 18: Inference speed comparison of our method with DPT [50 on the indoor depth model finetuned on NYU. Diffusion model inference is bottlenecked by the large number of denoising steps. We show that some efficiency gains be achieved by simply reducing the number denoising steps. Our model with 24 denoising steps is comparable performance to DPT while being ~-5x slower (modulo differences hardware). we use the step-time reported in the DPT paper at resolution of 384 384, however, the DPT performance metrics on NYU are with model trained at resolution of 480 x 640, for which the step time will be higher.
Method
Architecture
Resolution
Total Time [ms]
Inference steps
REL +
RMS 4
DPT-Hybrid
Nvidia RTX 2080
384 384*
38*
0.110
0.357
204
24
0.104
0.378
272
32
0.086
0.342
DDVM
TPU v4
240 x 320
544
64
0.077
0.324
1089
0.075
0.324
Efficiency. Inference speed with diffusion models is a well-known issue, as multiple denoising steps used to transform noise target signal. This can be prohibitive for vision tasks where near
25

---

real-time latency is often desired. Table 18 compares the inference speed of our diffusion model for depth against DPT [50]. Despite having an efficient denoiser backbone (~8.55 ms per denoising step on TPU v4), the diffusion model is considerably slower than DPT in total wall time. The most obvious way to reduce inference lateney is to reduce the number of denoising steps. This can be done with only moderate reduction in performance. As shown in Table T8] we perform comparably with DPT with as few as 24 denoising steps. However, more thorough study into optimizing the inference speed of these models while preserving the generation quality is warranted. With the use of progressive distillation 22 [6] it is likely possible reduce latency even further, as this approach has been shown to successfully distill generative image models with over 1000 denoising steps into those with just 2-4 steps.
Fine-tuning on Sintel. In Section 5 we discuss possible reasons for why our model's superior zero- shot performance compared to Former (23] does transfer to fine-tuning on Sintel. Figure T6 provides qualitative examples to further support the claims.
Temporally consecutive input frames
Ours
FlowFormer 23
Ambush
EPE = 29.330
8.148
Cave
2.042
EPE =
Market
EPE 0.719
EPE 1.174
Figure 16: Visual results on Sintel test. We compare with Flowformer 23 and provide flow visualization and error map each scene. On Ambush 1, FlowFormer can better predict the motion of the girl's right arm that
of the image boundary, likely due the global reasoning capability of attention. On Cave and Market our method provides much finer details on motion boundaries with lower end-point error (EPE).
26

---

Input image
Variance heat map
Multimodal prediction samples
Wt SOEE
Figure 17: Qualitative examples of multimodal estimation on the NYU depth dataset showing examples where our model's uncertainty gets captured in the multimodal posterior. In the examples above, the model confuses the play mat (farther away from the viewpoint) for table (closer the viewpoint).
Uncertainty in depth estimation. We observe certain cases where the model is uncertain about the depth estimates. Interestingly, this uncertainty appears to be well captured in the predictive posterior, illustrated in Figure I7
27


Reviews:
[{"review_id": "B0BgNecRE6", "reviewer": "Reviewer_8jCd", "summary": "The paper demonstrates that diffusion models are effective general-purpose solutions for dense optical flow and monocular depth regression tasks. The paper shows that the same architecture and loss functions lead to at-par or better performance on these tasks, compared to existing methods that use domain knowledge and problem-specific architectures. The main insights presented are the use of a pre-training phase for higher quality, and imputation of the missing values, and a step-unrolled diffusion step, for dealing with incomplete GT training data.", "questions": "- Please answer the questions raised on pre-training, and on self-guidance. \n- Is it possible to fine-tune an existing diffusion model, such as stable diffusion, instead of training one from scratch? Could it help avoid the use of synthetic data?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and motivates the contributions perfectly. The simplicity of the architecture and the loss functions are very clear. The results support the claims and show at-par or better results than the state of the art. Extensive experiments on different real datasets help evaluate the method's quality. Multi-modality of the outputs is promising! I would have loved to see some more analysis there.  The presented application is also exciting, showing directions for text to 3D reconstructions.", "weaknesses": "While there is limited technical novelty, this is a good paper that demonstrates a simple method for solving two different regression tasks.\n\n- I did not understand why the pre-training phase needs to be separate. The method uses supervised pre-training, that is different for each task. The loss functions are also identical between the pre-training and the fine-tuning states. Why not combine all available datasets and just train the model once (inc. all the tricks used for fine-tuning)? I did not understand this distinction between phases, especially when the experiments are explained and the datasets keep moving from one phase to the other. \n\n- Step-unrolled diffusion is a little similar to self-guidance, introduced in \"Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning\" [Chen et al.] I wonder if self-guidance is already good at addressing the domain shift problem, or whether step-unrolled diffusion is really needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for your thoughtful comments and questions. They will certainly help improve the revised paper. Please see our response below.\n\n> limited technical novelty\n\nOne of the main motivations for this paper is to understand how well vanilla diffusion models perform on classical dense computer vision tasks which are traditionally solved using specialized techniques. As a result, we intentionally kept the network design and diffusion formulation simple.\n\nWe do however, identify and solve several issues unique to the application of diffusion models for the tasks of optical flow and monocular depth estimation. For example, innovations in training are necessary for the application of diffusion models on dense regression problems with noisy and incomplete ground truth data. To that end, we introduce infilling and step-unrolling, which greatly  improves the performance.  For optical flow on the KITTI dataset, baseline diffusion tends to diverge and SOTA performance only results from the combination of both these additions. For KITTI depth, infilling and unrolling result in a substantial improvement in REL (0.222 to 0.056). In addition, we find that our optical flow network requires different training data than regression based optical flow techniques and solve this through a new training regime. Beyond these innovations, we also  introduce a coarse-to-fine refinement scheme which provides further performance gains and greater flexibility in the image resolutions to which the model can be applied. We also demonstrate the use of imputation for text-to-3D generation.\n\nBut the main novelty in the paper is the demonstration that it is possible to generate SOTA results on well studied regression problems for which previous methods have relied heavily on specialized techniques such as the use of cost volumes [1] for flow and binning [14] for depth. We propose a common architecture and training procedure across two different dense vision problems. We think this framework is encouraging and motivates new directions for vision research, with a common architecture for many vision problems. Further we show that the diffusion framework is sufficiently powerful to capture the multi-modal predictive distributions (e.g. capturing ambiguity in flow or depth estimation). \n\n> More analysis of multi-modality\n\nThe ability of diffusion models to represent complex multimodal distributions, without excessive mode collapse is arguably one of the key properties that has led to the recent success and excitement around diffusion models. Their ability to capture uncertainty in the predictive posterior distributions over depth and optical flow, including multi-modality in cases of ambiguity, is quite interesting. Figures 1-4 in the supplementary material show examples and variance heat maps of the multi-modal predictions for both depth and flow. For depth we observe multimodality in transparent and reflective surfaces, such as mirrors and windows of cars, and around object boundaries. For flow we observe multimodality in transparent surfaces and shadows, thereby capturing the layered nature of the scene. Notably we also observe multimodality in out-of-frame motion where the flow is ambiguous. We think this is an interesting finding and opens up further avenues of research and also provides a way to measure uncertainty which can be important for downstream applications.\n\n> I did not understand why the pre-training phase needs to be separate.\n\nModel pre-training on large-scale datasets is commonplace for optical flow [1, 2] and monocular  depth  [3, 4]. E.g. for optical flow the training data schedule (e.g. FlyingChairs -> FlyingThings3D -> mixture of Sintel / KITTI / VIPER etc.) has been heavily studied and shown to be crucial to achieve good performance (Table 5 of [12] and Table 1 of [13]).  Considerable effort has recently been spent designing better pre-training datasets [10, 11]. This combination of pre-training and fine-tuning provides the advantages of large-scale datasets, with pre-trained models often yielding good zero-shot performance, with the ability to fine-tune models to a specific dataset to maximize performance (perhaps with some loss of generality).\n\n> I wonder if self-guidance is already good at addressing the domain shift problem\n\nIn our experiments with self-guidance we found that it does not address the domain shift problem (see Table 1 in attachment). This is expected because, even with self-guidance, there remains a training/inference distribution shift for the latent $y_t$, which is what we address through step-unrolled diffusion training.\n\n> Is it possible to fine-tune an existing diffusion model, such as stable diffusion, instead of training one from scratch? Could it help avoid the use of synthetic data?\n\nIndeed, we use an existing self-supervised pre-trained diffusion model (i.e., the Palette model [5]) for image to image translation. As shown in Tables 5 and 6 of our supplementary material, this self-supervised pre-training substantially improves results. \n\nIt may also be possible to use a pre-trained text-conditional image generation model, especially in light of recent works such as [8, 9], which exploit features from pretrained text-to-image models for depth regressors. We briefly considered fine-tuning existing text-to-image models but found that available pixel-space models at the time, such as Imagen [7], were computationally expensive (2B params in the base model) which might limit their utility in practical vision applications and latent space models like Stable Diffusion additionally require dealing with holes in the autoencoder training (like in [6]). Hence, we left this exploration to future work.\n\nIt may be possible that the use of significantly more text-image training data or larger datasets for image to image translation would allow one to avoid training with synthetic data. We have not explored this but we agree that this is important future work.\n"}, {"review_id": "5BHUtnCtbE", "reviewer": "Reviewer_xrrD", "summary": "The authors study the use of diffusion models for the tasks of single-image depth estimation and optical flow estimation. Self-supervised pre-training, supervised fine-tuning with synthetic and real data, combined with a couple of tricks to leverage imperfect GT, lead to competitive results with nearly no task-specific modifications to the diffusion models. The authors also demonstrated unique capabilities enabled by the diffusion models, e.g., capturing multimodality and completion from partial data. ", "questions": "Please refer to the three points in the weakness section above. ", "limitations": "I agree with the limitation the authors brought up in the supplementary. Optical flow and depth estimation are low-level tasks commonly used at early stages of real-world application systems and therefore demand higher efficiency. As the authors already pointed out, this is where the proposed diffusion models fall short. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality and significance**: \n\nAs far as I know, this is the first work to study the use of generative diffusion models for optical flow and depth estimation tasks. Training good optical flow and depth models typically requires extensive task-specific knowledge in terms of architecture designs and loss functions, so the authors' finding that diffusion models can compete well on these tasks with almost no task-specific treatments is non-trivial and valuable. The competitive results in various settings further add to the significance of the work. \n\n**Technical quality**: \n\nTraining diffusion models successfully involves many technical details. The authors generally follow best practices and propose reasonable solutions to unique challenges. More specifically, the use of pre-trained PALETTE, further supervised pre-training with mixtures of synthetic and real datasets, addressing imperfect GT with infilling and step-unrolling, etc. are all well-motivated and proven effective. \n\n**Writing quality**: \n\nThe paper is nicely written, with precise language, adequate details, and clear explanations. Conclusions are justified with plenty of results, visualizations, ablation studies, and overall convincing. ", "weaknesses": "The authors claim that diffusion models can be a generic framework for vision dense prediction tasks, but only consider the tasks of depth estimation and optical flow in this work. Both these tasks are relatively low-level, and it'd be interesting to offer some insights, discussions, or analyses regarding how higher-level tasks, such as semantic segmentation, differ from them and if the claim still holds. \n\nTwo relatively minor complaints/suggestions: \n* Can authors provide some theoretical analysis or proof for the step-unrolling step? Could unrolling more steps bring further improvements? \n* It's unclear how much the use of pallete self-supervised pre-training help since it's not part of the ablation study. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for your thoughtful comments and questions. They will certainly help improve the revised paper. Please see our response below.\n\n> Do the claims hold on higher-level tasks such as semantic segmentation\n\nThe extent to which a generic diffusion model is effective on other vision tasks, including higher-level tasks is a topic of ongoing work. Diffusion models have indeed been shown to work well on panoptic segmentation [15], which is encouraging.\n\n> Theoretical analysis or proof for step-unrolling\n\nOne can view an unrolling step at time t as a Langevin update of a MCMC sampler for which the target distribution is the marginal distribution of the latent $y_t$ (i.e., the distribution of noisy optical flow fields or depth maps). From this perspective unrolling steps act like corrector steps in the predictor-corrector sampler of Song et al [16]. We will include further discussion in the paper.\n\n> Could unrolling more steps bring further improvements\n\nGreat question. For datasets in which the ground truth flow or depth maps have more missing data (e.g., like KITTI) one would expect more unrolling steps to be useful in matching the marginal distribution of the latent $y_t$. In our experiments we did find this to be the case (please see Table 2 in attachment).  For instance, increasing the number of unrolling steps from one to four improves KITTI REL from 0.056 to 0.053, and the RMS from 2.700 to 2.568. On NYU, improvements are marginal (REL improves from 0.075 to 0.074 and RMS from 0.324 to 0.315) as one might expect since the ground truth data have fewer missing depth values.\n\n> how much the use of pallete self-supervised pre-training help\n\nPlease see Tables 5 and 6 in our supplementary material.  They show that self-supervised pre-training clearly improves results for monocular depth estimation. This isn’t entirely surprising since tasks like inpainting and colorization entail some form of ‘semantic understanding’. We expect similar findings for optical flow estimation, but we did not perform this study on optical flow estimation since pre-training is compute intensive.\n"}, {"review_id": "ooOcLPQlLR", "reviewer": "Reviewer_F9qi", "summary": "This paper proposes to use diffusion models to solve monocular depth and optical flow estimation tasks. Unlike previous task-specific models for depth and flow, this paper uses a generic diffusion model. This paper studies the effect of training data (synthetic and real) and processing of sparse depth and flow ground truth when training the diffusions models. Experiments are conducted for depth and flow tasks on representative benchmarks, the proposed method achieves state-of-the-art depth performance on NYU and state-of-the-art optical flow performance on KITTI.", "questions": "- Due to the uncertainty in diffusion models, a same model might produce different results when running twice. How large is the fluctuation and how's the final quantitative results reported? Are the authors using some averaging?\n- How sensitive of the proposed architecture to different image resolutions for both depth and flow tasks? For example, what if the inference image resolution is different from training, will the model still perform reasonably?\n- The optical flow results on KITTI is very strong, but the results on Sintel seem less robust (as also analysed in the supplementary material). What might cause the different behaviours on Sintel and KITTI, could the authors further comment on this?\n- I think one key message from this paper is that the experiments show the importance of training data. When comparing Table 1 and Table 6 for the results of RAFT and the proposed method, we can observe that RAFT outperforms diffusion models when only using AutoFlow for pre-training. However, diffusion models perform better when more datasets are added to the pre-training stage. I am wondering whether this indicates that diffusion models can benefit more from larger datasets than previous regression methods?", "limitations": "Yes, the authors have carefully analyzed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- **The idea of using diffusion models to solve depth and optical flow tasks is interesting.** Depth and optical flow are typically approached as regression tasks, it's unclear how the popular diffusion models will perform for both tasks. This paper explores this direction and shows some interesting results.\n- **Several strategies are proposed to handle the issue of data for training diffusion models.** It's not straightforward to apply diffusion models to depth and flow tasks, the paper proposes data infilling, step-unrolling and an L1 loss to tackle the challenges.\n- **The experiments are extensive and informative.** Training data plays a significant role in training diffusion models, this paper studies the effect of different training datasets for both depth and flow tasks. The performance on KITTI for optical flow task is especially strong, outperforming previous 2-frame optical flow methods by a large margin.\n- **A detailed discussion of limitations is presented in the supplementary material.** This paper gives a deep analysis of the limited performance on Sintel test set and the results indicate that a particular sequence on the test set severely affects the averaged performance, which might provide some hints for further improvement in future.", "weaknesses": "I didn't observe major weakness and would put some minor points to the Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for your thoughtful comments and questions. They will certainly help improve the revised paper. Please see our response below.\n\n> Due to the uncertainty in diffusion models, a same model might produce different results when running twice. How large is the fluctuation and how's the final quantitative results reported? Are the authors using some averaging?\n\nFigure 1, 2, 3, and 4 in the supplemental show multiple samples from the predictive posterior, along with variance heat maps. As visualized in the variance heat maps, the magnitude of local fluctuation depends on the nature of the multi-modality; variability is common in regions of ambiguity (e.g. transparent/reflective surfaces, object boundaries, or occlusions). \nWe do average multiple samples for both depth and flow (see Section 4.1). Table 3 shows small but consistent improvements in depth estimation as we average more samples. For optical flow, we average 8 samples for the coarse-grained estimate, but we do not use sample averaging in the high resolution refinement. We will clarify this in the final version of the paper.\n\n> How sensitive of the proposed architecture to different image resolutions for both depth and flow tasks? For example, what if the inference image resolution is different from training, will the model still perform reasonably?\n\nLike prior regression methods (Figure 4 of [17]) our model's performance degrades if one naively runs inference at a resolution that is different from the training resolution. However, in the paper we explain how to use the diffusion model within a coarse-to-fine refinement scheme (see Section 3.3). This way we are able to effectively run inference on high resolution images, first at a coarse-resolution, and then patch-wise at high resolution, conditioned on the coarse-grained estimate to provide global context. Table 5 shows the improved performance on optical flow estimation with this approach.\n\n> The optical flow results on KITTI is very strong, but the results on Sintel seem less robust (as also analysed in the supplementary material). What might cause the different behaviours on Sintel and KITTI, could the authors further comment on this?\n\nIn Table 9 of the supplementary material, we show that we outperform FlowFormer on all Sintel test sequences but one, ambush_1. This sequence has significant ambiguity due to an all-white background, large camera and object motions, and the existence of numerous objects that occlude one another and move out of frame. This forces the model to depend heavily on inductive biases and learned motion priors, which is challenging for our method. It is possible that FlowFormer’s inductive bias (with the cost volume etc) make it better suited to reason about this sequence’s large out-of-frame motions.\n\n> RAFT outperforms diffusion models when only using AutoFlow\n\nAs discussed in Section 3.1 (paragraph 2) and shown in Figure 3, we find that when trained solely on AutoFlow the diffusion model learns to reproduce shapes from the AutoFlow data, yielding poorer qualitative and quantitative performance. The denoiser’s bias toward \npolygonal shapes in AutoFlow could partially be explained by recent work (eg section 4.2 of [18]) on shape vs texture bias in neural classifiers.  As a result, using AutoFlow alone causes the model to hallucinate on real data and try to identify the best AutoFlow shapes to represent the real objects. Interestingly, this problem is solved by training with larger, more diverse data.\n\n> I am wondering whether this indicates that diffusion models can benefit more from larger datasets than previous regression methods?\n\nThis is a great question! The finding that diffusion models benefit more from larger datasets is indeed surprising. Like most regression based flow networks, RAFT has several architectural elements which bias the network towards modeling flow; eg, RAFT features an all-pairs cost volume, which compares all the pixels (in encoding space) in frame 1 to frame 2, and then accesses this cost volume through a lookup operation based on the flow. This network element strongly encourages the network to use pixel comparisons to generate the predicted flow. As a result, despite being trained on only AutoFlow (a synthetic dataset), RAFT can quickly learn to use pixel or patch similarity to compute optical flow, an ability which generalizes quickly to real world videos. In contrast, our diffusion pipeline lacks any of these specific model biases and must learn these biases through data. However, it is important to note that RAFT's ability to learn quickly comes with a tradeoff; namely, the inductive biases that are hard coded into its architecture may be suboptimal. Given enough data, learning these biases may be better than designing them through manual architecture design.\n"}, {"review_id": "Msyw0CRMwB", "reviewer": "Reviewer_LqWs", "summary": "This paper approaches the task of predicting optical flow from a pair of images and depth from a single image. It proposes to do so using diffusion models, providing a training pipeline which includes contributions to deal with noisy training data. The proposed method is competitive with SOTA in depth prediction on NYU and KITTI, is similar or better to SOTA in optical flow prediction on Sintel and KITTI zero-shot, and is SOTA in optical flow after finetuning on KITTI. Additional experiments show the proposed pretraining pipeline meaningfully improves RAFT and the proposed model, that contributions dealing with noisy training data are helpful, and that the proposed method can predict multimodal outputs in cases of uncertainty.\n\nEdit: Thanks to the authors for the rebuttal.\n\nAfter reading the rebuttal and other reviews I will keep my rating at 7 - accept. I believe this paper should be accepted because it (1) introduces a simple but interesting method that performs SOTA on competitive benchmarks and (2) is very polished and experiments clearly defend all main contributions of the method.", "questions": "- L179 is a bit confusing and could be rewritten “Training high resolution diffusion models is often slow and memory intensive but model performance has been shown to improve with resolution” \n- Figure 4 (top) is dark and indistinct, making it a hard to see. Are there failure cases for RAFT on more easily readable examples?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper provides a simple method that is similar-to or better than SOTA on competitive optical flow and depth tasks\n- A new data pipeline is introduced which significantly boosts performance of prior SOTA e.g. RAFT, while improving further with the proposed method\n- The method is built off a standard image-to-image diffusion method Palette, adds minimal detail needed to handle flow and depth data (e.g. L1 loss, infilling + unrolling, coarse-to-fine), and is competitive across a wide variety of experiments\n- Real data is tricky to handle for diffusion models as it can contain incomplete flow and depth maps. The proposed method has an interesting idea of using its own predictions late in training. Combined with infilling, this is helpful.\n\nThe paper is very polished and experiments clearly defend all main contributions of the method.\n- Writing is very clear, figures and tables are attractive and helpful\n- Choice of training data is very important (Figure 3, Table 6, 7); it even improves RAFT meaningfully (Table 1)\n- The method can also produce Multimodal samples faced with ambiguity e.g. transparent/translucent/reflective (Figure 1, 4, 5)\n- Figure 6 and Table 5 shows the importance of coarse-to-fine refinement, which enables the method to produce better detail than RAFT (Figure 4, 5)\n- Infilling + step-unrolling yields massive improvement on KITTI (Table 4)\n- L1 loss (Supp Table 4), pretraining for depth (Supp Table 5, 6)\n- Zero-shot depth completion is a cool application of the method", "weaknesses": "Finetuning performance on real data yields weaker performance on Sintel vs. other methods (minor weakness, but addressing could further strengthen paper!)\n- On Sintel (Table 2), most other methods use warm start, i.e. initializing flow prediction from previous frame. Is it possible to do this with diffusion, i.e. starting from a previous prediction, perhaps denoising fewer steps? If not, this is an important weakness of diffusion models in this setting\n- Given most other method warm-start, it is harder to analyze the ability of the proposed method to finetune on real data. Perhaps it does not do as well relative to FlowFormer because (1) it does not handle real data as well as regression methods, even after contributions in this area, or (2) it does not overfit as well to specific datasets given others tend to have task-specific architecture. A comparison to more methods without warm-start, or by using the proposed method with warm-start, it would be very interesting to see this analyzed.\n\nThe main novel contribution of this paper beyond data pipeline is step-unrolled denoising diffusion training, which yields modest improvement over infilling alone (minor weakness)\n- L1 loss is a design decision, infilling holes using bilinear interpolation is not a substantial contribution in my eyes. So the remaining technical contribution is unrolling.\n- I understand the contributions are deliberately simple, which is a positive given performance gain. However, it is still important to analyze the reasons for the method’s success. In this case, the most novel component, “unroll”, improves optical depth REL from 0.077 to 0.075, RMS from 0.338 to 0.324, KITTI REL from 0.057 to 0.056, RMS from 2.744 to 2.700, AEPE from 1.53 to 1.47, and F1-all from 5.24% 4.74%. These contributions are nontrivial, but not substantial on their own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for your thoughtful comments and questions. They will certainly help improve the revised paper. Please see our response below.\n\n> Finetuning performance on real data yields weaker performance on Sintel vs. other methods (minor weakness, but addressing could further strengthen paper!)\n\nIn Table 9 of the supplementary material, we show that we outperform FlowFormer on all Sintel test sequences but one, ambush_1. This sequence has significant ambiguity due to an all-white background, large camera and object motions, and the existence of numerous objects that occlude one another and move out of frame. This forces the model to depend heavily on inductive biases and learned motion priors, which could be challenging for our method. It is possible that FlowFormer’s inductive bias (with the cost volume etc) make it better suited to reason about this sequence’s large out-of-frame motions. In general, we agree with the reviewer that our model’s worse performance on Sintel compared to models such as FlowFormer, which we outperform by a large margin on KITTI, is surprising and deserves further exploration.\n\n> Is it possible to do warm start with diffusion \n\nThere are a couple of ways of doing warm start with diffusion. One would be to use the warped flow as the initial estimate and perform partial denoising as the reviewer suggests. Another would be to guide the denoising process (similar to classifier guidance [19]) using a loss that ensures cross frame consistency. These are interesting directions, but we have not explored them in detail to date.\n\n> (1) it does not handle real data as well as regression methods, even after contributions in this area, or (2) it does not overfit as well to specific datasets given others tend to have task-specific architecture.\n\nAs the reviewer has noted, our model achieves SOTA performance on KITTI (a real dataset) by a large margin (Table 2). Relative to FlowFormer, our model’s weaker test performance on Sintel is somewhat surprising. This may be attributable to the differences in the data used for pre-training the Flowformer model vs our diffusion model, or to the presence of a task-specific bias in the FlowFormer architecture. We agree with the reviewer that this is worthy of future exploration.  \n\n> Minor weaknesses: 1) L1 loss is a design decision, infilling holes using bilinear interpolation is not a substantial contribution in my eyes. So the remaining technical contribution is unrolling. 2) These contributions are nontrivial, but not substantial on their own.\n\nWe agree that L1 loss and bilinear interpolation are not technical novelties by themselves. The paper’s contribution, in this context, is a way to train diffusion models with noisy and incomplete data, where the train-test distribution shift is problematic. We found that a combination of L1, bilinear infilling and step unrolling are effective for diffusion model training with missing data. This enables one to train generic diffusion models that yield remarkably strong performance on both optical flow and monocular depth estimation without the specialized architectures and loss functions that have been common in SOTA models to date.\n\n> Questions:\n\nQ1: Thank you for the suggestion. We will write this more clearly when revising the paper.\n\nQ2: We have included extra examples in Figure 1 in the attachment. We will include more examples in the final version of the paper.\n"}]

Author rebuttal:
References

[1] RAFT: Recurrent All-Pairs Field Transforms for Optical Flow, Teed and Deng, 2020

[2] FlowFormer: A Transformer Architecture for Optical Flow, Huang et al, 2022

[3] Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer, Ranftl et al, 2020

[4] Vision Transformers for Dense Prediction, Ranftl et al, 2021

[5] Palette: Image-to-Image Diffusion Models, Saharia et al, 2022

[6] All in Tokens: Unifying Output Space of Visual Tasks via Soft Token, Ning et al, 2023

[7] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, Saharia et al, 2023

[8] Unleashing Text-to-Image Diffusion Models for Visual Perception, Zhao et al, 2023

[9] Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model, Chen at al, 2023

[10] AutoFlow: Learning a better training set for optical flow, Deqing et al 2021

[11] Self-supervised AutoFlow, Huang et al, 2023

[12] Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation, Sun et al, 2018

[13] FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks, Ilg et al, 2017

[14] AdaBins: Depth Estimation using Adaptive Bins, Bhat et al, 2020 

[15] A Generalist Framework for Panoptic Segmentation of Images and Videos, Chen et al, 2023

[16] Score-based Generative Modeling through Stochastic Differential Equations. Song et al, 2021

[17] Vision Transformers for Dense Prediction, Ranftl et al, 2021

[18] Text-to-Image Diffusion Models are Zero-Shot Classifiers, Clark and Jaini, 2023

[19] Diffusion Models Beat GANs on Image Synthesis, Dhariwal et al, 2021


Meta-review: The paper has achieved a consensus for acceptance after the rebuttal. The technical contribution, which demonstrates that image diffusion models excel in handling general-purpose dense prediction tasks, is clearly articulated. All reviewers are positive about both the technical novelty and the results presented. The rebuttal addressed most minor concerns and clarity issues; therefore, acceptance should be granted without further discussion. Particularly due to its novelty and the rich insights provided, the AC recommends this paper for a spotlight.
Decision: Accept (oral)