Title: DiffSF: Diffusion Models for Scene Flow Estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Scene Flow Estimation

Denoising Diffusion Models

Uncertainty
Abstract: Scene flow estimation is an essential ingredient for a variety of real-world applications, especially for autonomous agents, such as self-driving cars and robots. While recent scene flow estimation approaches achieve reasonable accuracy, their applicability to real-world systems additionally benefits from a reliability measure. Aiming at improving accuracy while additionally providing an estimate for uncertainty, we propose DiffSF that combines transformer-based scene flow estimation with denoising diffusion models. In the diffusion process, the ground truth scene flow vector field is gradually perturbed by adding Gaussian noise. In the reverse process, starting from randomly sampled Gaussian noise, the scene flow vector field prediction is recovered by conditioning on a source and a target point cloud. We show that the diffusion process greatly increases the robustness of predictions compared to prior approaches resulting in state-of-the-art performance on standard scene flow estimation benchmarks. Moreover, by sampling multiple times with different initial states, the denoising process predicts multiple hypotheses, which enables measuring the output uncertainty, allowing our approach to detect a majority of the inaccurate predictions. The code is available at https://github.com/ZhangYushan3/DiffSF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel diffusion model designed for scene flow estimation, aiming to enhance both accuracy and robustness, particularly in the presence of noisy inputs or occlusions. The proposed denoising diffusion models effectively handle noisy data by modeling stochastic processes, filtering out sensor-induced noise, and focusing on underlying patterns. This approach not only improves prediction accuracy but also introduces a measure of uncertainty by averaging over multiple hypotheses without altering the training process. Additionally, the paper presents a new architecture that integrates transformers with diffusion models, further boosting accuracy and robustness across various benchmark datasets.\n\nSTRENGTHS:\n1. The motivation for using diffusion models in scene flow estimation is well-justified. Real-world data often presents challenges such as severe occlusion and noisy inputs, necessitating robust models for scene flow estimation. \n2. The proposed method demonstrates state-of-the-art performance through extensive experiments on multiple benchmarks, including FlyingThings3D, KITTI Scene Flow, and Waymo-Open.\n3. The paper is well-written and easy to understand, making the concepts accessible to readers.\n4. The reproducibility of the results is strong, ensuring that the findings can be reliably replicated.\n\nWEAKNESSES:\nThe paper lacks qualitative results. I would've liked to see visual comparison of scene flow results (especially for challenging cases) on various datasets. This makes it hard to interpret where the proposed approach performs better than current methods and more importantly where the proposed approach fails.\n\nQUESTIONS:\n1. Line 250 states \"showcasing DiffSF’s effectiveness on real-world data..\" Can the authors provide some challenging samples of data that your model is robust against?\n2. Which datasets are used in generation of results for figures 3 and 4?\n\nLIMITATIONS:\nAuthors have presented the limitations of the work (which is a good thing), but this has been mostly discussed in the supplementary section (which is outside of the main paper). I would encourage the authors to include a brief summary of the limitations in the main paper as well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper deals with scene flow estimation in 3D point clouds. It proposes a formulation based on diffusion model. The model takes the source and target frames of 3D point clouds as condition and turn the problem into a conditional generation problem. Different from naive conditional generation, the formulation considered the characteristic of the scene flow problem to make each reverse diffusion step as a "flow adjustment". Experiments are done on standard benchmarks.

STRENGTHS:
- The proposed formulation considered the characteristic of the scene flow problem to make each reverse diffusion step as a "flow adjustment". It makes so much sense and is a elegant solution to this problem.
- The presentation of this paper is clear and easy to follow.
- The experiment results significantly outperform existing baselines.
- Code and visualization are provided.

WEAKNESSES:
- The author could consider providing more visualizations of the (reverse) diffusion process other than the only one GIF attached in the supplementary. This will be very helpful.formulation
- Minor: the citation notation seems to be wrong. For example, citing paper 1 should be "[1]", not "(1)".
- Minor: in Tables 1, 2 and 6, there are numbers in both bold and underlines. What are underlined numbers mean? This should be explained.

QUESTIONS:
What is the definition of "uncertainty"? I understand "EPE" and "outlier". Is there a definition of "uncertainty" somewhere in the paper?

LIMITATIONS:
Limitation was mentioned. No other limitation is found besides the ones mentioned in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DiffSF, which integrates transformer-based scene flow estimation with denoising diffusion models. The diffusion process involves progressively perturbing the ground truth scene flow vector field by adding Gaussian noise. Conversely, during the reverse process, the scene flow vector field prediction is recovered from randomly sampled Gaussian noise by conditioning on a source and a target point cloud. The DiffSF achieves state-of-the-art performance on FlyingThings3D, KITTI Scene Flow, and Waymo-Open.\n\nSTRENGTHS:\nExtending denoising diffusion models for scene flow estimation is a novel approach that is straightforward to understand. \n\nThe ablation study is comprehensive, such as evaluating the impact of varying the number of time steps during both training and sampling for the diffusion model.\n\nWEAKNESSES:\nThe tech contribution is limited. This paper does not invent a new concept given a relevant work of optical flow estimation using diffusion models [1]. \n\nTo enhance understanding of the diffusion process for scene flow estimation, visualizations of source points with added noise at different time steps should be included.\n\nThe study is limited to the GMSF model architecture. It is strongly recommended to evaluate other popular architectures, as the key concept of the paper is to introduce the diffusion model for scene flow estimation. Additionally, it would be valuable to investigate whether this approach works in the context of self-supervised scene flow estimation such as [2][3].\n\n[1] The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation\n\n[2] Self-Supervised Robust Scene Flow Estimation via the Alignment of Probability Density Functions\n\n[3] Just Go with the Flow: Self-Supervised Scene Flow Estimation\n\nQUESTIONS:\nIt would be beneficial to include a discussion or exploration on the effects of adding noise directly to the ground truth scene flow vector field versus adding noise to the embedding of the ground truth scene flow vector field, as achieved by a separate network such as a VAE model.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a scene flow estimation method that estimates scene flow from point clouds. The proposed method combines a previous scene flow method, GMSF, and diffusion model, where a denoising block is introduced based on GMSF.  During training, the source point cloud is first warped via noisy scene flow, and then the warped source point cloud and the target point cloud are fed to the denoising block.  Experimental results demonstrate the proposed method achieves the best performance compared with state-of-the-art approaches.  \nIn addition, the paper shows interesting results that the proposed method can be used to measure the hardness of input point clouds. That is, given a pair of point clouds, the proposed method can obtain multiple scene flow results using different initial randomly sampled noise, where the uncertainty (standard deviation) of scene flow results is highly correlated with their accuracy.\n\nSTRENGTHS:\n1. The idea of the proposed method is interesting and sounds good. \n\n2. The result of uncertainty-error correspondence is interesting, indicating the proposed method can be used to predict the hardness (estimation difficulty ) of the input point cloud pair.\n\n3. Extensive experimental results shows the effectiveness of the proposed method\n\nWEAKNESSES:\n1. The organization of the technical part can be improved. It is a bit difficult to follow Sec. 3.2. For example, the paper has already introduced the forward process, reverse process and the training loss, and then introduces the details of the reverse process again.\n\n2. The proposed method uses the same feature extractors with shared weights to extract features from the warped and target point clouds. However, the distributions of the warped point cloud may be rather different from the target one, since the warped point cloud is noisy and the target point cloud is clean.   Would such differences negatively affect the training of feature extractors? Similarly, the training of the linear layers in the Local-Global-Cross Transformer may also be negatively affected. If the feature extractor of the warped point cloud does not share weights with that of the target one, would the estimation performance of the proposed method be improved?  \n\n3. Table 7 in the Appendix shows that when GMSF and the proposed method use the same architecture, the proposed method's performance improvement is marginal on F3D_o, compared with GMSF. Does \bsuch performance also occur on other datasets ( such as KITTI and Waymo-Open datasets)? It is not clear which part of the proposed method mainly contributes to the performance improvement, the architecture or the denoising process.\n\n4. Some statements may be a bit overclaimed. The paper claims that \"To the best of our knowledge, we are the first to apply a diffusion model successfully for the task of scene flow estimation.\". However, as described in the main paper, a few methods have attempted to employ diffusion models for scene flow estimation [24]. The statement of \"the first\" may be overclaimed and can be replaced with a more proper claim.\n\nQUESTIONS:\nPlease refer to my comments above.\n\nLIMITATIONS:\nThe paper provides the limitations of the proposed method, while it does not discuss the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}