Title: Self-Distilled Depth Refinement with Noisy Poisson Fusion
Venue: NeurIPS 2024
Accepted: yes
Keywords: Depth refinement

Noisy Poisson fusion

Self-distilled training
Abstract: Depth refinement aims to infer high-resolution depth with fine-grained edges and details, refining low-resolution results of depth estimation models. The prevailing methods adopt tile-based manners by merging numerous patches, which lacks efficiency and produces inconsistency. Besides, prior arts suffer from fuzzy depth boundaries and limited generalizability. Analyzing the fundamental reasons for these limitations, we model depth refinement as a noisy Poisson fusion problem with local inconsistency and edge deformation noises. We propose the Self-distilled Depth Refinement (SDDR) framework to enforce robustness against the noises, which mainly consists of depth edge representation and edge-based guidance. With noisy depth predictions as input, SDDR generates low-noise depth edge representations as pseudo-labels by coarse-to-fine self-distillation. Edge-based guidance with edge-guided gradient loss and edge-based fusion loss serves as the optimization objective equivalent to Poisson fusion. When depth maps are better refined, the labels also become more noise-free. Our model can acquire strong robustness to the noises, achieving significant improvements in accuracy, edge quality, efficiency, and generalizability on five different benchmarks. Moreover, directly training another model with edge labels produced by SDDR brings improvements, suggesting that our method could help with training robust refinement models in future works.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Self-distilled Depth Refinement (SDDR) framework to enhance robustness against noise. This framework primarily includes depth edge representation and edge-based guidance. And they design an edge-guided gradient loss and an edge-based fusion loss.  Furthermore, experiments on five benchmarks have validated the effectiveness of the framework.\n\nSTRENGTHS:\nThe paper is well-written, and the method and results are presented clearly. Using the strategy of iterative depth refinement is helpful for this task. The framework achieves excellent performance on widely used deblurring datasets.\n\nWEAKNESSES:\n1. The authors did not specify in the paper which sensor or modality was used for depth estimation.\n2. The authors decouple depth prediction errors into two degradation components: local inconsistency noise and edge deformation noise. And they give two samples in Fig. 2. However, I don't think this makes it clear that the errors fall into either category. The authors should provide accurate sources of error and analyze the significance of these two types of errors among all the errors.\n3. Line 121: In Motivation Elaboration, the authors primarily focused on analyzing the limitations of previous work but did not offer a compelling justification for the motivation behind their current study.\n4. Line 149: The authors claim to propose an edge-guided gradient loss, but this loss has also been used in the following two papers. Please clarify the differences. \\\n     [1] Wang Z, Ye X, Sun B, et al. Depth upsampling based on deep edge-aware learning[J]. Pattern Recognition, 2020, 103: 107274. \\\n    [2] Qiao X, Ge C, Zhang Y, et al. Depth super-resolution from explicit and implicit high-frequency features[J]. Computer Vision and Image Understanding, 2023, 237: 103841.\n\nQUESTIONS:\nThe paper proposes a novel and interesting framework and demonstrates a clear advancement in this task performance, so I would like to accept the paper. However, the lack of clear motivation makes the paper somewhat confusing to read, so I recommend borderline. If the authors can address my concerns, I am ready to change my recommendation based on the comments.\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel framework called Self-Distilled Depth Refinement (SDDR) to enhance depth refinement, which aims to infer high-resolution depth maps with fine-grained edges from low-resolution depth estimations. The authors propose modeling depth refinement as a noisy Poisson fusion problem, addressing local inconsistency and edge deformation noises. The SDDR framework consists of depth edge representation and edge-based guidance. Through coarse-to-fine self-distillation, SDDR generates low-noise depth edge representations, which serve as pseudo-labels to guide the refinement process. The method demonstrates significant improvements in accuracy, edge quality, efficiency, and generalizability across five different benchmarks.\n\nSTRENGTHS:\n1. Innovative Approach: The modeling of depth refinement as a noisy Poisson fusion problem is a novel and insightful approach that effectively addresses common issues in depth refinement.\n\n2. Robust Framework: The self-distillation technique employed in SDDR enhances robustness against noise, resulting in high-quality depth maps with accurate edges.\n\n3. Comprehensive Evaluation: The authors conduct extensive experiments across five benchmarks, showcasing the method's superior performance in various scenarios.\n\n4. Efficiency: SDDR achieves higher efficiency compared to two-stage tile-based methods, reducing computational costs while maintaining or improving accuracy and edge quality.\n\n5. Generalizability: The framework demonstrates strong generalizability, performing well on both synthetic and real-world datasets.\n\nWEAKNESSES:\n1. Real-World Application: The paper primarily focuses on benchmarks and does not provide extensive discussion on real-world applications and potential limitations in practical scenarios.\n\n2. Edge Case Handling: The method’s performance in handling extreme edge cases or highly noisy data is not thoroughly explored.\n\n3. Ablation Studies: More detailed ablation studies are needed to understand the contribution of each component within the SDDR framework.\n\nQUESTIONS:\n1. In Line 107-108, the logic between the previous works and your motivation is unsmooth; it would be better to provide more analysis.\n\n2. In Figure 2, the author didn't show the edge deformation visualization. What kind of deformation would occur in depth estimation? Most off-the-shelf depth estimators have not produced deformed results in depth edges or other regions.\n\n3. Do the results in state-of-the-art work like DepthAnything-V2 exhibit the artifacts, consistency, and deformations proposed by the authors?\n\n4. Can you provide more insights into the performance differences between synthetic and real-world datasets?\n\n5. The performances compared to other works seem not very significant due to the combination of several contributions in this paper. How can the authors judge whether the gains are from randomness of training, the settings of the models' parameters, or other factors?\n\n6. What is the training time comparison between SDDR and other state-of-the-art methods?\n\n7. How sensitive is the SDDR framework to hyperparameter settings?\n\n8. How does SDDR perform on extreme edge cases with very high noise levels?\n\n9. Can SDDR be adapted for real-time applications, and what modifications would be necessary?\n\nLIMITATIONS:\nThe paper provides extensive quantitative and qualitative results showcasing the strengths of the SDDR framework. However, it lacks a detailed analysis of potential failure cases or scenarios where the method does not perform well, which could provide insights for further improvements.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework, SDDR, for enhancing the resolution and detail of depth maps generated by estimation models. By conceptualizing depth refinement within the context of noisy Poisson fusion, the authors have developed a method that effectively tackles the prevalent issues of inefficiency and inconsistency. SDDR incorporates a self-distillation technique that enhances depth edge precision and diminishes noisy disturbances, resulting in a significant boost in both the accuracy and quality of the depth maps.\n\nSTRENGTHS:\n1. The paper introduces SDDR that addresses the limitations of traditional depth refinement methods by treating the problem as a noisy Poisson fusion task.\n2. SDDR demonstrates strong robustness against local inconsistency and edge deformation noise, which are common in depth prediction tasks, leading to improved accuracy and edge quality.\n3. The use of edge-guided gradient loss and edge-based fusion loss as part of the optimization objective results in more accurate and noise-free depth edge representations.\n\nWEAKNESSES:\n1. The performance of SDDR might be highly dependent on the quality of the initial depth predictions; noisy or low-quality inputs could affect the final output.\n2. The refinement process might inadvertently smooth out important details or edges in the pursuit of noise reduction.\n3. The authors claim that the proposed method have a promising performance, especially on edge sharpness, however, it seems not that visually competitive compared with some SOTA monocular depth estimation methods, such as DepthAnythingV2. I understand that there are some differences on the targets between DepthAnything and this paper.\n\nQUESTIONS:\n1. In 4.2, Coarse-to-fine Edge Refinement, is S=3 enough for getting the best results? Higher value of S can be explored. is there any trade-off?\n2. The paper demonstrates its promising performance on edge regions. Edge-specific measurements should be presented to highlight this point.\n\nLIMITATIONS:\n1. The method is designed to address specific types of noise (local inconsistency and edge deformation) and may not perform as well with other noise characteristics or in the presence of different degradations.\n2. The refined depth maps, despite improvements, might still contain artifacts or inconsistencies, particularly in complex scenes with challenging depth structures. The authors should present some of the results of those corner cases. If there is no such cases, the authors should point out the reasons and give a deep analysis for readers.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}