Title: Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: monocular depth estimation; generative depth estimation models; diffusion transformers; relative depth
Abstract: This paper presents **Pixel-Perfect Depth**, a monocular depth estimation model based on pixel-space diffusion generation that produces high-quality, flying-pixel-free point clouds from estimated depth maps. Current generative depth estimation models fine-tune Stable Diffusion and achieve impressive performance. However, they require a VAE to compress depth maps into the latent space, which inevitably introduces flying pixels at edges and details. Our model addresses this challenge by directly performing diffusion generation in the pixel space, avoiding VAE-induced artifacts. To overcome the high complexity associated with pixel-space generation, we introduce two novel designs: 1) **Semantics-Prompted Diffusion Transformers** (**SP-DiT**), which incorporate semantic representations from vision foundation models into DiT to prompt the diffusion process, thereby preserving global semantic consistency while enhancing fine-grained visual details; and 2) **Cascade DiT Design** that progressively increases the number of tokens to further enhance efficiency and accuracy. Our model achieves the best performance among all published generative models across five benchmarks, and significantly outperforms all other models in edge-aware point cloud evaluation. Project page: https://pixel-perfect-depth.github.io/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an innovative approach to relative depth estimation using Semantics-Guided Diffusion Transformers (SG-DiT). Focused on generating high-quality depth maps, the paper effectively addresses challenges such as flying pixels and edge artifacts. The claims are robustly supported by both theoretical foundations and experimental results throughout the study, showcasing the model's ability to generalize across diverse scenarios. Furthermore, the paper provides insights into the computational efficiency of the proposed methods. Overall, it is well-structured, thoroughly addressing vital issues concerning generalization and reproducibility, thereby marking a significant advancement in the domain of more precise relative depth estimation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper presents detailed theoretical and experimental validation, thoroughly discussing the model's architecture while providing quantitative evaluation metrics and visual quality demonstrations. It proposes an innovative solution for monocular relative depth estimation, with considerable potential for practical applications. The paper effectively addresses typical issues related to flying pixels and edge artifacts, offering robust comparisons and solutions.\n\nWeaknesses:\n\n- It seems that post-processing techniques, such as filtering depth at certain positions using image gradient information, could also resolve the flying pixels issue?\n\n- Compared to relative depth, monocular metric depth estimation is more practical. Can the method proposed in this paper be extended to monocular metric depth estimation?\n\n- How does this method perform in extremely detailed regions of an image, such as hair details showcased in Depth Pro?\n\nQUESTIONS:\nSee in the weaknesses part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Pixel-Perfect Depth, a monocular depth estimation framework that operates in pixel space rather than the more common latent space, thereby avoiding flying-pixel artifacts introduced by VAE compression. To address the complexity of high-resolution generation, the authors propose two novel architectural components: Semantics-Guided Diffusion Transformers (SG-DiT) and a Cascade DiT design (Cas-DiT). This method achieves SOTA results and introduces an edge-aware evaluation metric for assessing depth quality near object boundaries. I find the results promising, and I believe the paper would be further strengthened if the experimental issues noted in the Weaknesses section are addressed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The motivation is clear and well-justified: the method targets the flying pixel issue in depth estimation models, particularly at object boundaries. By proposing a pixel-space diffusion framework with semantic guidance, the authors offer a novel and effective solution. The qualitative and quantitative results across diverse benchmarks demonstrate strong performance.\n2. The paper introduces a new edge-aware evaluation metric designed to quantify depth estimation accuracy specifically at object boundaries. This addresses a gap in current evaluation protocols and is a meaningful contribution to the field.\n\nWeaknesses:\n1. In the qualitative ablation analysis (Figure 5), only depth map visualizations are shown. Including corresponding point cloud visualizations (as in Figure 1) would help better assess how this component impacts the flying-pixel artifacts.\n2. Since the core motivation of the paper is to reduce flying pixels, and a new edge-aware evaluation metric is proposed for that purpose, it would strengthen the paper to conduct ablation studies using this metric. This would provide more direct evidence of how each module contributes to mitigating boundary errors.\n\nQUESTIONS:\nPlease refer to the Weaknesses. If the authors can adequately address those concerns in the rebuttal, I would consider increasing my score from 4 to 5.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the limitations of current generative and discriminative models for monocular depth estimation. Specifically, it identifies two major issues: (1) the training instability and poor performance of pixel-based diffusion models, and (2) the boundary over-smoothing problem in discriminative models. To address these challenges, the authors propose a novel geometry-guided function model, which effectively stabilizes the training of diffusion models and preserves sharp depth boundaries. The experimental results support the authors' analysis and demonstrate that their method achieves state-of-the-art performance with a relatively simple network architecture.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. To the best of my knowledge, this is the first work that combines a foundation geometry model with a pixel-space diffusion model for monocular depth estimation. Pixel-based diffusion model is particularly meaningful, as latent-space representations inevitably suffer from information loss. Therefore, the proposed approach is both novel and potentially impactful.\n2. The paper provides insightful analysis of the limitations in current depth estimation models, which may inspire future research directions. Moreover, the proposed solution is simple yet effective. By adopting a pure-transformer architecture, the method remains lightweight and easily extensible, which is beneficial for follow-up work.\n3.The experiments are comprehensive and convincingly demonstrate that the proposed method achieves state-of-the-art performance.\n4. The writing is clear, well-organized, and easy to follow. \n\nWeakNesses.\n\n1.Although the simple geometry integration achieves SOTA, it is unclear if more efficient or better alternatives than normalize + concat exist. The improvement over DepthAnything v2 is not very significant, and more ablation studies are needed to justify this design choice.\n\n2. The paper lacks a thorough analysis of efficiency. While it mentions that diffusion models inevitably slow down inference, it does not quantify how much slower they are. A more precise comparison of inference time would be necessary.\n\n3. The paper lacks deeper theoretical insight. While it is somewhat expected that geometry priors can enhance diffusion-based depth estimation, the paper does not clearly analyze the underlying challenges or implications of combining the two. As a result, the novelty seems to reside more in the practical combination of known components than in a fundamentally new conceptual contribution.\n\nQUESTIONS:\n1. What is the inference time for a 512x512 images.\n2.The paper mentions that the geometry features are normalized due to their smaller magnitude compared to diffusion features. However, if the diffusion features are not globally normalized either, wouldn't the scale mismatch still affect feature fusion? Would it be more appropriate to normalize both or apply a learnable scaling? Would you mind do an ablation about this?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Pixel-Perfect Depth, a monocular depth estimation model that leverages pixel-space diffusion transformers to generate high-quality, flying-pixel-free depth maps. Unlike prior generative models (e.g., Marigold) that use Variational Autoencoders (VAEs) for latent-space compression, this framework avoids VAE-induced artifacts by performing diffusion directly in the pixel space. Key contributions include:\n\nSemantics-Guided Diffusion Transformers (SG-DiT): Integrates high-level semantic representations from vision foundation models (e.g., DINOv2, Depth Anything v2) to guide the diffusion process, enabling accurate modeling of global structures and fine-grained details. This is achieved via L2-normalized semantic features and a bilinear interpolation mechanism to align with transformer tokens .\n\nCascade DiT Design (Cas-DiT): Adopts a coarse-to-fine strategy where early transformer blocks use large patch sizes (16×16) for global structure modeling, and later blocks use smaller patches (8×8) for fine details. This reduces computational cost by 30% while improving accuracy .\n\nState-of-the-Art Performance: Trained on 54K samples from Hypersim, the model outperforms all published generative models on five benchmarks (NYUv2, KITTI, etc.), achieving an AbsRel of 4.3 on NYUv2. It also introduces an edge-aware point cloud metric, demonstrating 0.08 Chamfer Distance (vs. 0.12 for VAE-based GT) .\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nOriginality:\n\nThe direct pixel-space diffusion approach eliminates flying pixels caused by VAE compression, a critical limitation in prior work .\nSemantic guidance from pretrained models and the cascade design are novel, addressing the challenge of balancing global-local features in high-resolution generation .\n\nMethodological Rigor:\n\nAblation studies validate the necessity of SG-DiT and Cas-DiT, showing up to 78% improvement in AbsRel when both components are used .\nCross-dataset evaluation on diverse scenes (indoor/outdoor) confirms generalization .\n\nPractical Significance:\n\nThe edge-aware metric and flying-pixel-free results are crucial for real-world applications like 3D reconstruction and robotic manipulation .\n\nWeaknesses\n\nComputational Overhead:\n\nTraining requires 8 NVIDIA GPUs for 800K steps, limiting accessibility for small labs .\n\nDataset Limitations:\n\nThe training dataset (Hypersim) is synthetic, and real-world datasets lack diversity in recording devices or demographics .\n\nInference Speed:\n\nMulti-step diffusion leads to slower inference (0.14s on RTX 4090) compared to discriminative models like Depth Anything v2 .\n\nQUESTIONS:\nQuestion1: How does the model perform with depth data from consumer-grade devices (e.g., iPhone LiDAR) vs. clinical systems? Have you tested for device-specific noise patterns?\n\nQuestion2: The paper notes flickering depth in video applications. What strategies are planned to integrate temporal consistency (e.g., optical flow or video diffusion priors)?\n\nQuestion3: Are there plans to implement techniques like knowledge distillation or token caching to reduce inference time for edge deployment?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}