Title: Evaluating Robustness of Monocular Depth Estimation with Procedural Scene Perturbations
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision

3d vision

procedural generation

evaluation

monocular depth estimation
Abstract: Recent years have witnessed substantial progress on monocular depth estimation, particularly as measured by the success of large models on standard benchmarks. However, performance on standard benchmarks does not offer a complete assessment, because most evaluate accuracy but not robustness. 
In this work, we introduce PDE (Procedural Depth Evaluation), a new benchmark which enables systematic evaluation of robustness to changes in 3D scene content.
PDE uses procedural generation to create 3D scenes that test robustness to various controlled perturbations, including object, camera, material and lighting changes. 
Our analysis yields interesting findings on what perturbations are challenging for state-of-the-art depth models, which we hope will inform further research. Code and data are available at https://github.com/princeton-vl/proc-depth-eval.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to evaluate a crucial, though not exhaustive, aspect of depth estimation models: robustness. The evaluation is conducted by introducing procedural scene perturbations to generate diverse variations of the original scenes. The authors perform an extensive assessment of recent state-of-the-art models and identify which types of perturbations each model is more robust to, compared to others.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The authors introduce an important, though not comprehensive, evaluation perspective for existing monocular depth estimation (MDE) models: robustness, which is a critical factor for practical deployment.\n- The study conducts extensive evaluations on recent state-of-the-art methods.\n- The paper reports some findings(L73-79) which highlight areas that could guide future improvements.\n\n**Weaknesses:**\n\n- My main concern is the data source: since the perturbations are applied primarily to a synthetic dataset (Infinigen), evaluating **robustness** solely on **synthetic** data may not provide sufficiently persuasive results.\n- The paper evaluates the effect of one perturbation at a time, whereas real-world out-of-distribution (OOD) images often involve multiple factors combined (e.g.,transparent object + dim lighting + out-of-focus blur).\n- Table 3 is confusing; it appears to report self-(in)consistency, yet the metric is again listed as Avg.Err, which is unclear.\n\nQUESTIONS:\n- I would like the authors to clarify why perturbations on synthetic datasets are considered representative of real-world generalization.\n- It would be valuable to see results for combinations of different perturbations, even if only a few combinations are tested.\n- I am interested in seeing how VGGT[1] performs under these perturbations. Since it is primarily trained with multi-input images, it may exhibit greater robustness to camera-related perturbations.\n\n[1] Wang J, Chen M, Karaev N, et al. Vggt: Visual geometry grounded transformer[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5294-5306.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo evaluate the robustness of monocular depth models, the paper introduces a synthetic dataset created with the procedural generator Infinigen. The dataset includes 5 objects, 38 distinct scenes and 12 controlled perturbations. The paper performs a comprehensive evaluation of the robustness of 9 state-of-the-art monocular depth models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Understanding the robustness of models to various perturbations is crucial, and the new dataset could offer a valuable tool for the research community.\n- The paper is well-written and easy to follow. Table 1 and Figure 1 provide a good example and summary of the involved perturbation types. \n\n**Weaknesses**\n- In L119, the paper states, *“We ensure the object of interest is salient in the scene”*. However, It is unclear how this is guaranteed. Furthermore, according to the Kaggle link provided in the appendix, for objects e.g.fish, in the scene 3674e176 , the target fish object only occupies a very small portion of the image, which raises questions about the claim of salience. \n- In Section 3.1, the perturbation type Camera Pedestal/Truck is mentioned, but it is not included among the 12 scene perturbations listed in Table 1 and Figure 1. If it is not part of the dataset, it should be removed from the text to avoid confusion.\n- In Table 3, according to the title the reported metrics are self-(in)consistency, while it shows as Avg.Err in the table which is the same as the metric name in Table 2, and it could be confusing. \n- There are no evaluation results based on the full scene, only object-based results are reported. Including full-scene results would provide a more comprehensive understanding of the models’ performance.\n\nQUESTIONS:\n- It would be good to explain the motivation for the design of some perturbations. For example, why the 2D size of objects have to be roughly maintained for perturbation e.g. Camera Dolly Zoom? Is it to keep the saliency and the visibility? \n- In L207-208, it is mentioned that *when evaluating on an object of interest, we compute alignment using only the depth values for the object*. Would it lead to metrics that capture only local robustness/consistency and potentially missing the global inconsistency among the predictions? What if the alignment is computed on the entire scene and then the error metrics are computed on the object area?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper evaluates the robustness of state-of-the-art monocular depth estimation models (e.g., MiDaS, DepthAnything, ZoeDepth) to procedural scene perturbations, including changes in camera parameters, lighting, materials, and object geometry. Using synthetic 3D scenes generated via Infinigen, the authors systematically test model performance under controlled perturbations, measuring both accuracy stability (consistency against ground truth) and self-consistency (prediction stability across perturbations). Key findings include:\nModels are robust to lighting changes but struggle with camera perturbations (e.g., dolly zoom, roll) and material swaps.\nOcclusions and non-rigid deformations are particularly challenging, revealing limitations in geometric reasoning.\nDepthPro and UniDepthV2 emerge as top performers, though trade-offs exist between accuracy and robustness.\nThe study highlights gaps in standard benchmarks (e.g., KITTI, NYU-D) by focusing on robustness rather than just accuracy, providing insights for real-world applications like robotics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This work introduces a procedurally generated dataset with controlled perturbations, enabling systematic robustness testing beyond standard benchmarks.\n\n2. Tests 12 perturbation types (e.g., camera intrinsics, material swaps, occlusions) across 9 models, revealing understudied failure modes (e.g., sensitivity to focal length changes).\nIncludes object-level evaluation (not just full-scene metrics) for finer-grained insights.\n\n3. This work identifies camera perturbations as a critical weakness, informing deployment in dynamic environments (e.g., autonomous vehicles). Highlights material invariance as an open challenge, suggesting directions for future work (e.g., physics-aware training).\n\n4. Plans to release open-source code for benchmarking new models, fostering community engagement.\n\n\nWeaknesses\n1. The work relies solely on synthetic data (Infinigen), which may not fully capture real-world noise (e.g., sensor artifacts, texture variations).\nDomain gap could affect generalizability, despite Infinigen’s photorealism.\n\n2.Experiment tests perturbations individually, while real-world scenarios often involve combined perturbations (e.g., lighting + occlusion).\n\n3. The coverage of urban/driving scenes is limited, which are critical for applications like autonomous navigation.\n\n4. Model Scope excludes recent foundation models (e.g., those leveraging diffusion or multimodal pretraining), which may exhibit different robustness profiles.\n\n5. The is no analysis of computational efficiency vs. robustness trade-offs (e.g., latency for real-time systems).\n\nQUESTIONS:\nQuestions \n\n1. Why were certain perturbation types (e.g., camera dolly zoom) prioritized over others (e.g., motion blur or weather effects)? How were perturbation parameters (e.g., occlusion percentage) determined?\n\n2. For perturbations that alter ground truth depth (e.g., object translation), how do you justify the \"small neighborhood\" constraint? Could larger perturbations reveal additional failure modes?\n\n3. Self-consistency is undefined for affine-invariant models (e.g., MiDaS). Did you explore alternative metrics (e.g., scale-invariant error) to include these models in consistency analysis?\n\n4. AbsRel error emphasizes large depth errors but may miss local inconsistencies. Did you consider edge-aware metrics (e.g., gradient-based errors) to evaluate geometric fidelity?\n\n5.Infinigen scenes lack real-world noise (e.g., sensor artifacts). How would results change if tested on real-world datasets with paired perturbations (e.g., Replica-CE)?\nThe object categories (chairs, fish, etc.) are limited. Would results generalize to complex urban scenes (e.g., crowded streets)?\n\n6.The paper identifies camera perturbations as highly challenging. How should robotics/AR systems mitigate this in practice (e.g., sensor fusion, temporal smoothing)?\n\n7. Will the perturbation code support custom scenes (e.g., Unity/Unreal Engine assets) beyond Infinigen?\n\n8. Could this framework evaluate video-based depth estimation (e.g., robustness to temporal inconsistencies)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Procedural Depth Evaluation (PDE), a new benchmark to assess the robustness of monocular depth estimation models beyond standard accuracy metrics. Using procedural generation, the authors create controlled 3D scene variations (perturbations in object properties, camera parameters, materials, lighting, etc.) to test whether state-of-the-art depth models maintain performance under changes in scene content. Several recent depth models including MiDaS, ZoeDepth, DepthPro, etc. are evaluated on synthetic scene with different types of perturbations, measuring robustness via accuracy stability and self-consistency metrics. The experiments reveal that while models are fairly robust to lighting changes and small 3D pose adjustments, they struggle with certain perturbations– notably occlusions, material swaps, and camera parameter shifts (e.g. camera rotation or focal length changes cause large performance drops). These findings highlight brittle cases for even top-performing depth estimators and underscore the need for improving robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Novel Benchmark for 3D Scene Perturbations: This work fills an evaluation gap by focusing on scene content changes (object shape/pose, camera intrinsics/extrinsics, lighting, materials) rather than only 2D image corruptions. To the reviewer's knowledge, robustness to such 3D perturbations had not been systematically evaluated before, making PDE a novel contribution.\n- Comprehensive and Controlled Evaluation: The benchmark uses 12 distinct perturbation types covering virtually all aspects of scene and imaging variations- from camera zoom/roll/pan, object rotations/translations/resizing, to material swaps, lighting changes, occlusion insertion, non-rigid object deformations, and even out-of-distribution background swaps (e.g. placing a chair underwater). Each perturbation varies only one factor at a time, enabling a clear analysis of its isolated effect on depth predictions.\n- Thorough Evaluation of SOTA Models: The authors evaluate 9 state-of-the-art monocular depth models, including both zero-shot and fine-tuned methods, large foundation models (e.g. DepthAnything, UniDepthV2) and highly accurate recent models like DepthPro. The analysis introduces clear metrics (accuracy stability and self-consistency) to quantify robustness. The results are well-documented, with comprehensive tables of error and robustness scores for each model and perturbation, along with qualitative examples. The paper yields insightful findings (e.g. camera rotations and focal length changes degrade performance far more than object pose changes), which are empirically backed by the data, and could guide future methods to improve their designs accordingly.\n\nWeaknesses:\nThis is a nice and complete work and all these are minor points for dataset improvements:\n- Synthetic Domain Only: The evaluation is conducted entirely on procedurally rendered scenes. While the synthetic data is photorealistic and comes with perfect ground truth, there remains a domain gap to real-world images (e.g., textures, sensor noise). The paper justifies this choice and argues humans can interpret the synthetic depths, but it does not demonstrate that models' robustness on PDE correlates with real-world robustness. The lack of any validation on real images or physical scenes means it’s unclear how the reported robustness issues translate to actual deployments. In fact, a small scale capture of such real-world data may be feasible and worth some discussion.\n- Limited Scene Diversity: The benchmark focuses on indoor objects and natural outdoor elements, but omits urban driving scenarios- a key domain for depth estimation (e.g. autonomous driving). There are no cars, streets, or building-rich scenes in PDE. Models might exhibit different robustness characteristics in street scenes (e.g. differing scale or motion) which is not evaluated.\n- Perturbations Tested in Isolation Only: PDE applies one perturbation at a time to a scene. Combined effects of perturbations (e.g. simultaneous changes in lighting and object pose, or compound weather+camera changes) are not explored. This is a sensible initial simplification to attribute causes. However it remains unknown if interactions between perturbations might further challenge depth models or if the effects simply add up.\n- Statistical Analysis and Uncertainty: The paper presents average error and stability metrics over the sampled scenes, but does not report any variability measures (e.g. error bars or statistical significance) for comparisons. Given the results involve many scenes and perturbation instances, it would strengthen the analysis to see variance or confidence intervals on model performance differences, and maybe picking up some more insights. As-is, one must assume the differences (often large) are meaningful, but formal significance tests are not discussed. Incorporating error bars in plots or statistical tests would better support claims that certain perturbations are ``much more difficult'' than others, beyond just avg values.\n\nQUESTIONS:\n- Real-World Correlation: Have you evaluated whether the robustness rankings or failure modes observed on synthetic PDE scenes carry over to real-world data? For example, if a model is sensitive to camera roll in PDE, is it also less reliable on real tilted-camera photos? Some confirmation on real images would strengthen confidence in the benchmark's relevance.\n- Combined Perturbations: You focused on single perturbations to isolate effects (which makes sense). Did you observe any indications that combining perturbations could be especially problematic? For instance, might a lighting change exacerbate the effect of an occlusion or camera change? It would be interesting to know if any preliminary tests were done or if you have thoughts on how robustness might compound (or not) in some of these cases.\n- Camera Perturbation : The finding that ``camera rotation and focal length changes are among the most challenging perturbations'' is interesting. Could you add more discussion to this? Is it due to how depth networks encode perspective (perhaps assumptions baked in from training data) or something like lack of camera calibration handling? DepthPro seems to handle some of these explicitly and how does it help? Understanding the cause might hint at how to make models more robust to these intrinsic/extrinsic changes.\n- Improving Robustness: Based on your results, do you have any insights or hypotheses on how to improve model robustness to the difficult perturbations? For example, would data augmentation with certain perturbations, or model architectures that explicitly account for camera intrinsics, help? While it’s beyond the paper’s scope to fully explore solutions, any suggestions for model developers (inspired by your analysis) would be valuable.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}