Title: RPG360: Robust 360 Depth Estimation with Perspective Foundation Models and Graph Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: monocular depth estimation

omnidirectional

foundation model

scale alignment

graph optimization
Abstract: The increasing use of 360$^\circ$ images across various domains has emphasized the need for robust depth estimation techniques tailored for omnidirectional images. However, obtaining large-scale labeled datasets for 360$^\circ$ depth estimation remains a significant challenge. In this paper, we propose RPG360, a training-free robust 360$^\circ$ monocular depth estimation method that leverages perspective foundation models and graph optimization. Our approach converts 360$^\circ$ images into six- face cubemap representations, where a perspective foundation model is employed to estimate depth and surface normals. To address depth scale inconsistencies across different faces of the cubemap, we introduce a novel depth scale alignment technique using graph-based optimization, which parameterizes the predicted depth and normal maps while incorporating an additional per-face scale parameter. This optimization ensures depth scale consistency across the six-face cubemap while preserving 3D structural integrity. Furthermore, as foundation models exhibit inherent robustness in zero-shot settings, our method achieves superior performance across diverse datasets, including Matterport3D, Stanford2D3D, and 360Loc. We also demonstrate the versatility of our depth estimation approach by validating its benefits in downstream tasks such as feature matching 3.2 ∼ 5.4% and Structure from Motion 0.2 ∼ 9.7% in AUC@5$^\circ$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RPG360, a novel training-free method for monocular 360-degree depth estimation. The core idea is to leverage the robust capabilities of pre-trained perspective foundation models, circumventing the need for large, labeled 360-degree datasets. The approach begins by converting an equirectangular (ERP) image into a six-face cubemap representation, where each face is processed independently by a perspective foundation model to predict initial depth and surface normal maps. The central contribution lies in a graph-based optimization framework designed to address the scale inconsistencies that arise when merging these independently estimated depth maps from different cubemap faces. This optimization refines the depth and normal maps by incorporating a per-face scale parameter, ensuring global depth scale consistency and preserving 3D structural integrity. The authors demonstrate the method's effectiveness and robustness in zero-shot settings across diverse datasets like Matterport3D, Stanford2D3D, and 360Loc, and illustrate its utility in downstream tasks such as feature matching and Structure-from-Motion.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths of RPG360\n\n* **Effective Adaptation of Perspective Foundation Models**: The paper presents an innovative strategy for leveraging powerful, pre-trained perspective depth models for 360-degree imagery. This \"training-free\" approach is a significant advantage, as it inherently bypasses the challenges of limited large-scale, labeled 360-degree datasets and mitigates domain generalization issues commonly faced by supervised learning methods in this domain.\n* **Novel Graph Optimization for Consistent Scale Alignment**: A key technical strength is the introduction of a novel graph-based optimization framework, notably incorporating a per-face scale parameter ($\\lambda_c$). This method offers an effective solution to the critical problem of depth scale inconsistencies that arise when fusing independently estimated depth maps from multiple cubemap faces. The ablation study (Table 5) clearly demonstrates the substantial positive impact of this per-face parameter on the quality of 3D reconstruction.\n* **Robustness and Strong Zero-Shot Generalization**: RPG360 consistently exhibits robust performance in zero-shot settings across a variety of datasets, including Matterport3D, Stanford2D3D, and 360Loc. This is a notable benefit, as the method effectively operates without requiring 360-degree training datasets and is consequently less susceptible to domain gaps often encountered by other approaches.\n* **Demonstrated Utility in Downstream Applications**: The paper effectively showcases the practical applicability of its high-quality depth maps by validating their benefits in common 360-degree vision tasks such as dense feature matching and Structure-from-Motion (SfM). This provides compelling evidence of the method's real-world impact beyond raw metric scores.\n\n### Weaknesses of RPG360\n\n* **Implicit Foundation Model Specification for Main Results**: While the ablation study (Table 5) lists and evaluates several foundation models, the specific model (including its version and backbone configuration) used to generate the primary \"RPG360 (Ours)\" results in Tables 1, 2, and 3 is not explicitly stated. For complete clarity and reproducibility, this crucial detail should be directly declared in the respective table captions or accompanying text.\n* **Limited Comparative Analysis of Optimization Strategies**: The paper thoroughly describes its proposed graph optimization approach and its internal components. However, it lacks a broader discussion or comparative analysis against other established or alternative global scale alignment or fusion optimization techniques present in the existing literature. This omission makes it challenging to fully assess the distinct advantages and necessity of their particular graph formulation compared to other potential solutions in the domain.\n* **Partial Reproducibility Details for Certain Baselines**: Although some baseline models are identified (e.g., \"BiFuse++ [59]\" for \"Depth Anywhere [60]\"), comprehensive retraining specifics, such as the exact number of epochs or precise learning rate schedules, are not provided for all models, particularly those that undergo training on pseudo-labeled data (e.g., \"Depth Anywhere [60]\" on 360Loc). This lack of detail could hinder the exact reproduction of certain baseline results.\n* **Lack of Explicit Surface Normal Ablation**: While surface normals are integrated into the graph optimization's formulation (Eq. 6), the paper does not present a dedicated ablation study that directly isolates and quantifies the performance impact of using normals versus entirely omitting them from the optimization framework. The current ablation (Table 5) primarily highlights the substantial impact of the per-face scale parameter, leaving the individual contribution of surface normals less clear.\n* **Imbalanced Space Allocation**: While the demonstrations of the method's utility in downstream tasks are valuable and well-executed, the considerable space dedicated to these aspects could potentially be optimized for conciseness. This might allow for more in-depth discussions, additional ablation studies focused directly on the core depth estimation methodology, its theoretical underpinnings, or deeper analyses of failure modes within the main paper.\n* **Limited Comparative Scope against 3D-Focused Methods**: While the paper effectively compares against learning-based 360-degree depth estimation methods, the absence of a direct comparison with established planar-based 3D reconstruction techniques, which also prioritize geometric alignment and structural correctness, represents a missed opportunity to fully contextualize RPG360's claimed superior 3D structural awareness.\n* **Unanalyzed Robustness to Image Degradations**: Although the paper claims \"robustness\" and demonstrates strong zero-shot generalization to diverse scenes, it does not provide an analysis of the method's performance when presented with common image degradations such as low-light conditions, noise, or blur. It remains unclear how the proposed graph optimization, which relies on initial depth and normal maps, performs or is affected under these challenging input scenarios.\n* **Suboptimal Table Formatting**: The visual presentation and separation of information within Tables 4 and 5 are suboptimal. The uneven or off-center alignment of elements within these tables can be distracting and hinder quick interpretation of the presented quantitative results.\n\nQUESTIONS:\n1. Specificity of Foundation Model Version: In Ln. 276, the paper references \"Metric3D v2 [25]\". Could the authors confirm that all instances where Metric3D is utilized as the foundation model throughout the paper, particularly for the \"RPG360 (Ours)\" results in Tables 1, 2, and 3, consistently refer to and use \"Metric3D v2\" as detailed in reference [25]?\n\n2. Could the authors reformat Table 4. and Table 5. for readability:\nThe presentation of quantitative results in Table 4 and Table 5 would benefit from enhanced readability. We suggest the authors consider reformatting these tables to improve clarity, such as by adjusting column spacing, ensuring consistent value alignment, or utilizing more distinct visual separations between logical sections. This would facilitate easier interpretation of the presented data.\n\n3. Comparison with Planar-Based 360 and 3D Reconstruction Methods: \nGiven the paper's emphasis on 3D metrics and structural correctness in depth estimation, a comparison with established planar-based 3D reconstruction methods would provide valuable context. These methods also prioritize 3D alignment and geometric accuracy. Could the authors include a comparative analysis against representative planar-based approaches to further contextualize RPG360's performance in 3D structural awareness?\n\n4. Inference Time and Computational Efficiency: \nThe paper currently lacks a comprehensive analysis of inference time. To fully assess the practical applicability and computational efficiency of RPG360, please provide the end-to-end processing time for a single 360-degree image. This should include the time taken for depth estimation by the foundation model and the subsequent graph optimization step.\n\n5. Robustness to Image Degradation: \nThe paper highlights the robustness of RPG360. While zero-shot generalization to diverse scenes is demonstrated, it is crucial to understand the method's performance under various image degradations (e.g., low-light conditions, noise, motion blur). Given that the initial depth and normal maps from perspective foundation models may degrade in such scenarios, how does the proposed graph optimization specifically perform and affect the final depth quality under these challenging input conditions? Does it exhibit graceful degradation, amplify errors, or provide resilience?\n\n6. Limitations in Thin Structures and Fine Details: In Section 5, the authors mention, 'Our optimization-based approach may produce less detailed depth estimations for thin structures compared to training-based methods.' Could the authors expand on the specific reasons for this limitation? Is it inherent to the graph optimization's smoothing properties, limitations of the cubemap projection near edges, or a characteristic inherited from the foundation models? Discussing potential strategies or future work to mitigate this limitation would enhance the paper's completeness.\n\n7. Visualizations of Failure Modes/Limitations: Beyond quantitative metrics, qualitative visualizations of the method's limitations or failure modes would be highly beneficial. Specifically, could the authors provide examples (e.g., in supplementary material) illustrating scenarios where 'less detailed depth estimations for thin structures' are evident, or where distortions or inconsistencies persist despite the graph optimization? This would offer crucial insights into the method's current boundaries.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method that offers a training-free 360° depth estimation method. They leverage perspective foundation models that predict depth for cubemap representations of the 360° scene. The predicted depth maps are aligned through graph-based optimization addressing scale inconsistencies. The method shows superior zero-shot performance compared to training-free foundation models on datasets like Materport3D, Stanford2D3D and 360Loc.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe 360-degree foundation model appears useful and relevant.\nFigure 4 presents compelling results.\nThe manuscript is well-written and easy to follow.\nThe authors plan to release the code, which will benefit the community.\n\nWeaknesses:\n\nA 360-degree foundation model with zero-shot performance is promising, but its value depends on how it can be used. It should be made clear whether the goal is real-time application or generating pseudo-ground truth. Since runtime is not reported, this remains uncertain.\n\nWhile the zero-shot performance on Matterport is strong, RPG360 lags behind Elite360D by over 10% on Stanford2D3D and on 360Loc. The authors suggest Elite360Ds limited performance on Matterport may be due to Stanford2D3D’s limited training size, which limits generalization for models like Elite360D. However, this comparison seems unfair, as RPG360's foundation models were pre-trained on a much larger and more diverse dataset corpus. It should be examined whether training competing methods such as [1] on more data (excluding Matterport, as it’s used only for zero-shot evaluation) would significantly improve their performance on Matterport. Not addressing this leaves open whether a properly trained alternative could outperform the proposed method as a foundation model.\n\nUnclear how model performs in outdoor environments where surface assumptions (walls etc) don’t apply as much such as indoors -> Dur360BEV (360 autonomous driving dataset) or similar dataset evaluation would be interesting\n\n* Yuan, Chao, et al. \"Dur360BEV: A Real-world 360-degree Single Camera Dataset and Benchmark for Bird-Eye View Mapping in Autonomous Driving.\" ICRA 2025\n\nAblation studies on both the efficiency trade-off (number of optimization iterations vs. performance) and the impact of incorporating surface normals (and the extent of their improvement) would be helpful.\n\nIn line 244ff, the authors note that using a model predicting standard (non-inverse) depth leads to degraded performance in 360MonoDepth. However, their reimplementation uses Metric3D V2, which they acknowledge performs suboptimally. While I agree that 360MonoDepth’s limited adaptability to standard depth values is a drawback, it would be more appropriate to evaluate the proposed method by adapting it, and the evaluation metrics, to 360MonoDepth as-is, rather than reimplementing 360MonoDepth with a different model. A comparison with the proposed method using the foundation model of 360MonoDepth (e.g., MiDaS V3, with surface normals computed via post-processing if needed) and traditional metrics (e.g., AbsRel, MAE, δ₁,₂,₃) would make the reported performance gains more convincing.\n\nIt would be valuable to demonstrate that this method can generate high-quality pseudo-ground truth and enhance 360° depth estimation approaches like those in Depth Anything, which rely on perspective-based pseudo labels. For example, evaluating whether zero-shot performance of top-performing models like Elite360D can be further improved using this approach would be interesting.\n\n\nMinor:\n\n- “For methods not covered in these papers, we referred to the original 30 publications to obtain their reported values.” \nWhy do the authors mention this? The reader might not have the time to go through all related papers to retrieve reported value. Relevant methods should be included, if not enough space in the main paper they should be included in the supplemental material.\n\nQUESTIONS:\nCould the authors clarify whether the model is intended for real-time deployment or for generating pseudo-ground truth, and provide runtime measurements to support this?\n\nGiven that RPG360 is pre-trained on a large and diverse dataset corpus, whereas competing methods like Elite360D are trained only on Stanford2D3D—which you note may be too small to generalize well—have the authors considered whether training such methods on more data (excluding Matterport, which is used only for zero-shot evaluation) would significantly improve their performance on Matterport? Without this comparison, it remains unclear whether RPG360’s superior zero-shot performance stems from its architecture or simply from access to more training data.\n\nCould the authors evaluate the model’s performance on challenging outdoor 360° datasets such as Dur360BEV or scenes to demonstrate its generalization beyond indoor environments where surface assumptions are valid only to a limited extent?\n\nCould the authors provide ablation studies showing the trade-off between the number of optimization iterations and performance, as well as the specific impact of incorporating surface normals?\n\nWhy did the authors choose to reimplement 360MonoDepth with a different backbone (Metric3D V2) instead of evaluating your method using 360MonoDepth's original foundation model (e.g., MiDaS V3) and standard 2D metrics?\n\nCan the authors demonstrate whether your method improves existing models like Elite360D when used to generate pseudo-ground truth, in line with approaches such as Depth Anything?\n\nLIMITATIONS:\nNo, runtime is potentially a limitation and should be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes RPG360, a training-free pipeline for monocular 360-degree depth estimation. A panorama is converted to a 6-face cubemap; each face is fed to an off-the-shelf perspective foundation model to predict depth and normals. A subsequent graph optimisation introduces a learnable per-face scale λ_c and planar depth–normal constraints, aligning all faces into a single, scale-consistent ERP depth map.

STRENGTHS AND WEAKNESSES:
The method is solid and yields large 360-depth gains without retraining, giving high practical value; yet novelty is mostly λ_c + graph optimisation, with no runtime/memory/hyper-parameter study, limited performance on thin structures, no comparison to recent diffusion approaches, and some notational clutter.

QUESTIONS:
1. Considering that graph optimization may involve complex iterative calculations and may be limited by computational resources in practical applications.  Please report average optimisation time and GPU/CPU memory per 1024×512 panorama.  A plot of objective vs iterations would help.
2. How does the optimization of the per-face scale parameter λc in Eq 8 prevent depth discontinuity at cubemap face boundaries? Is there a conflict between λc (global scale alignment) and local smoothness constraints (Eq 6)?
3. Provide a robustness table for (η_p, η_d, η_n, σ_int, σ_spa) across datasets, or propose an automatic heuristic.

LIMITATIONS:
yes—Authors honestly mention smoothing of thin structures and slower inference vs feed-forward networks.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a depth map estimation method from omnidirectional(360) images. The method utilizes the strength of recent foundation models to estimate accurate depth maps. Since those foundation models are trained based on perspective images rather than omnidirectional images, the method starts with partitioning the image into cube maps. After that, the method estimates the ERP space scale map using graph optimization with depth maps and normal maps. By using scale, the final resulting depth map is obtained. The method is evaluated on three datasets, using standard 3D and 2D metrics.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed method uses perspective foundation models for the omnidirectional image, which seems to work. The core contribution to this is graph optimization for scale alignment. The proposed graph formulation, weight estimation, and optimization equation seem technically sound. The paper also includes comprehensive comparisons and ablation experiments, including variations on foundation models (Table 5) and other partitioning methods (Table 3). The resulting quantitative metrics and images show the state-of-the-art performance. There are no major technical flaws; only minor considerations are listed in the Questions section below.\n\nQUESTIONS:\nThere is some missing information and details:\n- What is the running time? How long does it take for inference? How long is the optimization time?\n- Neither the main paper nor the supplementary material includes the estimated ERP space scale map. It may be correct, since the downstream tasks and depth map metrics, but providing this along with the initial/final results (Figure 2 in supp.) may help readers to understand what the updates are.\n- When merging cube maps to the ERP image again, how is the interpolation handled? There may be possibilities for choosing an improper sampling method for such remapping, which can lead to artifacts in edge cases.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}