Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation

Authors: ning-hsu  wang
yu-lun  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 09:33:18

Abstract: Accurately estimating depth in 360-degree imagery is crucial for virtual reality, autonomous navigation, and immersive media applications. Existing depth estimation methods designed for perspective-view imagery fail when applied to 360-degree images due to different camera projections and distortions. We propose a new depth estimation framework that uses unlabeled 360-degree data effectively. Our approach uses state-of-the-art perspective depth estimation models as teacher models to generate pseudo labels through a six-face cube projection technique, enabling efficient labeling of depth in 360-degree images. This method leverages the increasing availability of large datasets. It includes two main stages: offline mask generation for invalid regions and an online semi-supervised joint training regime. We tested our approach on benchmark datasets such as Matterport3D and Stanford2D3D, showing significant improvements in depth estimation accuracy, particularly in zero-shot scenarios. Our proposed training pipeline can enhance any 360 monocular depth estimator and demonstrate effective knowledge transfer across different camera projections and data types.

Main paper: 
 introduction: 
 In recent years, the field of computer vision has seen a surge in research focused on addressing the challenges associated with processing 360-degree images. The widespread use of panoramic imagery across various domains, such as virtual reality, autonomous navigation, and immersive media, has underscored the need for accurate depth estimation techniques tailored specifically for 360-degree images. However, existing depth estimation methods developed for perspective-view images encounter significant difficulties when applied directly to 360-degree data due to differences in camera projection and distortion. While many methods aim to address depth estimation for this camera projection, they often struggle due to the limited availability of labeled datasets.To overcome these challenges, this paper presents a novel approach for training state-of-the-art (SOTA) depth estimation models on 360-degree imagery. With the recent significant increase in the amount of available data, the importance of both data quantity and quality has become evident. Research efforts on perspective perceptual models have increasingly focused on augmenting the volume of data and developing foundation models that generalize across various types of data. Our method leverages SOTA perspective depth estimation foundation models as teacher models and generates pseudo labels for unlabeled 360-degree images using a six-face cube projection approach. By doing so, we efficiently address the challenge of labeling depth in 360-degree imagery by leveraging perspective models and large amounts of unlabeled data. This figure demonstrated the improvement of our proposed training pipeline tested on the Stan-ford2D3D [2] dataset in a zero-shot setting.Our approach consists of two key stages: offline mask generation and online joint training. During the offline stage, we employ a combination of detection and segmentation models to generate masks for invalid regions, such as sky and watermarks in unlabeled data. Subsequently, in the online stage, we adopt a semi-supervised learning strategy, loading half of the batch with labeled data and the other half with pseudo-labeled data. Through joint training with both labeled and pseudo-labeled data, our method achieves robust depth estimation performance on 360-degree imagery.To validate the effectiveness of our approach, we conduct extensive experiments on benchmark datasets such as Matterport3D and Stanford2D3D. Our method demonstrates significant improvements in depth estimation accuracy, particularly in zero-shot scenarios where models are trained on one dataset and evaluated on another. Furthermore, we demonstrate the efficacy of our training techniques with different SOTA 360-degree depth models and various unlabeled datasets, showcasing the versatility and effectiveness of our approach in addressing the unique challenges posed by 360-degree imagery.Our contributions can be summarized as follows:• We propose a novel training technique for 360-degree imagery that harnesses the power of unlabeled data through the distillation of perspective foundation models.• We introduce an online data augmentation method that effectively bridges knowledge distillation across different camera projections.• Our proposed training techniques significantly benefit and inspire future research on 360degree imagery by showcasing the interchangeability of state-of-the-art (SOTA) 360 models, perspective teacher models, and unlabeled datasets. This enables better results even as new SOTA techniques emerge in the future.2 Related Work 360 monocular depth. Depth estimation for 360-degree images presents unique challenges due to the equirectangular projection and inherent distortion. Various approaches have been explored to address these issues:• Directly Apply: Some methods directly apply monocular depth estimation techniques to 360-degree imagery. OmniDepth [69] leverages spherical geometry and incorporates Sph-Conv [42] to improve depth prediction with distortion. [70, 49] use spherical coordinates to overcome distortion with extra information. [12, 17] leverage other ground truth supervisions to assist on depth estimation. SliceNet [31] and ACDNet [68] propose advanced network architectures tailored for omnidirectional images. EGFormer [64] and HiMODE [18] introduce a transformer-based model that captures global context efficiently, while [45, 44] focuses on integrating geometric priors into the learning process. [13] proposed to generate large-scale datasets with SfM and MVS, then apply to test-time training. • Cube: Other approaches use cube map projections to mitigate distortion effects. 360-SelfNet [46] is the first work to self-supervised 360 depth estimation leveraging cubepadding [9] . BiFuse [47] and its improved version BiFuse++ [48] are two-branch architectures that utilize cube maps and equirectangular projections. UniFuse [16] combines equirectangular and cube map projections and simplifies the architecture. [3] combines two-branch techniques with transformer network. • Tangent Image: Tangent image projections are also popular. [37, 23, 30] convert equirectangular images into a series of tangent images, which are then processed using conventional depth estimation networks. PanoFormer [40] employs a transformer-based architecture to handle tangent images, while SphereNet [11] and HRDFuse [1] enhance depth prediction by collaboratively learning from multiple projections.360 other works Beyond depth estimation, 360-degree imagery has been applied to depth completion tasks as follows [26, 8, 32, 57, 58, 15] . Other methods, such as [25] and [43] focus on the projection between camera models, while the former projects pinhole camera model images into a large field of view, whereas the latter transforms convolution kernels.Unlabeled / Pseudo labeled data. Utilizing unlabeled or pseudo-labeled data has become a significant trend to mitigate the limitations of labeled data scarcity. Techniques like [22, 71, 41, 56] leverage large amounts of unlabeled data to improve model performance through semi-supervised learning. In the context of 360-degree depth estimation, our approach generates pseudo labels from pre-trained perspective models, which are then used to train 360-degree depth models effectively.Zero-shot methods. Zero-shot learning methods aim to generalize to new domains without additional training data. [7, 54] target this directly with increasing training data., MiDaS [35, 5, 34] and Depth Anything [59] are notable for their robust monocular depth estimation across diverse datasets leveraging affine-invariant loss. [61] takes a step further to investigate zero-shot on metric depth. Marigold [19] leverages diffusion models with image conditioning and up-to-scale relative depth denoising to generate detailed depth maps. ZoeDepth [4] further these advancements by incorporating scale awareness and domain adaptation. [14, 50] leverage camera model information to adapt cross-domain depth estimation.Foundation models. Foundation models have revolutionized various fields in AI, including natural language processing and image-text alignment. In computer vision, models like CLIP [33] demonstrate exceptional generalization capabilities. [28] proposed a foundation visual encoder for downstream tasks such as segmentation, detection, depth estimation, etc. [20] proposed a model that can cut out masks for any objects. Our work leverages a pre-trained perspective depth estimation foundation model [59] as a teacher model to generate pseudo labels for 360-degree images, enhancing depth estimation by utilizing the vast knowledge embedded in these foundation models.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Yes, the main claims presented in the abstract and introduction accurately portray the contributions and scope of our paper. As evidenced by Figure 2 and Table 3 , our work indeed introduces a training pipeline that demonstrates improvements in the zero-shot testing, making it a significant contribution. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: As described in Sec. 5, our work relies heavily on the quality of the data and the accuracy of the pseudo labels. We have also highlighted that disregarding or neglecting data cleaning procedures may result in undesirable training outcomes. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 In this work, we propose a novel training approach for 360-degree monocular depth estimation models. Our method leverages a perspective depth estimation model as a teacher and generates pseudo labels for unlabeled 360-degree images using a 6-face cube projection. Figure 2 illustrates our training pipeline, incorporating the use of Segment Anything to mask out sky and watermark regions in unlabeled data during the offline stage. Subsequently, we conduct joint training using both labeled and unlabeled data, allocating half of the batch to each. The joint training avoids limiting performance by teacher model. The unlabeled data is supervised using pseudo labels generated by Depth Anything, a state-of-the-art perspective monocular depth foundation model. With the benefit of our teacher model, the 360-degree depth model demonstrates an observable improvement on the zero-shot dataset, as shown in Figure 1 .Dataset statistics. 360-degree data has become increasingly available in recent years. However, compared to perspective-view depth datasets, labeling depth ground truths for 360-degree data presents greater challenges. Consequently, the availability of labeled datasets for 360-degree data is considerably smaller than that of perspective datasets. In this paper, we use Depth Anything [59] to generate pseudo ground truth for training. However, more advanced techniques could be applied. These perspective-view monocular depth estimators fail to produce reasonable equirectangular depth as a domain gap exists. Therefore, we distill knowledge by inferring six perspective cube faces and passing them through perspective-view monocular depth estimators.To ensure stable and effective training, we propose generating a valid pixel mask with Segment Anything [20] while calculating loss. (c) Furthermore, we augment random rotation on RGB before passing it into Depth Anything, as well as on predictions from the 360 depth model. 1 presents the data quantities available in some of the most popular 360-degree datasets, including Matterport3D [6] , Stanford2D3D [2] , and Structured3D [65] . Additionally, we list a multi-modal dataset, SpatialAudioGen [29] , which consists of unlabeled 360-degree data used in our experiments. Notably, the amount of labeled and unlabeled data used in the perspective foundation model, Depth Anything [59] , is significantly larger, with 1.5 million labeled images [24, 52, 10, 60, 55, 51] and 62 million unlabeled images [38, 63, 53, 62, 39, 21, 66, 20] , making the amount in 360-degree datasets approximately 170 times smaller.Data cleaning and valid pixel mask generation Unlabeled data often contains invalid pixels in regions such as the sky and watermark, leading to unstable training or undesired convergence. To address this issue, we applied the GroundingSAM [36] method to mask out the invalid regions. This approach utilizes Grounded DINOv2 [27] to detect problematic regions and applies the Segment Anything [20] model to mask out the invalid pixels by segmenting within the bounding box. While Depth Anything [59] also employs a pre-trained segmentation model, DINOv2, to select sky regions. Brand logos and watermarks frequently appear after fisheye camera stitching. Therefore, additional labels are applied to enhance the robustness of our training process. We also remove all images with less than 20 percent of valid pixels to stablize our training progress.Perspective foundation models (teacher models). To tackle the challenges posed by limited data and labeling difficulties in 360-degree datasets, we leverage a large amount of unlabeled data alongside state-of-the-art perspective depth foundation models. Due to significant differences in camera projection and distortion, directly applying perspective models to 360-degree data often yields inferior results. Previous works have explored various methods of projection for converting equirectangular to perspective depth, as stated in Section 2. Among these, cube projection and tangent Figure 3 : Valid Pixel Masking. We used Grounded-Segment-Anything [36] to mask out invalid pixels based on two text prompts: "sky" and "watermark." These regions lack depth sensor ground truth labels in all previous datasets. Unlike Depth Anything [59] , which sets sky regions as 0 disparity, we follow ground truth training to ignore these regions during training for two reasons:(1) segmentation may misclassify and set other regions as zero, leading to noisy labeling, and (2) watermarks are post-processing regions that lack geometrical meaning.projection are the most common techniques. We selected cube projection to ensure a larger field of view for each patch, enabling better observation of relative distances between pixels or objects during the inference of the perspective foundation model and enhancing knowledge distillation. The comparison table can be find in the supplementary material.In our approach, we apply projection to unlabeled 360-degree data and then run Depth Anything on these projected patches of perspective images to generate pseudo-labels. We explore two directions for pseudo-label supervision: projecting the patch to equirectangular and computing in the 360-degree domain or projecting the 360-degree depth output from the 360 model to patches and computing in the perspective domain. Since training is conducted in an up-to-scale relative depth manner, stitching the patch of perspective images back to equirectangular with an aligned scale will lead to failure in training Figure 4 , which is an additional research topic that is worth investigation. We opt to compute the loss in the perspective domain, facilitating faster and easier training without the need for additional alignment optimization.Directly applying Depth Anything on cube-projected unlabeled data does not yield improvements due to ignorance of cross-cube-face relation, leading to cube artifacts (Figure 5 ). This issue arises from the separate estimation of perspective cube faces, where monocular depth is estimated based on semantic information, lacking a comprehensive understanding of the entire scene. To address this, we propose a random rotation preprocessing step in front of the perspective foundation model.As depicted in Figure 2 , the rotation is applied to equirectangular projection RGB images using a random rotation matrix, followed by cube projection. This results in a more diverse set of cube faces, capturing relative distances between ceilings, walls, windows, and other objects more effectively.With the proposed random rotation technique, knowledge distillation becomes more comprehensive as the point of view is not static. The inference by the perspective foundation model is performed on the fly, with parameters frozen during the training of the 360 model.In order to perform random rotation, we apply a rotation matrix on the equirectangular coordinates, noted as (θ, ϕ), and rotation matrix as R.EQUATIONFor equirectangular to cube projection, the field-of-view (FoV) of each cube face is equal to 90 degrees; each face can be considered as a perspective camera whose focal length is w/2, and all faces share the same center point in the world coordinate. Since the six cube faces share the same center point, the extrinsic matrix of each camera can be defined by a rotation matrix R i . p is then the pixel on the cube faceEQUATIONwhere,EQUATIONwhere θ and ϕ are longitude and latitude in equirectangular projection and q is the position in Euclidean space coordinates.The training process closely resembles that of MiDaS, Depth Anything, and other cross-dataset methods. Our goal is to provide depth estimation for any 360-degree images. Following previous approaches that trained on multiple datasets, our training objective is to estimate relative depth. The depth values are first transformed into disparity space using the formula 1/d and then normalized to the range [0, 1] for each disparity map.To adapt to cross-dataset training and pseudo ground truths from the foundation model, we employed the affine-invariant loss, consistent with prior cross-dataset methodologies. This loss function disregards absolute scale and shifts for each domain, allowing for effective adaptation across different Ignoring cross-cube relationships results in poor knowledge distillation. To address this, as shown in Figure 2 (c), we randomly rotate the RGB image before inputting it into Depth Anything. This enables better distillation of depth information from varying perspectives within the equirectangular image.datasets and models.EQUATIONwhere d * i and d i are the prediction and ground truth, respectively. ρ represents the affine-invariant mean absolute error loss:EQUATIONHere, di and d * i are the scaled and shifted versions of the prediction d * i and ground truth d i :EQUATIONwhere t(d) and s(d) are used to align the prediction and ground truth to have zero translation and unit scale:EQUATION4 ExperimentsThese notations apply for all tables: M: Matterport3D [6] , SF: Stanford2D3D [2] , ST: Struc-tured3D [65] , SP: Spatialaudiogen [29] , -all indicates using the entire train, validation, and test sets of the specific dataset, and (p) denotes using pseudo ground truth generated by Depth Anything [59] . Due to space limits, we provide the experimental setup in the appendix, including implementation details and evaluation metrics.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: As indicated in Section 4.3 and Table 3 , our train pipeline numerically improves the existing 360 methods. We've also demostrated the effectiveness of pseudo ground truth on Structured3D in Table 4 .• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
baselines: 
 Recent state-of-the-art methods [1, 64, 47, 48, 16, 31, 40] have emerged. We chose UniFuse and BiFuse++ as our baseline models for experiments, as many of the aforementioned methods did not fully release pre-trained models or provide training code and implementation details. It's worth noting that PanoFormer [40] is not included due to incorrect evaluation code and results. Both selected models are re-implemented with affine-invariant loss on disparity for a fair comparison and to demonstrate improvement. We conduct experiments on the Matterport3D [6] benchmark to demonstrate performance gains within the same dataset/domain, and we perform zero-shot evaluation on the Stanford2D3D [2] test set to demonstrate the generalization capability of our proposed training technique. To further validate its robustness, we evaluate additional baseline models [45, 64] in zero-shot setting, showcasing the effectiveness of our approach for non-dual-projection models. UniFuse [16] Affine-Inv M M 0.102 0.893 0.970 0.989 UniFuse [16] Affine-Inv M, ST-all (p) M 0.089 0.911 0.975 0.991 BiFuse++ [48] Affine-Inv M M 0.094 0.914 0.974 0.989 BiFuse++ [48] Affine-Inv M, ST-all (p) M 0.085 0.917 0.976 0.991We conducted our in-domain improvement experiment on the widely used 360-degree depth benchmark, Matterport3D [6] , to showcase the results of perspective foundation model distillation on the two selected baseline models, UniFuse [16] and BiFuse++ [48] . In Table 2 , we list the metric depth evaluation results from state-of-the-art methods on this benchmark. Subsequently, we present the re-trained baseline models using affine-invariant loss on disparity to ensure a fair comparison with their original depth metric training. Finally, we demonstrate the improvement achieved with results trained on the labeled Matterport3D training set and the entire Structured3D dataset with pseudo ground truth.Our goal is to estimate depths for all 360-degree images, making zero-shot performance crucial. Following previous works [47, 16] , we adopted their zero-shot comparison setting, where models trained on the entire Matterport3D [6] dataset are tested on the Stanford2D3D [2] test set. In Table 3 , the upper section lists methods trained with metric depth ground truth, with numbers sourced from their respective papers. The lower section includes models trained with affine-invariant loss on disparity ground truth. As shown in Figure 6 , [16, 48] demonstrate generalization improvements with a lower error on the Stanford2D3D dataset.Depth Anything [59] and Marigold [19] are state-of-the-art zero-shot depth models trained with perspective depths. As shown in Table 3 , due to the domain gap and different camera projections, foundation models trained with perspective depth cannot be directly applied to 360-degree images. We demonstrated the zero-shot improvement on UniFuse [16] , BiFuse++ [48] and non-dual-projection methods [45, 64] with models trained on the entire Matterport3D [6] dataset with ground truth and the entire Structured3D [65] or SpatialAudioGen [29] dataset with pseudo ground truth generated using Depth Anything [59] .As Structured3D provides ground truth labels for its dataset, we also evaluate our models on its test set to assess how well they perform with pseudo labels. Table 4 shows the improvements achieved on the Structured3D test set when using models trained with pseudo labels. It's worth noting that even when the 360 model is trained on pseudo labels from SpatialAudioGen, it performs similarly well. This demonstrates the success of our distillation technique and the model's ability to generalize across different datasets.We demonstrated the qualitative results in Figure 8 and Figure 7 360-degree images that were either captured by us or downloaded from the internet 1 . These examples showcase the zero-shot capability of our model when applied to data outside the aforementioned 360-degree datasets. Affine-Inv Pers. SF 0.195 0.692 0.942 0.982 UniFuse [16] Affine-Inv M-all SF 0.090 0.914 0.976 0.990 UniFuse [16] Affine-Inv M-all, ST-all (p) SF 0.086 0.924 0.977 0.990 UniFuse [16] Affine-Inv M-all, SP-all (p) SF 0.090 0.920 0.978 0.990 BiFuse++ [48] Affine-Inv M-all SF 0.090 0.921 0.976 0.990 BiFuse++ [48] Affine-Inv M-all, ST-all (p) SF 0.082 0.931 0.979 0.991 BiFuse++ [48] Affine-Inv M-all, SP-all (p) SF 0.086 0.926 0.979 0.991 HoHoNet [45] Affine-Inv M-all SF 0.095 0.906 0.975 0.991 HoHoNet [45] Affine-Inv M-all, ST-all (p) SF 0.088 0.920 0.979 0.992 EGFormer [64] Affine-Inv M-all SF 0.098 0.906 0.972 0.989 EGFormer [64] Affine We use our pre-trained model as an initial weight and fine-tune on Stanford2D3D [2] metric depth to demonstrate the effectiveness of our pre-trained relative depth model's ability to adapt to metric depth with a single epoch in Table 5 5 ConclusionOur proposed method significantly advances 360-degree monocular depth estimation by leveraging perspective models for pseudo-label generation on unlabeled data. The use of cube projection with random rotation and affine-invariant loss ensures robust training and improved depth prediction accuracy while bridging the domain gap between perspective and equirectangular projection. By effectively addressing the challenges of limited labeled data with cross-domain distillation, our approach opens new possibilities for accurate depth estimation in 360 imagery. This work lays the groundwork for future research and applications, offering a promising direction for further advancements in 360-degree depth estimation.Limitations. Our work faces limitations due to its heavy reliance on the quality of unlabeled data and pseudo labels from perspective foundation models. The results are significantly impacted by data quality (Section 3.1). Without data cleaning, the training process resulted in NaN values. Another limitation is that although with unlabeled data, the scarcity of data still exists compared to other tasks.A.1 Experimental Setup Implementation details. Our work is divided into two stages: (1) offline mask generation and (2) online joint training. (1) In the first stage, we use Grounded-Segment-Anything [36] , which combines state-of-the-art detection and segmentation models. We set the BOX_THRESHOLD and TEXT_THRESHOLD to 0.3 and 0.25, respectively, following the recommendations of the official code. We use "sky" and "watermark" as text prompts. All pixels with these labels are set to False to form our valid mask for the second stage of training. (2) In the second stage, each batch consists of an equal mix of labeled and unlabeled data. We follow the backbone model's official settings for batch size, learning rate, optimizer, augmentation, and other hyperparameters, changing only the loss function to affine-invariant loss. Unlike Depth Anything, which sets invalid sky regions to zero disparity, we ignore these invalid pixels during loss calculation, consistent with ground truth training settings. We average the loss for ground truth and pseudo ground truth during updates. All our experiments are conducted on a single RTX 4090, both offline and online. However, if future 360-degree state-of-the-art methods or perspective foundation models require higher VRAM usage, the computational resource requirements may increase.In line with previous cross-dataset works, all evaluation metrics are presented in percentage terms. The primary metric is Absolute Mean Relative Error (AbsRel), calculated as:1 M M i=1 |a i -d i |/d i ,where M is the total number of pixels, a i is the predicted depth, and d i is the ground truth depth. The second metric, δ j accuracy, measures the proportion of pixels where max(ai/di, di/ai) is within 1.25 j . During evaluations, we follow [16, 47, 48] to ignore areas where ground truth depth values are larger than 10 or equal to 0. Given the ambiguous scale of self-training results, we apply median alignment after converting disparity output to depth before evaluation, as per the method used in [67] :EQUATIONwhere d is the predicted depth from inverse disparity and d is the ground truth depth. This ensures a fair comparison by aligning the median depth values of predictions and ground truths.We demonstrate additional zero-shot qualitative results in Figure 9 and in-the-wild results in Figure 12 .In-domain results on the Matterport3D test sets are showcased in Figure 10 and Figure 11 .As described in Sec.3.1 of the main paper, there is a significant difference in the number of images between the perspective and equirectangular datasets. Detailed statistics of the datasets are listed in Table 6 .Unlike many previous knowledge distillation approaches that use a higher proportion of pseudo labels during model training, we opt for an equal ratio of ground truth to pseudo labels. Through an ablation study exploring the relationship between this data ratio and model performance, we observe a robust improvement starting from 1 : 1 in Table 7 .There are various perspective camera projections for panoramic imagery, with cube and tangent image projections being the most common, both widely used in previous works. In Table 8 , we compare these two projections and observe similar improvements when applying our proposed training pipeline, demonstrating the effectiveness of our method. For robust knowledge distillation in relative depth estimation, we select the cube projection due to its wider field of view coverage. CVRP 2020 115 K ✓ SpatialAudioGen [29] NeurIPS 2018 344K -HRWSI [55] CVPR 2020 20K ✓ IRS [51] ICME 2021 103K ✓ ImageNet-21K [38] IJCV 2015 13.1M -BDD100K [63] CVPR 2020 8.2M -Google Landmarks [53] CVPR 2020 4.1M -LSUN [62] arXiv 2015 9.8M -Objects365 [39] ICCV 2019 1.7M -Open Images V7 [21] IJCV 2020 7.8M -Places365 [66] TPAMI 2017 6.5M -SA-1B [20] ICCV 2023 11.1M - Figure 12 : Additional in-the-wild results. We compare our proposed joint-training method (Matter-port3D (GT) + SpatialAudioGen (Pseudo)) with a model trained only on the Matterport3D dataset, using data randomly downloaded from the internet. This comparison demonstrates the significant improvement of our method along with its generalization ability and effectiveness on real-world data.NeurIPS Paper ChecklistQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We have listed all data cleaning preprocessing steps in Sec.3.1, and the implementation details are described in Sec.A.1. This ensures transparency and enables reproducibility of our main experimental results.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: While the code for the work will be released upon acceptance, we have chosen not to submit it during the reviewing stage. However, as stated in the previous question, we have provided all the necessary information in the paper for reproducibility of the main experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: As specified in Sec. A.1, our training process follows the official implementation of [16, 48] , employing an affine-invariant loss and a weighted average between loss from ground truth labels and pseudo labels.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The error and metric calculations listed in Tables 2, 3 , and 4 all adhere to prior works conducted on affine-invariant relative depth on disparity, as described in Sec. A. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: As we propose a new training process to bridge the knowledge distillation between camera models, this work does not focus on single model development or dataset collection. Therefore, there are no safeguard issues associated with the release of data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Stig Nygaard, https://www.flickr.com/photos/stignygaard/49659694937, CC BY2.0 DEED Dominic Alves, https://www.flickr.com/photos/dominicspics/28296671029/, CC BY 2.0 DEED Luca Biada, https://www.flickr.com/photos/pedroscreamerovsky/6873256488/, CC BY 2.0 DEED Luca Biada, https://www.flickr.com/photos/pedroscreamerovsky/6798474782/, CC BY 2.0 DEED 
 
 Appendix:  


Decision: The paper proposes a simple yet effective general-purpose framework to deal with training data scarcity in 360-degree depth estimation by exploiting the knowledge of a SOTA pre-trained conventional (perspective) monocular depth estimation network. Reviewers acknowledged the proposal's relevance but raised concerns, especially regarding novelty and the experimental evaluation with only two networks, and requested several clarifications. In the rebuttal/discussion, the authors provided satisfactory responses and convincing experimental results on two additional networks (HoHoNet and EGformer), leading all reviewers towards acceptance. Although some key concepts were already proposed for another task (stereo), as noticed by reviewer 3Nfa, the AC finds the overall contribution valuable for the monocular setup and recommends acceptance. However, compared to the original submission, the paper needs a significant revision to include additional experiments and clarifications that emerged after the rebuttal.

Reviews: review: paper_summary: paper_summary: The authors present a training strategy for single-image depth estimation on 360-degree equirectangular images. The strategy centers around leveraging strong pre-trained models for perspective images as teacher networks. It does not depend on any particular network architecture and therefore can benefit any 360 depth estimation networks. Experimental results validate that the strategy improves models otherwise trained only with limited ground-truth annotations.
strength_weakness: strength_weakness: 1. A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, and avoid generic remarks. 

Despite focusing on a slightly different task (stereo depth), FoVA-Depth (Lichy et al. 3DV 2024) presents a few very similar concepts: 
  - leveraging abundance of perspective depth GT
  - cube map as a intermediate representation to gap between 360 and perspective images
  - random rotation augmentation
It is worth some discussion regarding similarities and differences. 

2. A very simple yet critical baseline is missing: directly project pseudo GT on cubemap to equirectangular images. A good stitching strategy may be challenging, but with something simple or even without any additional scaling, it should help clarify how much the pre-trained depth anything model contribute to the overall performance. 

3. The paper addresses only relative depth estimation. Since several baselines (e.g. upper section of Tab.2) already have metric counterparts, and depth anything has metric variants, I feel some experiments and analysis in that regards should be straightforward and nicely complement the relative depth results. 

4. As the premise of the work is the usefulness of the abundant pseudo GT compard to limited 360 depth GT, it is necessary to understand how does the benefit from pseudo GT scale (how many pseudo GT can be as useful as a real 360 GT?), and how does the student network compare to the teacher network in terms of estimation quality (is the quality of teacher network already a bottleneck?). The paper offers little insight in these questions.
questions: questions: I am looking forward to answer to the questions raised above, namely:
- How does training scale in terms of the amount of pseudo GT vs real GT? 
- Is the approach applicable to metric depth estimation? 
- Additional related work and baseline as described above.
limitations: limitations: The only limitation the authors bring up (quality of unlabelled data) already has a solution in the paper, so not really a limitation? 

I do think there are a few other things worth mentioning: 
- 360 data, even without requiring GT, is still scarce compared to perspective data. The fact the authors can only evaluate with two such datasets is an evidance. This is a limitation since it prevents further scaling up training. 
- Only equirectangular images are supported (though it seems that, in principle, the approach should work in more general camera models).
paper_summary: paper_summary: This paper introduces a novel depth estimation framework specifically designed for 360-degree data using an innovative two-stage process: offline mask generation and online semi-supervised joint training. Initially, invalid regions such as sky and watermarks are masked using detection and segmentation models. The method then employs a semi-supervised learning approach, blending labeled and pseudo-labeled data derived from state-of-the-art perspective depth models using a cube projection technique for effective training. This framework demonstrates adaptability across different state-of-the-art models and datasets, effectively tackling the challenges of depth estimation in 360-degree imagery.
strength_weakness: strength_weakness: 1. The proposed method is straightforward and the performance gains provided by the proposed method are described as marginal.
2. The method's effectiveness is demonstrated only with specific models, UniFuse and BiFuse++, limiting evidence of its broader applicability.
3. Inconsistencies in the decimal points used in quantitative results tables make direct performance comparisons challenging.
questions: questions: 1. Is there any reason the author only applied the proposed method to the UniFuse and BiFuse++?
2. There seems to be a discrepancy in the reported Absolute Relative (Abs Rel) error for BiFuse++ (Affine-Inv, M-all, SP-all(p)) at 0.858, which is ten times higher than results from competitive methods, while other metrics (\delta_1, \delta_2, \delta_3) align closely. Could this be an error?
limitations: limitations: See the weakness and the question parts.
paper_summary: paper_summary: This paper effectively utilizes unlabeled data by employing the SAM and DepthAnything models to generate masks and pseudo-labels respectively. When projecting data onto a cube, the authors use random rotation techniques to minimize cube artifacts, thereby enhancing the accuracy of 360-degree monocular depth estimation. Additionally, the method was tested in zero-shot scenarios, demonstrating its effective knowledge transfer.
strength_weakness: strength_weakness: - The paper stated that "As depicted in Figure 2, the rotation is applied to the equirectangular projection RGB images using a random rotation matrix, followed by cube projection. This results in a more diverse set of cube faces, effectively capturing the relative distances between ceilings, walls, windows, and other objects." However, from observing Figure 2, the unlabeled data seems to be entirely outdoor panoramas, which makes the mention of indoor elements such as ceilings, walls, and windows confusing. If there are indeed indoor panoramic images in the unlabeled data, what elements might the SAM model need to segment in such indoor panoramas? The description in the paper appears somewhat unclear.
- The paper mentioned, "We chose UniFuse and BiFuse++ as our baseline models for experiments, as many of the aforementioned methods did not release pre-trained models or provide training code and implementation details." However, methods such as HRDfuse [1], EGFormer [50], BiFuse and BiFuse++ [35, 36], UniFuse [11], and PanoFormer [29] have all made their source codes available, making this reason in the paper seem insufficient. Additionally, "EGFormer is not included since its experiments are mainly conducted on other datasets and benchmarks" appears to be an inadequate reason for not including it in the experiments.
- In Table 3, despite introducing more unlabeled data on the Unifuse training set, i.e., SP-all (p), the performance of the method is not significantly improved. This phenomenon seems to be only effective on the BiFuse model, but not on other methods.
- In Table 3, there is a typographical error in the recording of the Abs Rel value; it should not be 0.858. Additionally, in Table 2, "UniFuse" is incorrectly written as "UniFise."
questions: questions: - The paper mentioned, "Subsequently, in the online stage, we adopt a semi-supervised learning strategy, loading half of the batch with labeled data and the other half with pseudo-labeled data." However, common semi-supervised strategies typically set the ratio of labeled to total data at 1/2, 1/4, 1/8, 1/16, etc. The experiments in the paper were only conducted at a 1:1 ratio (labeled: unlabeled), and thus the performance at other ratios remains unknown.
- The third contribution mentioned in the paper refers to "interchangeability" and "This enables better results even as new SOTA techniques emerge in the future." This suggests that the strategy might be adaptable to a variety of models. However, the experiments were conducted only on models like UniFuse and BiFuse++ which use a dual projection fusion of Cube and ERP. Whether this approach would perform well with other transformer-based models remains an unresolved question.
limitations: limitations: - The paper stated, "Cube projection and tangent projection are the most common techniques. We selected cube projection to ensure a larger field of view for each patch." This is merely a theoretical assertion, with no experimental evidence to prove which projection method is superior. Additionally, using cube projection directly can lead to cube artifacts. Following the suggestion in [1], setting each panoramic image to have 10 or 18 tangent images using more polyhedral faces, rather than the standard six faces, might reduce the artifacts caused by direct cube projection.
[1]  Cokelek, M., Imamoglu, N., Ozcinar, C., Erdem, E. and Erdem, A., 2023. Spherical Vision Transformer for 360-degree Video Saliency Prediction. BMVC 2023.
paper_summary: paper_summary: This paper proposes a method to improve 360 monocular depth estimation using perspective distillation and augmentation with unlabeled data. It introduces the concept of "perspective distillation," which leverages the available 360 monocular depth maps and their corresponding equirectangular images to generate pixel-wise depth supervision signals. This technique helps to address the lack of ground truth depth data for training in the 360 domain. Additionally, the paper presents an unlabeled data augmentation approach that utilizes the geometric properties of 360-degree images. By exploiting the spherical geometry, the authors generate synthetic stereo pairs to augment the training dataset without requiring paired depth information. The proposed method is evaluated on benchmark datasets and achieves significant improvements in terms of depth estimation accuracy compared to existing approaches. The results demonstrate the effectiveness of perspective distillation and unlabeled data augmentation in enhancing the performance of 360 monocular depth estimation.
strength_weakness: strength_weakness: 1. Complexity: The proposed method introduces additional complexity, such as perspective distillation and synthetic stereo pair generation, which may require more computational resources and training time.

2. Dataset dependency: The effectiveness of the proposed method heavily relies on the availability and quality of the benchmark datasets used for evaluation, which may affect its generalizability to real-world scenarios.

3. Limited scope: The paper focuses specifically on 360 monocular depth estimation, which may limit its applicability to other depth estimation tasks or domains, such as 360 monocular depth completion. 

4. Insufficient related work: Adding the latest panoramic depth estimation and panoramic depth completion methods is preferred.
questions: questions: Overall, the idea of this paper is interesting, allowing existing depth estimation models to benefit from unlabeled data in a semi-supervised manner, but there are concerns:

1. Based on the experimental results, the author only validated the performance of this training technique on models that employ dual-projection fusion, such as Unifuse and Bifuse. There was no effective validation or analysis provided for other non-dual-projection fusion models, such as methods based on horizontal compression (e.g., HorizonNet, HohoNet) or transformer-based methods (e.g., EGFormer). However, it is essential to explicitly state the scope of applicability for this training strategy. Different training strategies, data processing methods, and even device variations can lead to unfair comparisons. The author should conduct fair experiments and comparisons under the same experimental conditions instead of directly copying the data results from the original paper. This is crucial for validating the effectiveness of this training strategy. Therefore, the author's mention of "as many of the aforementioned methods did not release pre-trained models or provide training code and implementation details. It’s worth noting that PanoFormer [29] is not included due to incorrect evaluation code and results, and EGFormer [29] is not included since its experiments are mainly conducted on other datasets and benchmarks" is not convincing.

2. The issue of cross-domain experiments is indeed important. Given the challenges of obtaining real-world data, it would be meaningful if this framework could benefit models trained on synthetic data (e.g., training on synthetic data and testing on real data). However, the significance of this training strategy in that regard is not yet clear. Further research and experimentation are necessary to determine whether training on synthetic data using this framework can indeed yield benefits when applied to real-world data. It would be valuable to investigate the effectiveness and generalization capabilities of the trained models in real-world scenarios.

3. Median alignment is currently not widely utilized in existing depth estimation methods. Based on existing findings, aligning the ground truth (GT) depth with the predicted depth can indeed lead to improved results. However, when comparing the proposed method with baseline methods like Unifuse, which explicitly state that median alignment is not used, directly comparing the results to those from the original paper leads to unfair comparisons. This raises concerns about the advantages claimed for this training framework. To ensure fair comparisons, it is important to apply the same alignment techniques consistently across all methods being compared.
limitations: limitations: This paper proposes an interesting solution for panoramic depth estimation task, ie.e,  perspective distillation and unlabeled data augmentation. It could contribute a lot for the community. The reviewer suggests introducing more related works, including the latest panoramic depth estimation and panoramic depth completion approaches.
paper_summary: paper_summary: The authors propose a technique that utilizes unlabeled 360-degree data to improve previous methods, which includes two main stages: offline mask generation for invalid regions and an online semi-supervised joint training regime.  Experimental results indicate that the proposed method outperforms previous methods on the Matterport3D and Stanford2D3D datasets.
strength_weakness: strength_weakness: My major concern is that the technical contribution seems limited. The core idea of utilizing unlabeled data has been proposed by DepthAnything. The proposed method is more like an application of this idea to the 360 data;

As shown in Tables 2 and 3, previous 360 depth estimation methods typically use the metric loss for training. While the affine-invariant approach employed by the authors enables training on multiple datasets, it also impedes real-world applications due to the lose of metric scale.
questions: questions: Please see the weakness.
limitations: limitations: N/A

Comments: title: Please let us know if you have additional questions after reading our response
Please let us know if you have additional questions after reading our response
Feedback to authors
Feedback
Please let us know if you have additional questions after reading our response
Discussion about Submission 2414
comment: Thanks for the response. I tend to maintain the initial rating, since the current version of this paper needs to solve many weaknesses. But I am still willing to give a **borderline accept** score.
Thank you for your valuable review and feedback, which have significantly improved the completeness of our paper. We will ensure that the updated clarifications and experiments are incorporated into the final version.
Dear Reviewer,

As the discussion period ends soon, have we addressed all your concerns? If any issues remain, please inform us promptly. We appreciate your help in improving our manuscript.

Thank you!
Thanks for the rebuttal, which addressed some of my concerns. I would like to increase the rating. The updated clarification and experiments are expected to be presented in the final version.
Dear Reviewer ewNL,

We have listed the latest **panoramic depth estimation** and **completion** methods and will discuss them in the related work section of the final version. 

Could you please confirm that we did not miss any essential references and addressed all your concerns?

Thank you!
Dear Reviewer AagT,

We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further questions after reading our rebuttal.

We aim to address all the potential issues during the discussion period.

Thank you!

Best, Authors
Dear Reviewer,

As we approach the end of the discussion period, we want to confirm whether we have successfully addressed your concerns. Should any lingering issues require further attention, please let us know as early as possible so we can answer them soon.

We appreciate your time and effort in enhancing the quality of our manuscript.

Thank you!
Thank you for your constructive review, which has significantly contributed to the improvement of our paper. We will ensure that the materials from the rebuttal are incorporated into the final version.
Thank you for the responses! I think these are all valuable details and hopefully much of them will be integrated into the paper. I don't have further questions.
Dear Reviewers AagT and Reviewer 3NfA,

The authors replied to the questions raised in your initial evaluation report. 
Did the author address your concerns? Please post your feedback about the rebuttal.

Thank you
Dear reviewer Kp4U,

You raised concerns about the paper's technical novelty and contribution, and the authors responded in detail. 
Please share your feedback with us.

Thank you
Dear Reviewers,

We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further questions after reading our rebuttal.

We aim to address all the potential issues during the discussion period.

Thank you!

Best, Authors
Thank you for your constructive review and valuable feedback. Your insights have been instrumental in enhancing the quality of our paper.
Thank you for providing the additional experiments. The experiments for the other baseline are quite convincing, and I have increased my initial rating accordingly.
Dear reviewers,

please share your opinion about the rebuttal; it provides a detailed response to the questions raised in your reviews, including issues regarding novelty and experimental evaluation.

Thank you
AC
Thank you for your suggestion. In addition to those already cited in the original paper, we will add the following papers to our related work discussion.

**Panoramic Depth Estimation**
- $ 𝑆^2 $ Net: Accurate Panorama Depth Estimation on Spherical Surface
- High-Resolution Depth Estimation for 360-degree Panoramas through Perspective and Panoramic Depth Images Registration	
- Adversarial Mixture Density Network and Uncertainty-based Joint Learning for 360 Monocular Depth Estimation
- Learning high-quality depth map from 360 multi-exposure imagery
- Distortion-Aware Self-Supervised 360 Depth Estimation from A Single Equirectangular Projection Image
- SphereDepth: Panorama Depth Estimation from Spherical Domain
- 360 Depth Estimation in the Wild -- the Depth360 Dataset and the SegFuse Network
- GLPanoDepth: Global-to-Local Panoramic Depth Estimation
- Neural Contourlet Network for Monocular 360 Depth Estimation
- HiMODE: A Hybrid Monocular Omnidirectional Depth Estimation Model
- Geometric Structure Based and Regularized Depth Estimation From 360 Indoor Imagery
- Deep Depth Estimation on 360 Images with a Double Quaternion Loss

**Panoramic Depth Completion**
- Cross-Modal 360° Depth Completion and Reconstruction for Large-Scale Indoor Environment
- 360 ORB-SLAM: A Visual SLAM System for Panoramic Images with Depth Completion Network
- Deep panoramic depth prediction and completion for indoor scenes
- Multi-Modal Masked Pre-Training for Monocular Panoramic Depth Completion
- Distortion and Uncertainty Aware Loss for Panoramic Depth Completion
- Indoor Depth Completion with Boundary Consistency and Self-Attention

 Please let us know if any specific paper is missing.
Thanks for the responses. As mentioned in weakness 4, more related works should be involved in the Related Work section, including the latest **panoramic depth estimation** and **panoramic depth completion** methods.