Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation
Authors: Ning-Hsu Wang
Yu-Lun Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Monocular depth estimation
Knowledge distillation
Unlabeled data augmentation
Primary area: machine_vision

Abstract: Accurately estimating depth in 360-degree imagery is crucial for virtual reality, autonomous navigation, and immersive media applications. Existing depth estimation methods designed for perspective-view imagery fail when applied to 360-degree images due to different camera projections and distortions. We propose a new depth estimation framework that uses unlabeled 360-degree data effectively. Our approach uses state-of-the-art perspective depth estimation models as teacher models to generate pseudo labels through a six-face cube projection technique, enabling efficient labeling of depth in 360-degree images. This method leverages the increasing availability of large datasets. It includes two main stages: offline mask generation for invalid regions and an online semi-supervised joint training regime. We tested our approach on benchmark datasets such as Matterport3D and Stanford2D3D, showing significant improvements in depth estimation accuracy, particularly in zero-shot scenarios. Our proposed training pipeline can enhance any 360 monocular depth estimator and demonstrate effective knowledge transfer across different camera projections and data types.

Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation
Ning-Hsu Wang albert a.bert..nwangeeeeeee.......... www.nngwgwawaa.ccommmm com
Yu-Lun Liu
Department of Computer Science National Yang Ming Chiao Tung University yulunliu@cs, nyeu. edu.
Abstract
Accurately estimating depth in 360-degree imagery is erucial for virtual reality, au- tonomous navigation, and immersive media applications. Existing depth estimation methods designed for perspective-vies images due to different camera projections and distortions, whereas 360-degree methods perform inferior due to the lack of labeled data pairs. We propose a new depth estimation framework that utilizes unlabeled 360-degree data effectively. Our approach uses saate-of-the-an perspective depth estimation models as teacher models to generate pscudo labels through six- fix-face cube projection technique, enabling efficient labeling of depth in 360-degree images. This method leverages the increasing availability of large datasets. Our approach includes two main stages: offline mask generation for invalid regions and an online semi-supervised joint training regime. We tested our approach benchmark datasets such Matter- port3D and Stanford2D3D, showing significant improvements depth estimation accuracy, particularly in zero-shot scenarios. Our proposed training pipeline can enhance any 360 monocular depth estimator and demonstrates effective knowledge transfer across different camera projections and data types. See our project page for results: albert10012 w.githuu.io/ Dowpepph--nnnheeeeeeeeeeeeeeeeee Anywhere.
imagery fail when applied to 360-degree
Introduction
In recent years, the field of computer vision has surge in research focused addressing the challenges associated with processing 360-degree images. The widespread use of panoramic imagery across various domains, such virtual reality, autonomous navigation, and immersive media, has underscored the need for accurate depth estimation techniques tailored specifically for 360-degree images. However, existing depth estimation methods developed for perspective-view images encounter significant difficulties when applied directly to 360-degree data due to differences in camera projection and distortion. While many methods aim to address depth estimation for this camera projection, they often struggle due to the limited availability of labeled datasets.
To overcome these challenges, this paper presents novel approach for training vate-offthhearl (SOTA) depth estimation models on 360-degree imagery. With the recent significant increase in the amount of available data, the importance of both data quantity and quality has become evident. Research efforts on perspective perceptual models have increasingly focused on augmenting the volume of data and developing foundation models that generalize across various types of data. Our method leverages SOTA perspective depth estimation foundation models as teacher models and generates pseudo labels for unlabeled 360-degree images using six-face cube projection approach. By doing so, efficiently address the challenge of labeling depth in 360-degree imagery by leveraging perspective models and large amounts of unlabeled data.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Enor
RG8
Depih
Truth
BFusstt
(p)
Figure 1: Our proposed training pipeline improves existing 360 monocular depth estimators. This figure demonstrated the improvement of our proposed training pipeline tested on the Stan- ford2D3D [2] dataset in a zero-shot setting.
Our approach consists of two key stages: offline mask generation and online joint training. During the offline stage, we employ a combination of detection and segmentation models to generate masks for invalid regions, such sky and watermarks in unlabeled data. Subsequently, in the online stage, we adopt a semi-supervised learning strategy, loading half of the batch with labeled data and the other half with pseudo-labeled data. Through joint training with both labeled and pseudo-labelt labeled data, method achieves robust depth estimation performance on 360-degree imagery.
To validate the effectiveness of our approach, we conduct extensive experiments on benchmark datasets such as Matterport3D and Stanford2D3D.. Our method demonstrates significant improvements in depth estimation accuracy, particularly in zero-shot scenarios where models are trained on one dataset and evaluated on another. Furthermore, we demonstrate the efficacy our training techniques with different SOTA 360-degree depth models and various unlabeled datasets, showcasing the versatility and effectiveness our approach in addressing the unique challenges posed by 360-degree imagery.
Our contributions can be summarized as follows:
We propose novel training technique for 360-degree imagery that harnesses the power of unlabeled data through the distillation of perspective foundation models.
We introduce an online data augmentation method that effectively bridges knowledge distillation across different camera projections.
Our proposed training techniques significantly benefit and inspire future research on 360- degree imagery by showcasing the interchangeability of state-of-the (SOTA) 360 models, perspective teacher models, and unlaheled datasets. This enables better results even as new SOTA techniques emerge in the future.
Related Work
360 monocular depth. Depth estimation for 360-degree images presents unique challenges due the equirectangular projection and inherent distortion. Various approaches have been explored to address these issues:
Directly Apply: Some methods directly apply monocular depth estimation techniques to 360-degree imagery. OmniDepth [69] leverages spherical geometry and incorporates Sph- Conv [42] to improve depth prediction with distortion. [70, 49] use spherical coordinates to overcome distortion with extra information. [12, 17] leverage other ground truth supervisions to assist on depth estimation. SliceNet [31] and ACDNet [68] propose advanced network architectures tailored for omnidirectional images. EGFormer [64] and HiMODE [18] in- troduce munsformer-basee model that captures global context efficiently, while [45, 44]

---

focuses on integrating geometric priors into the learning process. [13] proposed to generate large-seale datasets with SfM and MVS, then apply test-time training.
Cube:
Other approaches
cube map projections to mitigate distortion effects. 360-
SelfNet [46] is the first work to self-supervised 360 depth estimation leveraging cube- padding [9]. BiFuse [47] and its improved version BiFuse++ [48] two-branch archi-
tectures that utilize cuhe maps and equirectangular projections. UniFuse [16] combines
equirectangular and cube map projections and simplifies the architecture. [3] combines two-branch techniques with transformer network.
Tangent Image: Tangent image projections are also popular. [37, 23, 30] convert equirectan- gular images into a series of tangent images, which are then processed using conventional depth estimation networks. PanoFormer [40] employs a transformer braeedareeerraeddddddd architecture to handle tangent images, while SphereNet [1]] and HRDFuse [I] enhance depth prediction by collaboratively learning from multiple projections.
360 other works Beyond depth estimation, 360-degree imagery has been applied to depth com- pletion tasks as follows [26, 8, 32, 57, 58, 15J. Other methods, such as [25] and [43] focus on the projection between camera models, while the former projects pinhole camera model images into a large field of view, whereas the latter transforms convolution kernels.
Unlabeled Pseudo labeled data. Utilizing unlabeled or pseudo-labeled data has become
significant trend to mitigate the limitations of labeled data scarcity. Techniques like [22, 71, 41, 56] leverage large amounts of unlabeled data to improve model performance through semi-supervised learning In the context of 360-degree depth estimation, our approach generates pseudo labels from pre-trained perspective models, which are then used to train 360-degree depth models effectively.
Zero-shot methods. Zero-shot learning methods aim to generalize to new domains without ad- ditional training data. [7. 54] target this directly with increasing training data., MiDaS [35, 5, 34] and Depth Anything [59] are notable for their robust monocular depth estimation across diverse datasets leveraging affine-invariant loss. [61] takes a step further to investigate zero-shot on metric depth. Marigold [19] leverages diffusion models with image conditioning and up-to-scale relative depth denoising to generate detailed depth maps. ZoeDepth [4] further these advancements by incorporating scale awareness and domain adaptation. [14, 50] leverage camera model information to adapt cross-domain depth estimation.
Foundation models. Foundation models have revolutionized various fields in AI, including nat- ural language processing and image-text alignment. In computer vision, models like CLIP [33] demonstrate exceptional generalization capabilities. [28] proposed foundation visual encoder for downstream tasks such as segmentation, detection, depth estimation, etc. [20] proposed model that can cut out masks for any objects. Our work leverages a pre-trained perspective depth estimation foundation model [59] as teacher model to generate pseudo labels for 360-degres images, enhancing depth estimation by utilizing the vast knowledge embedded in these foundation models.
3 Methods
In this work, we propose a novel training approach for 360-degree monocular depth estimation models. Our method leverages a perspective depth estimation model as a teacher and generates pseudo labels for unlabeled 360-degree images using 6-face eube projection. Figure 2 illustrates our training pipeline, incorporating the use of Segment Anything to mask out sky and watermark regions in unlabeled data during the offline stage. Subsequently, labeled and unlabeled data, allocating half of the batch to each. The joint training avoids limiting performance by teacher model. The unlabeled data is supervised using pseudo labels generated by Depth Anything, state-of-the-at perspective monocular depth foundation model. With the benefit of our teacher model, the 360-degree depth model demonstrates an observable improvement on the zero-shot dataset, as shown in Figure 1.
conduct joint training using both
3.1 Unleashing the Power of Unlabel 360 data
Dataset statisties. 360-degree data has become increasingly available recent years. However, compared to perspective-view depth datasets, labeling depth ground truths for 360-degree data presents greater challenges. Consequently, the availability of labeled datasets for 360-degree data considerably smaller than that of perspective datasets.
3

---

350 model
Loss on GT
Labeled data
Depth prediction
Ground truth
Weglt sharing
[a] Supervised training
model
Equi. cube!
Unlabeled data
Depth Anything
SAM
U Samorandon rotation
Equi. cube
Psoudo
Equi. cube
Loss pseudo GT
Valid pixel mask
Valid masks
(b) Distillation with unlabeled data
(c) Random rotation
Figure 2: Training Pipeline. Our proposed training pipeline involves joint training both labeled 360 data with ground truth and unlabeled 360 data. (a) For labeled data, we train our 360 depth model with the loss between depth prediction and ground truth. (b) For unlabeled data, propose to distill knowledge from a pre-trained perspective-view monocular depth estimator. In this paper, use Depth Anything [59] to generate pseudo ground truth for training. However, more advanced techniques could be applied. These perspective-view monocular depth estimators fail to produce reasonable equirectangular depth as a domain gap exists. Therefore, distill knowledge by inferring six perspective cube faces and passing them through perspective-view monocular depth estimators. To ensure stable and effective training, we propose generating valid pixel mask with Segment Anything [20] while calculating loss. (c) Furthermore, we augment random rotation on RGB before passing it into Depth Anything, as well as on predictions from the 360 depth model.
Table 360 monocular depth estimation lacks large amount training data. The number of images in 360-degree monocular depth estimation datasets alongside perspective depth datasets from the Depth Anything methodology.
Perspective
Equirectangular
Labeled
1.5M
Unlabeled 62
Labeled
34K Unlabeled
344K
Table presents the data quantities available in some of the most popular 360-degree datasets, including Matterport3D [6]. Stanford2D3D [2], and Structured3D [65]. Additionally, we list multi-modal dataset, Spatial/udioGeen [29], which consists of unlabeled 360-degree data used in our experiments. Notably, the amount of labeled and unlabeled data used in the perspective foundation model, Depth Anything [59]. significantly larger, with 1.5 million labeled images [24, 52, 10, 60, 55, 51] and 62 million unlabeled images [38, 63, 53, 62, 39, 21, 66, 20], making the amount in 360-degree datasets approximately 170 times smaller.
Data cleaning and valid pixel mask generation Unlabeled data often contains invalid pixels in regions such as the sky and watermark, leading to unstable training or undesired convergence. To address this issue, we applied the GroundingSAM [36] method to mask out the invalid regions. This approach utilizes Grounded DINOv2 [27] to detect problematic regions and applies the Segment Anything [20] model mask out the invalid pixels by segmenting within the bounding box. While Depth Anything [59] also employs a pre-trained segmentation model, DINOv2, select sky regions. Brand logos and watermarks frequently appear after fisheye camera stitching. Therefore, additional labels are applied to enhance the robustness of our training process. We also remove all images with Less than 20 percent of valid pixels to stablize our training progress.
Perspective foundation models (teacher models). To tackle the challenges posed by limited data and labeling difficulties in 360-degree datasets, we leverage large amount of unlabeled data alongside state-of-the-a perspective depth foundation models. Due to significant differences in camera projection and distortion, directly applying perspective models to 360-degree data often yields inferior results. Previous works have explored various methods of projection for converting equirectangular to perspective depth, stated in Section 2. Among these, cube projection and tangent

---

Valid Pixel Masking, We used Grounded-Segnne Anything [36] to mask out invalid pixels based on two text prompts: "sky" and "watermark." " These regions lack depth sensor ground truth labels in all previous datasets. Unlike Depth Anything [59]. which sets sky regions as disparity, we follow ground truth training ignore these regions during training for two reasons: (1) segmentation may misclassify and set other regions as zero, leading to noisy labeling, and (2) watermarks are post-processing regions that lack geometrical meaning.
Figure 3:
projection are the most common techniques. We selected cube projection to ensure larger field of view for each patch, enabling better observation relative distances between pixels or objects during the inference of the perspective foundation model and enhancing knowledge distillation. The comparison table can be find the supplementary material.
In our approach, we apply projection to unlabeled 360-degree data and then run Depth Anything on these projected patches of perspective images to generate pseudo-labels. We explore two directions for pseudo-labe supervision: projecting the patch to equirectangular and computing in the 360-degree domain or projecting the 360-degree depth output from the 360 model to patches and computing in the perspective domain. Since training is conducted in up-to-scale relative depth manner, stitching patch of perspective images back to equirectangular with an aligned scale will lead to failure in training Figure 4, which is an additional research topic that is worth investigation. We opt to compute the loss in the perspective domain, facilitating faster and casier training without the need for additional alignment optimization.
3.2 Random Rotation Processing
Directly applying Depth Anything cube- projected unlabeled data does not yield improvements
due to ignorance tross-cube-face relation, leading to cube artifacts (Figure 5). This issue arises from the separate estimation of perspective cube faces, where monocular depth is estimated based on semantic information, lacking a comprehensive understanding of the entire scene. To address this, we propose random rotation preprocessing step in front of the perspective foundation model.
As depicted in Figure 2, the rotation is applied to equirectangular projection RGB images using a random rotation matrix, followed by cube projection. This results in more diverse set of cube faces, capturing relative distances between ceilings, walls, windows, and other objects more effectively, With the proposed random rotation technique, knowledge distillation becomes more comprehensive as the point of view is not static. The inference by the perspective foundation model is performed on the fly, with parameters frozen during the training of the 360 model.
In order to perform random rotation, we apply a rotation matrix on the equirectangular coordinates, noted as (0, Φ), and rotation matrix as R.
(0 = R (0,00)
For equirectangular to cube projection, the field-of-view (FoV) of each cube face is equal to 90 degrees; each face can be considered as perspective camera whose focal length is w/2, and all faces share the same center point in the world coordinate. Since the six cube faces share the same center point, the extrinsic matrix of each camera can be defined by rotation matrix R, is then the pixel
5

---

(w/ Rotation)
useudo GT
(c) Prod on Equi (ww/ Rotation)
Pseudo GT Rotation)
Prod on Equi Rotation
(e)
(f
Figure 4: Qualitative visualization of model trained directly pseudo equirectangular data without scale alignment. We propose calculating the loss with pseudo ground truth on cube faces due to scale misalignment between the six faces during the cube-to-equirectangulaa projection. We showcase the results of model trained on pseudo equirectangular data without scale alignment as simple baseline to demonstrate the importance of calculating loss separately on each of the six faces. The images are presented from top to bottom as follows: (a) RGB images. (b) Pseudo cube ground truth projected directly equirectangular. (c) Prediction trained with 2. (d) Pseudo cube ground truth with rotation projected directly to equirectangular. (e) Prediction trained with row 4. (f) Our model's 's predictions are trained on cube faces separately with rotation.
on the cube face
p K RT 4
(2)
where,
(f) cos cos( sin (b) cos cos Φ
W
w 2
q 9y
K
w
2
W /
L9=
where and & are longitude and latitude in equirectangular projection and q the position in Euclidean space coordinates.
3.3 Loss Function
The training process closely resembles that of MiDaS, Depth Anything, and other cross-dataset methods. Our goal is provide depth estimation for any 360-degree images. Following previous approaches that trained on multiple datasets, our training objective to estimate relative depth. The depth values are first transformed into disparity space using the formula 1/d and then normalized to the range [0, 1] for each disparity map.
To adapt to cross-dataset training and pseudo ground truths from the foundation model, we employed the affine-invariant loss, consistent with prior cross-dataset methodologies. This loss function
disregards absolute scale and shifts for each domain, allowing for effective adaptation across different

---

Figure 5: Cube Artifact. Shown in the center row of the figure, undesired cube artifact appears when we apply joint training with pseudo ground truth from Depth Anything [59] directly. This issue arises from independent relative distances within each cube face caused by static point of view. Ignoring cross-cube relationships results in poor knowledge distillation. To address this, as shown in Figure 2(c), randomly rotate the RGB image before inputting it into Depth Anything. This enables better distillation depth information from varying perspectives within the equirectangular image.
datasets and models.
HW HW Σ pld;, di);
C1
where dž and d, are the prediction and ground truth, respectively. P represents the affine-invariant mean absolute error loss:
p(d;,d)) = |4; d:l-
Here, de and d; are the scaled and shifted versions of the prediction dł and ground truth d,:
d d, ((d)
s(d)
where (cd) and sdd are used align the prediction and ground truth have zero translation and unit scale:
HW
t(d) median(d),
s(d)
Σ |d t(d)).
HW
Experiments
These notations apply for all tables: M: Matterport3D [6], SF: Stanford2D3D [2]. ST: Struc- tured3D [65], SP: Spatialaudiogen [29]. -all indicates using the entire train, validation, and test sets of the specific dataset, and (p) denotes using pseudo ground truth generated by Depth Anything [59]. Due to space limits, provide the experimental setup in the appendix, including implementation details and evaluation metrics.
4.1 Baselines
Recent state-of-the-art methods [1, 64, 47, 48, 16, 31, 40] have emerged. We chose UniFuse and BiFuse++ as our baseline models for experiments, as many of the aforementioned methods did not fully release pre-trained models or provide training code and implementation details. It's worth noting that PanoFormer [40] is not included due to incorrect evaluation code and results. Both selected models are re-implemented with affine-invariant loss on disparity for fair comparison and to demonstrate improvement. We conduct experiments on the Matterport3D [6] benchmark to demonstrate performance gains within the same dataset/domain, and we perform zero-shot evaluation on the Stanford2D3D [2] test set to demonstrate the generalization capability of our proposed training technique. further validate its robustness, we evaluate additional baseline models [45, 64) in zero-shot setting, showcasing the efffcctiveness of our approach for non-dual duall-uoojcctoin models.

---

Table 2: Matterport3D Benchmark, The upper section lists 360 methods trained with metric depths meters using BerHu loss. All numbers are sourced from their respective papers. The lower section includes selected methods retrained with relative depth (disparity) using affine-invarian loss.
Method
Loss
Train
Test
Abs Rel +
51 t
52
53 1
BiFuse [47]
BerHu
M
M
0.845
0,932
0.963
UniFuse [16]
BerHu
M
M
0.106
0.890
0,962
0.983
SliceNet [31]
BerHu
M
M
0.872
0,948
0.972
BiFuse++ [48]
BerHu
M
0.879
0,952
0.977
HRDFuse [1]
BerHu
M
M
0.097
0.916
0.967
0.984
UniFuse [16]
Aff
M
0.102
0.893
0,970
0.989
UniFuse [16]
Iff
M. ST-all (p)
M
0.089
0.911
0.975
0.991
BiFuse++ [48]
Inf M
M
0.094
0.914
0,974
0.989
BiFuse++ [48]
Inf
M. ST-all (p)
M
0.085
0.917
0.976
0.991
4.2 Benchmarks Evaluation
We conducted our in-domain improvement experiment on the widely used 360-degree depth bench- mark, Matterport3D [6]. to showcase the results of perspective foundation model distillation on the two selected baseline models, UniFuse [16] and [ifuse++[48]. In Table 2, we list the metric depth evaluation results from state-of mfate-of-thh-ar art methods on this benchmark. Subsequently, we present the re-trained baseline models using inffarninvariant loss on disparity to ensure fair comparison with their original depth metric training. Finally, we demonstrate the improvement achieved with results trained on the labeled Matterport3D training set and the entire Structured3D dataset with pseudo ground truth.
4.3 Zero-Shot Evaluation
Our goal is to estimate depths for all 360-degree images, making zero-shot performance crucial, Following previous works [47, 16], we adopted their zero-shot comparison setting, where models trained on the entire Matterport3D [6] dataset are tested on the Stanford2D3D [2] test set. In Table 3, the upper section lists methods trained with metric depth ground truth, with numbers sourced from their respective papers. The lower section includes models trained with affine-invariant loss on disparity ground truth. As shown in Figure 6, [16, 48] demonstrate generalization improvements with a lower error on the Stanford2D3D dataset.
Depth Anything [59] and Marigold [19] are state-of-the-art zero-shot depth models trained with perspective depths. As shown in Table 3, due to the domain gap and different camera projections, foundation models trained with perspective depth cannot be directly applied 360-degree images. We demonstrated the zero-shot improvement on UniFuse [16]. BiFuse++ [48] and non-dual- -projual-nooocttion methods [45, 64] with models trained on the entire Matterport3D [6] dataset with ground truth and the entire Structured3D [65] or SpatialAudioGen [29] dataset with pseudo ground truth generated using Depth Anything [59].
As Structured3D provides ground truth labels for its dataset, we also evaluate our models on its test set to assess how well they perform with pseudo labels. Table 4 shows the improvements achieved on the Structured3D test set when using models trained with pseudo labels. It's worth noting that even when the 360 model is trained on pseudo labels from Spatial AudioGen performs similarly well. This demonstrates the success of our distillation technique and the model's ability to generalize across different datasets.
4.4 Qualtative Results in the Wild
360-degree images that were either captured by us or downloaded from the internet'. These examples showease the zero-shot capability of our model when applied to data outside the aforementioned 360-degree datasets.
We demonstrated the qualitative results in Figure 8 and Figure 7
http::/wwwww....ooooocccsscas/wa CC BY 2.0 DEED Luca Biada, https///wwwttkkkoooooooopoooooos CC BY 2.0 DEED CC BY 2.0 DEED
Sttg https//wwww....iiccco..o///oooos
Dominic Alves, https://www.ficc.
Luca Biada, http:////www.iillooo//////////// CC BY 2.0 DEED

---

Table Zero-shot Evaluation on Stanford2D3D. We perform zero-shot evaluations with models trained on other datasets. Following the original training settings, train the 360 models [48, 16, 45, 64] on the entire Matterport3DD dataset and then test on Stanford3D's 8 test set.
Method
Loss
train
test
Abs Rel
61
62 1
83 1
BiFuse [47]
BerHu
M-all
SF
0.120
0.862
UniFuse [16]
BerHu
M-all
SF
0.094
0.913
BiFuse++ [48]
BerHu
M-all
SF
0.107
0.914
0.975
0.989
Depth Anything [59]
Affine-Im
Pers.
SF 0.248
0.635
0.899
0.97
Marigold [19]
Affine-Im
Pers.
SF
0.195
0.692
0.942
0,982
UniFuse [16]
Affine-Inv
M-all
SF
0.090
0.914
0.976
0.990
UniFuse [16]
Affine-Inv
M-all, ST-all (p)
SF
0.086
0.924
0.977
0,990
UniFuse [16]
Affine-Inv
M-all, SP-all (p)
SF
0.090
0.920
0.978
0.990
BiFuse++ [48]
Affine-Inv
M-all
SF
0.090
0.921
0.976
0.990
BiFusc++ [48]
Affine-Inv
M-all, ST-all (p)
SF
0.082
0.931
0.979
0.991
BiFusc++ [48]
Affine-Inv
M-all, SP-all (p)
SF
0.086
0.926
0.979
0.991
HoHoNet [45]
Affine-Inv
M-all
SF
0.095
0.906
0.975
0.991
HoHoNet [45]
Iff
M-all, ST-all (p)
SF
0.088
0.920
0.979
0.992
EGFormer [64]
Affine-Inv
M-all
SF
0.098
0.906
0.972
0.989
EGFormer [64]
Affine-Inv
M-all, ST-all (p)
SF
0.086
0.923
0.976
0.990
UniFuse
BiRAe
Error
Error
Puse (o)
BiFuse++ (o)
Error
Figure 6: Zero-shot qualitative with UniFuse [16] (left) and BiFuse++ [48] (right) tested Stanford2D3D.
Table 4: Structured3D Test Set. We demonstrate the improvement on the Structured3D test set using pseudo ground truth for training. The lower section shows enhancements with models trained on pseudo ground truth from Matterport3D and SudtialAudioGen, indicating similar improvements. This highlights the successful distillation of Depth Anything.
Method
Loss
train
test
Abs Rel |
61 1
82
0g 1
UniFuse [16]
Affine-Inv
M-all
ST
0.202
0.759
0.932
0,970
UniFuse [16]
Affine-Inv
M-all, ST-all (p)
ST
0.130
0.887
0.953
0.977
UniFuse [16]
Affine-Inv
M-all, SP-all (p)
ST
0.152
0.864
0.946
0.972

---

Table 5: Metric depth fine-tuning. We fine-tune our model trained with Matterport3D [6] ground truth label and Structured3D [65] pseudo label on relative depth with Stanford2D3D [2]'s training set metric depths with a single epoch.
Method
MAE 4
Abs Rel +
RMSE +
RMSElog +
81 1
82 1
5g 1
UniFuse [16]
0.208
0.111 0.118
0.369
0.072
0.871
0.966
0.988
UniFuse [16] (Ours)
0.206
0.351
0.049
0.910
0.971
0.987
RGB
Unit Uni
UniFuse
Figure Generalization ability in the wild with depth map visualization. We showcase zero-shot qualitative results using combination of images captured and randomly sourced from the internet assess the model's generalization ability. For privacy reasons, we have obscured the cameraman in the images.
Figure Generalization ability in the wild with point cloud visualization. We showcase zero-shot qualitative results in the point cloud using a combination of images we captured and randomly sourced from the internet to assess the model's generalization ability.
4.5 Fine- Tune-Tunedd to Metric Depth Estimation
We our pre-trained model initial weight and fine-tune Stanford2D3D [2] metric depth to demonstrate the effectiveness of our pre-trained relative depth model's ability to adapt to metric depth with single epoch in Table 5
Conclusion
Our proposed method significantly advances 360-degree monocular depth estimation by leveraging perspective models for pseudo-label generation on unlabeled data. The use of cube projection with random rotation and affine-invariant loss ensures robust training and improved depth prediction accuracy while bridging the domain gap between perspective and equirectangular projection. By effectively addressing the challenges of limited labeled data with cross-domain distillation, our approach opens new possibilities for accurate depth estimation in 360 imagery. This work lays the groundwork for future research and applications, offering promising direction for further advancements in 360-degree depth estimation.
Our work faces limitations due to its heavy reliance on the quality unlabeled data and pseudo labels from perspective foundation models. The results are significantly impacted by data quality (Section 3. Without data cleaning, the training process resulted NaN values. Another limitation is that although with unlabeled data, the scarcity of data still exists compared to other tasks.
Limitations.
10

---

References
[1] Hao Ai, Zidong Cao, Yan-Pei Cao, Ying Shan, and Lin Wang. Hrdfuse: Monocular 360deg depth estimation by collaboratively learning nolistie-with-regiond depth distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13273-13282, 2023.
[2] I. Armeni, A. Sax, A. R. Zamir, and S. Savarese. Joint 2D-3D-Semantic Data for Indoor Scene Understanding. ArXiv e-prints, February 2017.
[3] Jiayang Bai, Haoyu Qin, Shuichang Lai, Jie Guo, and Yanwen Guo. Glpanodepth: Global-to- local panoramic depth estimation. IEEE Transactions on Image Processing, 2024.
[4] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 12288, 2023.
[5] Reiner Birkl, Diana Wofk, and Matthias Müller. Midas v3.1 - a model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307. 14460, 2023.
[6] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[7] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild. Advances in neural information processing systems, 29, 2016.
[8] Yichen Chen, Yuqi Pan, Ruyu Liu, Haoyu Zhang, Guodao Zhang, Bo Sun, and Jianhua Zhang. 360orb-slam: A visual slam system for panoramic images with depth completion network. In 2024 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD)), pages 717-722. IEEE, 2024.
[9] Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, and Min Sun. Cube padding for superve-uper saliency prediction in 360 ('dee) videos. arXiv preprint 18Xivv1806.01
2018
[10] Jachoon Cho, Dongbo Min, Youngjung Kim, and Kwanghoon Sohn. Diml/cvl rgb-d dataset: 2m rgb-d images of natural indoor and outdoor scenes. arXiv preprint arXiv:2110.11590, 2021.
[11] Benjamin Coors, Alexandru Paul Condurache, and Andreas Geiger. Spherenet: Learning spher- ical representations for detection and elassification in omnidirectional images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 51 518-533, 2018.
[12] Brandon Yushan Feng, Wangiue Yao, Zheyuan Liu, and Amitabh Varshney. Deep depth estimation on 360 images with double quaternion loss. 2020 International Conference on 3D Vision (3DV), pages 233.533. IEEE, 2020.
[13] Qi Feng, Hubert PH Shum, and Shigeo Morishima. 360 depth estimation in the wild-the depth360 dataset and the segfuse network. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 663-673. IEFF, 2022.
[14] V. Guizilini, I. Vasiljevic, D. Chen, R. Ambrus, and A. Gaidon. Towards zero-shot scale- aware monocular depth estimation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9199-9209, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society, doi: 10.1 D0.100/CCCV11000022222 202.00020002222...... 3.0047.. URL https: / //ooi. htt////..ieeecoooe// weeecompttersocceet...........ee org/ 10. 1w.119//ICCV1100001 2023.00847.
[15] Yu- Kai Huang, Tsung- Han Wu, Yuch- Liu, and Winston H Hsu. Indoor depth completion with boundary consistency and self-attention. In Proceedings of the IEEF/CV international conference on computer vision workshops, pages 0-0, 2019.
[16) Hualie Jiang, Zhe Sheng, Siyu Zhu, Zilong Dong, and Rui Huang. Unifuse: Unidirectional fusion for 360 panorama depth estimation. IEEE Robotics and Automation Letters, 2021.
11

---

[17] Lei Jin, Yanyu Xu, Jia Zheng, Junfei Zhang, Rui Tang, Shugong Xu, Jingyi Yu, and Shenghua Gao. Geometric structure based and regularized depth estimation from 360 indoor imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 889-898, 2020.
[18] Masum Shah Junayed, Arezoo Sadeghzadeh, Md Baharul Islam, Lai- Kuan Wong, and Tarkan Aydın. Himode: A hybrid monocular omnidirectional depth estimation model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 222112 2022.
[19] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-basee image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), 2024.
[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643 2023.
[21] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[22] Mong-Hyun Lee. Pseudo-label The simple and efficient supervipe learning method for deep neural networks. 2013. URL https: //ppi.ppppppp semanti semanticscholar IDr 18507866.
[23] Yuyan Li, Yuliang Guo, Zhixin Yan, Xinyu Huang, Duan Ye, and Liu Ren. Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In 2022 Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, June 2022.
[24] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Computer Vision and Pattern Recognition (CVPR), 2018.
[25] Daniel Lichy, Hang Su, Abhishek Badki, Jan Kautz, and Orazio Gallo. Fova-depth: Field- of-view agnostic depth estimation for cross-dataset generalization. In 2024 International Conference on 3D Vision (3DV), pages 1-10, 2024. doi: 22111999DDV224532222..0000
[26] Ruyu Liu, Guodao Zhang, Jiangming Wang, and Shuwen Zhao. Cross-modal 360 depth completion and reconstruction for large-scale indoor environment. IEEE Transactions on Intelligent Transportation Systems, 23(12):25180-2 80-25119, 2022.
[27] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303. 055999 2023.
[28] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[29] Timothy Langlois Pedro Morgado, Nuno Vasconcelos and Oliver Wang. Self-supervised generation of spatial audio for 360deg video. In Neural Information Processing Systems (NIPS), 2018.
[30] Chi-Han Peng and Jiayao Zhang. High-resolution depth estimation for 360-degree panoramas through perspective and panoramic depth images registration. arXiv preprint arXiv:22 10..04400 2022.
12

---

[31] Giovanni Pintore, Marco Agus, Eva Almansa, Jens Schneider, and Enrico Gobbetti. SliceNet: deep dense depth estimation from single indoor panorama using slice-based representation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1536-11545, June 2021.
[32] Giovanni Pintore, Eva Almansa, Armando Sanchez, Giorgio Vassena, and Enrico Gobbetti. Deep panoramic depth prediction and completion for indoor scenes. Computational Visual Media, pages 1-20, 2024.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8766 8763. PMLR, 2021.
[34] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ICCV, 2021.
[35] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2022.
[36] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang. Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
[37] Manuel Rey-Area, Mingze Yuan, and Christian Richardt. 360MonoDepth: High-resolution 360 monocular depth estimation. In CVPR, 2022.
[38] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252 2015.
[39] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A scale, high-quality dataset for object detection. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), October 2019.
[40] Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao. Panoformer: Panorama transformer for indoor 360 depth estimation. European Conference on Computer Vision, pages 195-211 211...1.22 Springer, 2022.
[41] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
[42] Yu-Chuan Su and Kristen Grauman. Learning spherical convolution for fast features from 360 imagery. Advances in neural information processing systems, 30, 2017.
[43] Yu-Chuan Su and Kristen Grauman. Kernel transformer networks for compact spherical convolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[44] Cheng Sun, Chi-Wei Hsiao, Ning-Hsu Wang, Min Sun, and Hwann-Tzong Chen. Indoor panorama planar 3d reconstruction via divide and conquer. CVPR, 2021.
[45] Cheng Sun, Min Sun, and Hwann-Tzon Tzong Chen. Hohonet: 360 indoor holistic understanding with latent horizontal features. In CVPR, 2021.
[46] Fu-En Wang, Hou-Ning Hu, Hsien-Tzu Cheng, Juan-Ting Lin, Shang-Ta Yang, Meng-Li Shih, Hung-Kuo Chu, and Min Sun. Self-supervised learning of depth and camera motion from 360" videos. Asian Conference on Computer Vision, 2018 URL. https://api.
semanticscholar org/ CCrp/oorr::DSSSe 53290116..
13

---

[47] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai. Bifuse: Monocular 360 depth estimation via biprrojettion fusion. In The IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[48] Fu-En Wang, Yu-Hsuan Ych, Yi-Hsuan Tsai, Wei-Chen Chiu, and Min Sun. Bifuse++: Self- supervised and efficient bi-projection fusion for 360 depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5448 5460, 2023. doi: 10.110//PMMM.222.
3203516.
[49] Ning-Hsu Wang, Bolivar Solarte, Yi-Hsuan Tsai, Wei-Chen Chiu, and Min Sun. 360sd-net: 360 stereo depth estimation with learnable cost volume. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 582-588. IEEE, 2020.
[50] Ning-Hsu Wang, Ren Wang. Yu-Lun Liu, Yu-Hao Huang, Yu-Lin Chang, Chia-Ping Chen, and Kevin Jou. Bridging unsupervised and supervised depth from focus via all-in-focus supervision. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12621-
12631, 2021.
[51] Qiang Wang, Shizhen Zheng, Qingsong Yan, Fei Deng, Kaiyong Zhao, and Xiaowen Chu. Irs: A large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2021.
[52] Wenshan Wang. Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sehastian Scherer. Tartanair: A dataset to push the limits of visual slam. In 2020 IEEE/RS. International Conference on Intelligent Robots and Systems (IROS), pages 4909-4916. IEEE, 2020.
[53] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2575-2584, 2020.
[54] Ke Xian, Chunhua Shen, ZHIGUO CAO, Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo. Monocular relative depth perception with web stereo data supervision. 2018 IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 311-320, 2018. URL
atttps://apis htt//ppp..saaanicccollaaaea wwwpsccholaa...ooorouooossaseeee CorpusID::5280115
[55] Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao. Structure- guided ranking loss for single image depth prediction. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[56] Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[57] Zhiqiang Yan, Xiang Li, Kun Wang, Zhenyu Zhang, Jun Li, and Jian Yang. Multi-modal masked pre-traiming for monocular panoramic depth completion. In European Conference on Computer Vision, pages 578-395. Springer, 2022.
[58] Zhiqiang Yan, Xiang Li. Kun Wang, Shuo Chen, Jun Li, and Jian Yang. Distortion and uncertainty aware loss for panoramic depth completion. In International Conference on Machine Learning, pages 39099-39109 109. PMLR, 2023.
[59] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power large-scale unlabeled data. In CVPR, 2024.
[60] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 1790-1799 2020.
14

---

[61] Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from single image. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 9043-9053, 2023.
[62] Fisher Yu, Yinda Zhang, Shuran Song. Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1 03366, 2015.
[63] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning. In Proceedingy of the IEEE/CVE conference on computer vision and pattern recognition, pages 2636-2645, 2020.
[64] 1lwi Yun, Chanyong Shin, Hyunku Lee, Hyuk-Jae Lee, and Chae Eun Rhee. Egformer: Equirectangular geometry-biased transformer for 360 depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6101-6112, October 2023.
[65] Jia Zheng, Junfei Zhang. Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: large photo-realistic dataset for structured 3d modeling. In Proceedings of The European Conference Computer Vision (ECCV), 2020.
[66] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[67] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6612 6619, 2017. doi: 10.1109/VVP..007..700.
[68] Chuanqing Zhuang, Zhengda Lu, Yiqun Wang, Jun Xiao, and Ying Wang. Acdnet: Adaptively combined dilated convolution for monocular panorama depth estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3653-3661, 2022.
[69] Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, and Petros Daras. Omnidepth: Dense depth estimation for indoors spherical panoramas. In Proceedings of the European Conference on Computer Vision (ECCV), pages 448-465, 2018
[70] Nikolaos Zioulis, Antonis Karakottas, Dimitris Zarpalas, Federic Alvarez, and Petros Daras. Spherical view synthesis for self-supervised 360 depth estimation. International Conference on 3D Vision (3DV), September 2019.
[71] Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoe V. Le. Rethinking pre-training and self-training. ArXiv, abs/2006. 06882, 2020. URL https: //api. semanticschon httpatacccco.o...rr/rrrr......o 2196635973
15

---

A Appendix Supplemental Material
A.1 Experimental Setup
Implementation details. Our work is divided into two stages: (1) offline mask generation and (2) online joint training. (1) In the first stage, we use Seguuddd-SSeeee-AAnniiie
[36], which combines state-of-the-ar detection and segmentation models. We set the TOX_THRESHOLD and TEXT_THRESHOLA to 0.3 and 0.25, respectively, following the recommendations the official code. We use "sky" and "watermark" as text prompts. All pixels with these labels are set to False to form our valid mask for the second stage of training. (2) the second stage, each batch consists of an equal mix of labeled and unlabeled data. We follow the backbone model's official settings for batch size, learning rate, optimizer, augmentation, and other hyperparameters, changing only the loss function to affine-invariant loss. Unlike Depth Anything, which sets invalid sky regions to zero disparity, we ignore these invalid pixels during loss calculation, consistent with ground truth training settings. We average the loss for ground truth and pseudo ground truth during updates. All our experiments are conducted on a single RTX 4090, both offline and online. However, if future 360-degree state-of-the-arr methods perspective foundation models require higher VRAM usage, the computational resource requirements may increase.
Metrics. In line with previous cross-dataset works, all evaluation metrics presented in per- centage terms. The primary metric Absolute Mcan Relative Error (AbsRel), calculated as: M Ee la; del/d;, where M is the total number of pixels, 0; is the predicted depth, and d; the ground truth depth. The second metric, ón accuracy, measures the proportion of pixels where mar(ai/ diiiii di/on di/oi) is within 1.25'. During evaluations, we follow [16, 47, 48] ignore areas where ground truth depth values are larger than 10 or equal to O. Given the ambiguous scale of self-training results, we apply median alignment after converting disparity output depth before evaluation, as per the method used in [67]]
d' = d median(d) median(d)
(8)
where d is the predicted depth from inverse disparity and d is the ground truth depth. This ensures a fair comparison by aligning the median depth values of predictions and ground truths.
A.2
More Qualitative
We demonstrate additional zero-shot qualitative results in Figure and in-the-will results in Figure 12. In-domain results the Matterport3D test sets are showcased in Figure 10 and Figure 11.
A.3 Dataset Statistic
As described in Sec.3. of the main paper, there a significant difference in the number of images between the perspective and equirectangular datasets. Detailed statistics of the datasets are listed in Table 6.
A.4
Ground Truth and Pseudo Label Ratio Ablation
Unlike many previous knowledge distillation approaches that use a higher proportion of pseudo labels during model training, we opt for an equal ratio ground truth to pseudo labels. Through an ablation study exploring the relationship between this data ratio and model performance, we observe a robust improvement starting from 1 in Table 7.
A.5 Perspective Camera Projection Ablation
There are various perspective camera projections for panoramic imagery, with cube and tangent image projections being the most common, both widely used in previous works. In Table 8, we compare these two projections and observe similar improvements when applying our proposed training pipeline, demonstrating the effectiveness of our method. For robust knowledge distillation in relative depth estimation, we select the cube projection due to its wider field of view coverage.
16

---

GT
UniFuse (p)
BiFuse-+
BiFuser (p)
Figure 9: More qualitative tested on Stanford2D3D with zero-shot setting.
RGB
GT
UniFUse
UniFUse
Figure 10: In-domain qualitative with UniFuse.
17

---

RGB
GT
BiFuse++
BiFuser+ (P)
Figure 11: In-domain qualitative with BiFuse++.
Table 360 monocular depth estimation lacks large amount of training data. This table lists datasets used in 360-degree monocular depth estimation alongside perspective depth datasets from the Depth Anything methodology. The volume training data for 360-degree imagery (right column) is significantly smaller than that for perspective imagery (left column) by about 200 times. This highlights the need for using perspective distillation techniques to enhance the limited data available for 360-degree depth estimation. Ground truth (GT) labels are noted where applicable, showing the available resources for training these domains.
Perspective Venue
Equiroctangular Venue
Dataset
of images
GT labels
Dataset
images
GT labels
Mega Depth [24]
CVPR 2018
128K
Stanford2233 [2]
arXiv 2017
1.4K
TartanAir [52]
IROS 2020
306K
Matterpor: [6]
3DV 2017
10,8K
DIMI [10]
arXiv 2021
927K
Strnctured3D [65]
ROCV 2020
2188K
BlendedMVS
CVRP 2020
115
SpalialAudioGen
NeurIPS 2018
344K
HRWST [55]
CVPR 2020
20K
IRS [51]
ICME 2021
103K
ImageNet-21K [38]
ICV 2015
13.1M
BDD100K [63]
CVPR 2020
8.2M
Google Landmarks [53]
CVPR 2020
4.1M
LSUN [62]
arXiv 2015
9.8M
Objects365 [39]
ICCV 2019
1.7M
Open Images V7 [21]
JIC 2020
7,8M
Places365 [66]
SA-1B [20]
ICCV 2023
11.1M
Table 7: Ratios of GT and Pseudo label during training. We conduct additional experiments with varying ratios, which shows our method is robust across different ratios starting from 1:1.
Ratio
train(GT)
train(sseudo)
test
Abs Rel 1.
61 1
82 f
6g 1
1:1
M-all
ST-all(p)
SF
0.086
0.924
0.977
0.990
1:2
M-all
ST-all(p)
SF
0.087
0.923
0.977
0.990
1:4
M-all
ST-all(p)
SF
0.085
0.923
0.977
0.990
18

---

Table 8: Comparison between Tangent Image and Cube Projection. We compare two of the most commonly used perspective camera projections for panorama images. As shown the table, both projections yield similar quantitative results. However, we select the cube projection for knowledge distillation due to its broader field of view coverage.
Projection
Method
train
Abs Rel 1
82 f
53 f
Cube
UniFuse
M-alI+ST-all(p) M-all+ST-allpp)
SF
0.086
0.924
0.977
0.990
Tangent
UniFuse
SF
0.087
0.923
0.978
0.991
RGB
UniFuse
UniFuse (p)
Figure 12: Additional in-the-wild results. We compare our proposed
method (Matter- port3D (GT) AppatialAudoGGer (Pseudo)) with model trained only on the Matterport3D dataset, using data randomly downloaded from the interne.. This comparison demonstrates the significant improvement of method along with its generalization ability and effectiveness real-world data.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims presented in the abstract and introduction accurately portray the contributions and scope of our paper. As evidenced by Figure 2 and Table 3, our work indeed introduces training pipeline that demonstrates improvements in the zero-shot testing, making a significant contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: As described in Sec. 5, our work relies heavily on the quality of the data and the accuracy of the pseudo labels. We have also highlighted that disregarding or neglecting data cleaning procedures may result in undesirable training outcomes.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: As indicated in Section 4.3 and Table 3, our train pipeline numerically improves the existing 360 methods. We've also demostrated the effectiveness of pseudo ground truth on Structured3D in Table 4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have listed all data cleaning preprocessing steps in Sec.3. I, and the implementation details are described in Sec.A.1. This ensures transparency and enables reproducibility of our main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
21

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While the code for the work will released upon acceptance, we have chosen not to submit it during the reviewing stage. However, stated in the previous question, we have provided all the necessary information the paper for reproducibility of the main experimental results.
Guidelines;
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
Saaides/CodeSubiisee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.com wttpcccppppcccccecccc.c.cc.cccct gui httpbiiiiggddeeeeeeeeeiiiiieeeee www.eeeuuumissioooololooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: As specified in Sec. A.I, our training process follows the official implemen- tation of [16, 48], employing affine- invariantt loss and weighted average between loss from ground truth labels and pseudo labels.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
22

---

Justification:
The enror and metric calculations listed in Tables 2, 3, and 4 all adhere to prior works conducted on affine-invariant relative depth on disparity, as described in Sec. A.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: As specified in Sec. A.1, our work was conducted on a single RTX 4090, with the possibility of scaling up using interchangeable state-of-the-art 360 depth methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpps://///riiii /ww.rrnuuup...ee cc/ cc/public/ httppppppbppchceeeepeppppppppeee csGuidelines"
Answer: [Yes]
Justification: Our work focuses on Al training pipeline that bridges the gap between different camera models. No data are collected, no human-related crowdsoureing involved, and we address no specific topics that may lead to negative social impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
23

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As our work focus on bridge the knowledge distillation between differemce camera projection, it is relative general research on improving training progress with no negative broader impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As we propose new training process bridge the knowledge distillation between camera models, this work does not focus on single model development or dataset collection. Therefore, there are no safeguard issues associated with the release of data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
24

---

Answer: [Yes]
Justification: In this work, we ve utilized code [48, 16, 59, 20, 36, 27, 45, 64] and dataset [6, 2, 65, 29] from various sources. We'v ve've adhered their respective licenses, cited them appropriately, and incorporated their contributions into our research. For data used in in- the-wild generalization testing, we downloaded data that are free share and adapt. Their authors, URLs, and licenses are referenced properly in Sec. 4.4.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our work focuses improving the training pipeline of previous works [16, 48] with the addition of [59], all of which are publicly accessible and have been fully cited in our work. Additionally, we explain the training procedure in detail in the supplementary materials (Sec. A.1).
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA].
Justification: Our work does not involve any crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
25

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve any crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "yGqEKQX7u0", "reviewer": "Reviewer_3NfA", "summary": "The authors present a training strategy for single-image depth estimation on 360-degree equirectangular images. The strategy centers around leveraging strong pre-trained models for perspective images as teacher networks. It does not depend on any particular network architecture and therefore can benefit any 360 depth estimation networks. Experimental results validate that the strategy improves models otherwise trained only with limited ground-truth annotations.", "questions": "I am looking forward to answer to the questions raised above, namely:\n- How does training scale in terms of the amount of pseudo GT vs real GT? \n- Is the approach applicable to metric depth estimation? \n- Additional related work and baseline as described above.", "limitations": "The only limitation the authors bring up (quality of unlabelled data) already has a solution in the paper, so not really a limitation? \n\nI do think there are a few other things worth mentioning: \n- 360 data, even without requiring GT, is still scarce compared to perspective data. The fact the authors can only evaluate with two such datasets is an evidance. This is a limitation since it prevents further scaling up training. \n- Only equirectangular images are supported (though it seems that, in principle, the approach should work in more general camera models).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality and significance**: 360-degree imagery is becoming increasingly critical for many computer vision applications. However, there is still a significant gap in GT depth annotations compared to their perspective counterparts, and any solution to this issue can have a significant impact. The authors demonstrate that leveraging strong perspective depth models, e.g., Depth Anything, is a simple yet effective solution. \n\n**Quality**: The main idea and the several supporting procedures (e.g., random rotation processing, valid pixel masking, mixed labeled and unlabelled training) are all well-motivated and reasonably designed. Experimental results show consistent improvement by incorporating the proposed pseudo GT. Additional results including zero-shot and qualitative evaluations further help with understanding and make the approach overall more convincing. \n\n**Clarity**: Paper is well-written with good structure, clear expressions, and adequate details.", "weaknesses": "1. A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, and avoid generic remarks. \n\nDespite focusing on a slightly different task (stereo depth), FoVA-Depth (Lichy et al. 3DV 2024) presents a few very similar concepts: \n  - leveraging abundance of perspective depth GT\n  - cube map as a intermediate representation to gap between 360 and perspective images\n  - random rotation augmentation\nIt is worth some discussion regarding similarities and differences. \n\n2. A very simple yet critical baseline is missing: directly project pseudo GT on cubemap to equirectangular images. A good stitching strategy may be challenging, but with something simple or even without any additional scaling, it should help clarify how much the pre-trained depth anything model contribute to the overall performance. \n\n3. The paper addresses only relative depth estimation. Since several baselines (e.g. upper section of Tab.2) already have metric counterparts, and depth anything has metric variants, I feel some experiments and analysis in that regards should be straightforward and nicely complement the relative depth results. \n\n4. As the premise of the work is the usefulness of the abundant pseudo GT compard to limited 360 depth GT, it is necessary to understand how does the benefit from pseudo GT scale (how many pseudo GT can be as useful as a real 360 GT?), and how does the student network compare to the teacher network in terms of estimation quality (is the quality of teacher network already a bottleneck?). The paper offers little insight in these questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your positive assessment of our work's originality, significance, and clarity. We're glad you found our approach well-motivated and effective. We'll address your questions and concerns point by point:\n\n### `Q1. Comparison to FoVA-Depth (Lichy et al. 3DV 2024):`\nThank you for pointing this out. While there are indeed similarities in leveraging perspective depth data and using cube maps, our work differs in several key aspects:\n    - We focus on monocular depth estimation rather than stereo depth.\n    - Our approach is architecture-agnostic and can benefit any 360 depth estimation network.\n    - We introduce novel techniques like valid pixel masking and mixed labeled/unlabeled training.\nWe will add a discussion of these similarities and differences in the related work section.\n\n### `Q2. Baseline of directly projecting pseudo GT:`\nWe have tried this setting as an initial setting, and due to the un-aligned scale, the training leads to unstable results and artifacts as shown Figure 1 of the pdf. As can be seen, our method significantly outperforms direct projection, demonstrating the value of our training approach beyond just leveraging the pre-trained model\n\n### `Q3. Metric depth estimation:`\nWe focused on relative depth estimation due to its generalizability across different datasets. However, your point is well-taken, we will follow previous work (MiDaS[4]/Depth Anything[45]) and conduct metric depth finetuning in the final version.\n\n### `Q4. Scaling of pseudo GT vs. real GT:`\nWe have conducted an ablation study on the ratio between real GT and pseudo GT in a batch. These results show our method is robust across different ratios, and shows the effectiveness from a ratio of 1:1. We will include this analysis in the paper. We will add to the final version.\n\n| GT/Pseudo | Train              | Test | Abs Rel ↓ | δ₁ ↑ | δ₂ ↑ | δ₃ ↑ |\n|-----------|--------------------|------|-----------|------|------|------|\n| 1:1       | M-all, ST-all (p)   | SF   | 0.086     | 0.924| 0.977| 0.99 |\n| 1:2       | M-all, ST-all (p)   | SF   | 0.087     | 0.923| 0.977| 0.99 |\n| 1:4       | M-all, ST-all (p)   | SF   | 0.085     | 0.923| 0.977| 0.99 |\n\n### `Q5. Student vs. teacher network quality:`\nOur proposed method offers a cross-camera projection knowledge distillation. Our student models are SOTA methods designed for 360 images where as teacher model is initially designed for perspective images. Therefore, the teacher model is not yet a bottleneck due to cross-domain knowledge distillation, which is also shown in Table 3 in the original paper. We’ll include this discussion.\n\n### `Regarding limitations:`\n- We agree that the scarcity of 360 data, even unlabeled, is a limitation. We'll explicitly mention this.\n- While we focused on equirectangular images, our approach should indeed generalize to other projections. We'll note this potential extension.\n\nThank you again for your insightful feedback. We believe addressing these points will significantly strengthen our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTCIqshUH8", "reviewer": "Reviewer_F7bZ", "summary": "This paper introduces a novel depth estimation framework specifically designed for 360-degree data using an innovative two-stage process: offline mask generation and online semi-supervised joint training. Initially, invalid regions such as sky and watermarks are masked using detection and segmentation models. The method then employs a semi-supervised learning approach, blending labeled and pseudo-labeled data derived from state-of-the-art perspective depth models using a cube projection technique for effective training. This framework demonstrates adaptability across different state-of-the-art models and datasets, effectively tackling the challenges of depth estimation in 360-degree imagery.", "questions": "1. Is there any reason the author only applied the proposed method to the UniFuse and BiFuse++?\n2. There seems to be a discrepancy in the reported Absolute Relative (Abs Rel) error for BiFuse++ (Affine-Inv, M-all, SP-all(p)) at 0.858, which is ten times higher than results from competitive methods, while other metrics (\\delta_1, \\delta_2, \\delta_3) align closely. Could this be an error?", "limitations": "See the weakness and the question parts.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method employs models trained for traditional pinhole cameras to enhance 360-degree depth estimation, a first in the field. Moreover, the main motivation of this paper is reasonable.\n2. It outperforms conventional methods by incorporating pseudo labels from foundational models into the loss function.\n3. The paper demonstrates the model's generalizability to real-world scenarios, indicating its practical utility.", "weaknesses": "1. The proposed method is straightforward and the performance gains provided by the proposed method are described as marginal.\n2. The method's effectiveness is demonstrated only with specific models, UniFuse and BiFuse++, limiting evidence of its broader applicability.\n3. Inconsistencies in the decimal points used in quantitative results tables make direct performance comparisons challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our paper. We appreciate your recognition of our method's novelty and practical utility. We'd like to address your **concerns** and questions:\n\n### `C1. Regarding the straightforwardness of our method:` \nWhile our approach may appear straightforward, we believe this simplicity is a strength. It allows for easy integration with existing models and datasets. The performance gains, which you describe as marginal, are actually significant in the context of 360-degree depth estimation. For example, on the Stanford2D3D zero-shot test, our method improves AbsRel from 0.09 to 0.082 for BiFuse++, a 8.9\\% relative improvement. In computer vision, such improvements are considered substantial, especially for zero-shot scenarios. Moreover, our method demonstrates extraordinary improvements in the in-the-wild scenes, as shown in Figure 6 of the original image and Figure 2 of the PDF.\n\n### `C2. Limited application to specific models:`\nWe chose UniFuse and BiFuse++ as they are widely recognized baselines in 360-degree depth estimation. However, our method is designed to be model-agnostic. To demonstrate this, we've run additional experiments with a third model, HoHoNet[33]. Another method that has been mentioned by reviewers is EGformer[50]. Due to limited of training time during the rebuttal period, we have conducted zero-shot experiments on a subset of Matterport3D with $1/5$ of the original dataset size for EGformer[50]. The full dataset version will be added to the final version.\n| Method          | Train               | Test | Abs Rel ↓ | δ₁ ↑   | δ₂ ↑   | δ₃ ↑   |\n|-----------------|---------------------|------|-----------|--------|--------|--------|\n| HoHoNet         | M-all                | SF   | 0.095     | 0.906  | 0.975  | 0.991  |\n| HoHoNet (our)   | M-all, ST-all (p)    | SF   | 0.088     | 0.920  | 0.979  | 0.992  |\n| EGformer        | M-subset             | SF   | 0.169     | 0.764  | 0.924  | 0.972  |\n| EGformer (our)  | M-subset, ST-all (p) | SF   | 0.148     | 0.814  | 0.946  | 0.982  |\n\nThese results further demonstrate our method's effectiveness across different architectures. We will include these analysis in the paper and discuss the broader applicability of our approach.\n\n### `C3. Inconsistencies in decimal points:`\nWe apologize for this oversight. We will standardize all results to three decimal places for clarity and ease of comparison in the final version.\n\n### Regarding your specific **questions**:\n\n### `Q1. Choice of UniFuse and BiFuse++:`\nWe chose UniFuse and BiFuse++ as they are widely recognized baselines in 360-degree depth estimation. However, our method is designed to be model-agnostic. To demonstrate this, we've run additional experiments with HoHoNet [33] and EGformer [50] as shown in the Table in `C2`.\n\n### `Q2. Discrepancy in Abs Rel for BiFuse++:`\nThank you for catching this error. The correct value should be 0.085, not 0.858. This was a typo in our submission. We sincerely apologize for this mistake and will correct it in the final version.\n\n### Additional points:\n\n- Broader impact: Our method addresses a significant challenge in 360-degree vision and is highly adaptable due to the interchangeable teacher and student models. This flexibility has the potential to benefit a range of applications, including virtual reality, autonomous navigation, and more.\n- Efficiency: Our approach allows for better utilization of limited 360-degree data by leveraging abundant perspective data, which is particularly valuable given the scarcity of labeled 360-degree datasets.\n\nWe believe these clarifications and additional results address your main concerns. Our method, while conceptually straightforward, offers significant and consistent improvements across multiple models and datasets. We'd be happy to discuss any further questions or concerns you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GWNk56DYBT", "reviewer": "Reviewer_AagT", "summary": "This paper effectively utilizes unlabeled data by employing the SAM and DepthAnything models to generate masks and pseudo-labels respectively. When projecting data onto a cube, the authors use random rotation techniques to minimize cube artifacts, thereby enhancing the accuracy of 360-degree monocular depth estimation. Additionally, the method was tested in zero-shot scenarios, demonstrating its effective knowledge transfer.", "questions": "- The paper mentioned, \"Subsequently, in the online stage, we adopt a semi-supervised learning strategy, loading half of the batch with labeled data and the other half with pseudo-labeled data.\" However, common semi-supervised strategies typically set the ratio of labeled to total data at 1/2, 1/4, 1/8, 1/16, etc. The experiments in the paper were only conducted at a 1:1 ratio (labeled: unlabeled), and thus the performance at other ratios remains unknown.\n- The third contribution mentioned in the paper refers to \"interchangeability\" and \"This enables better results even as new SOTA techniques emerge in the future.\" This suggests that the strategy might be adaptable to a variety of models. However, the experiments were conducted only on models like UniFuse and BiFuse++ which use a dual projection fusion of Cube and ERP. Whether this approach would perform well with other transformer-based models remains an unresolved question.", "limitations": "- The paper stated, \"Cube projection and tangent projection are the most common techniques. We selected cube projection to ensure a larger field of view for each patch.\" This is merely a theoretical assertion, with no experimental evidence to prove which projection method is superior. Additionally, using cube projection directly can lead to cube artifacts. Following the suggestion in [1], setting each panoramic image to have 10 or 18 tangent images using more polyhedral faces, rather than the standard six faces, might reduce the artifacts caused by direct cube projection.\n[1]  Cokelek, M., Imamoglu, N., Ozcinar, C., Erdem, E. and Erdem, A., 2023. Spherical Vision Transformer for 360-degree Video Saliency Prediction. BMVC 2023.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper introduces a training technique for 360-degree imagery that enhances depth estimation performance by generating pseudo-labels with the DepthAnything model to leverage information from unlabeled data. Additionally, it employs the SAM model to segment irrelevant areas such as the sky in outdoor panoramic images. Furthermore, the method uses random rotation preprocessing to eliminate cube artifacts.", "weaknesses": "- The paper stated that \"As depicted in Figure 2, the rotation is applied to the equirectangular projection RGB images using a random rotation matrix, followed by cube projection. This results in a more diverse set of cube faces, effectively capturing the relative distances between ceilings, walls, windows, and other objects.\" However, from observing Figure 2, the unlabeled data seems to be entirely outdoor panoramas, which makes the mention of indoor elements such as ceilings, walls, and windows confusing. If there are indeed indoor panoramic images in the unlabeled data, what elements might the SAM model need to segment in such indoor panoramas? The description in the paper appears somewhat unclear.\n- The paper mentioned, \"We chose UniFuse and BiFuse++ as our baseline models for experiments, as many of the aforementioned methods did not release pre-trained models or provide training code and implementation details.\" However, methods such as HRDfuse [1], EGFormer [50], BiFuse and BiFuse++ [35, 36], UniFuse [11], and PanoFormer [29] have all made their source codes available, making this reason in the paper seem insufficient. Additionally, \"EGFormer is not included since its experiments are mainly conducted on other datasets and benchmarks\" appears to be an inadequate reason for not including it in the experiments.\n- In Table 3, despite introducing more unlabeled data on the Unifuse training set, i.e., SP-all (p), the performance of the method is not significantly improved. This phenomenon seems to be only effective on the BiFuse model, but not on other methods.\n- In Table 3, there is a typographical error in the recording of the Abs Rel value; it should not be 0.858. Additionally, in Table 2, \"UniFuse\" is incorrectly written as \"UniFise.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive feedback. We appreciate your recognition of our method's strengths and will address your concerns point by point:\n\n### `Q1. Indoor/Outdoor Data Clarification:`\nWe apologize for the confusion. While Figure 2 in the original paper shows outdoor examples of pseudo labels, our pseudo label dataset actually consists of both indoor and outdoor scenes. In the indoor panoramas from the pseudo label dataset, regions with infinite distance or no physical meaning, such as windows and watermarks, still appear frequently. SAM segments help mask out these undesired regions during the loss calculation in the training phase of depth estimation.\n\n### `Q2. Baseline Model Selection:`\nWe acknowledge your point about code availability, and our previous statement was imprecise. HRDFuse (https://github.com/haoai-1997/HRDFuse) does not provide any README documentation or instructions, while BiFuse (https://github.com/yuhsuanyeh/BiFuse) does not provide their training code. We attempted to reproduce their results but were unsuccessful. PanoFormer was initially included in our model list, but we removed it due to incorrect implementation and evaluation of their official code, which led to a marginal difference between their official code and the paper results.\nWe have conducted additional experiments on HoHoNet[33] and will include them in the final version. Due to the limited time available for rebuttal and the lengthy process of reproducing EGformer[50], we ran it on a subset of Matterport3D, with the size of $1/5$, and will add the results from the full set to our final version.\n| Method          | Train               | Test | Abs Rel ↓ | δ₁ ↑   | δ₂ ↑   | δ₃ ↑   |\n|-----------------|---------------------|------|-----------|--------|--------|--------|\n| HoHoNet         | M-all                | SF   | 0.095     | 0.906  | 0.975  | 0.991  |\n| HoHoNet (our)   | M-all, ST-all (p)    | SF   | 0.088     | 0.920  | 0.979  | 0.992  |\n| EGformer        | M-subset             | SF   | 0.169     | 0.764  | 0.924  | 0.972  |\n| EGformer (our)  | M-subset, ST-all (p) | SF   | 0.148     | 0.814  | 0.946  | 0.982  |\n\nWe have shown the results in the table, where both HoHoNet[33] and EGformer[50], when trained with our semi-supervised approach, demonstrate consistent improvement.\n\n### `Q3. Performance on UniFuse with SP-all (p):`\nOur improvements may appear small in some datasets quantitatively, but they consistently enhance performance across different methods and datasets. We have presented the results in the table in `Q2`, where both HoHoNet and EGformer[50] show significant improvements with our proposed method. Our analysis suggests that UniFuse's architecture is less capable of leveraging additional unlabeled data, as SP-all (p) differs more from SF. However, UniFuse with SP-all (p) shows extraordinary improvements in in-the-wild scenes, as demonstrated in Figure 2 of the PDF and Figure 6 of the original paper. We will include a discussion of this limitation and potential improvements in the final version.\n\n### `Q4. Typographical Errors:`\nThank you for catching these. The correct Abs Rel value for BiFuse++ (M-all, SP-all(p)) in Table 3 is 0.086. We will correct this and the \"UniFise\" typo in Table 2.\n\n### `Q5. Semi-Supervised Learning Ratios:`\nYour point about exploring different ratios is well-taken. We've conducted additional experiments with varying ratios in the following Table: \n| GT/Pseudo | Train              | Test | Abs Rel ↓ | δ₁ ↑ | δ₂ ↑ | δ₃ ↑ |\n|-----------|--------------------|------|-----------|------|------|------|\n| 1:1       | M-all, ST-all (p)   | SF   | 0.086     | 0.924| 0.977| 0.99 |\n| 1:2       | M-all, ST-all (p)   | SF   | 0.087     | 0.923| 0.977| 0.99 |\n| 1:4       | M-all, ST-all (p)   | SF   | 0.085     | 0.923| 0.977| 0.99 |\n\nThese results show our method is robust across different ratios starting from 1:1. We will include this analysis in the paper.\n\n### `Q6. Adaptability to Other Models`\nWe applied our methods to HoHoNet[33] (horizontal compression-based) and EGFormer[50] (transformer-based) and demonstrated significant improvements in these two non-dual projection methods, as shown in `Q2`.\n\n### `Q7. Projection Method:`\nWe appreciate your suggestion about cube projection vs. tangent projection. We will conduct comparative experiments using both projection methods, including the 10 and 18 face polyhedron suggestions from [1]. These results will be included in the final version to provide empirical evidence for our projection method choice.\n\nThank you again for your valuable feedback. We believe these additions and clarifications will significantly strengthen our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BsGqKQpBtf", "reviewer": "Reviewer_ewNL", "summary": "This paper proposes a method to improve 360 monocular depth estimation using perspective distillation and augmentation with unlabeled data. It introduces the concept of \"perspective distillation,\" which leverages the available 360 monocular depth maps and their corresponding equirectangular images to generate pixel-wise depth supervision signals. This technique helps to address the lack of ground truth depth data for training in the 360 domain. Additionally, the paper presents an unlabeled data augmentation approach that utilizes the geometric properties of 360-degree images. By exploiting the spherical geometry, the authors generate synthetic stereo pairs to augment the training dataset without requiring paired depth information. The proposed method is evaluated on benchmark datasets and achieves significant improvements in terms of depth estimation accuracy compared to existing approaches. The results demonstrate the effectiveness of perspective distillation and unlabeled data augmentation in enhancing the performance of 360 monocular depth estimation.", "questions": "Overall, the idea of this paper is interesting, allowing existing depth estimation models to benefit from unlabeled data in a semi-supervised manner, but there are concerns:\n\n1. Based on the experimental results, the author only validated the performance of this training technique on models that employ dual-projection fusion, such as Unifuse and Bifuse. There was no effective validation or analysis provided for other non-dual-projection fusion models, such as methods based on horizontal compression (e.g., HorizonNet, HohoNet) or transformer-based methods (e.g., EGFormer). However, it is essential to explicitly state the scope of applicability for this training strategy. Different training strategies, data processing methods, and even device variations can lead to unfair comparisons. The author should conduct fair experiments and comparisons under the same experimental conditions instead of directly copying the data results from the original paper. This is crucial for validating the effectiveness of this training strategy. Therefore, the author's mention of \"as many of the aforementioned methods did not release pre-trained models or provide training code and implementation details. It’s worth noting that PanoFormer [29] is not included due to incorrect evaluation code and results, and EGFormer [29] is not included since its experiments are mainly conducted on other datasets and benchmarks\" is not convincing.\n\n2. The issue of cross-domain experiments is indeed important. Given the challenges of obtaining real-world data, it would be meaningful if this framework could benefit models trained on synthetic data (e.g., training on synthetic data and testing on real data). However, the significance of this training strategy in that regard is not yet clear. Further research and experimentation are necessary to determine whether training on synthetic data using this framework can indeed yield benefits when applied to real-world data. It would be valuable to investigate the effectiveness and generalization capabilities of the trained models in real-world scenarios.\n\n3. Median alignment is currently not widely utilized in existing depth estimation methods. Based on existing findings, aligning the ground truth (GT) depth with the predicted depth can indeed lead to improved results. However, when comparing the proposed method with baseline methods like Unifuse, which explicitly state that median alignment is not used, directly comparing the results to those from the original paper leads to unfair comparisons. This raises concerns about the advantages claimed for this training framework. To ensure fair comparisons, it is important to apply the same alignment techniques consistently across all methods being compared.", "limitations": "This paper proposes an interesting solution for panoramic depth estimation task, ie.e,  perspective distillation and unlabeled data augmentation. It could contribute a lot for the community. The reviewer suggests introducing more related works, including the latest panoramic depth estimation and panoramic depth completion approaches.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative techniques: The paper introduces novel approaches, such as perspective distillation and unlabeled data augmentation, to address the challenges of 360 monocular depth estimation.\n\n2. Improved depth estimation: The proposed method achieves significant improvements in depth estimation accuracy compared to existing approaches, as demonstrated through rigorous evaluations on benchmark datasets.\n\n3. Use of unlabeled data: By leveraging unlabeled data and synthetic stereo pairs, the method reduces the reliance on paired depth information, which is often difficult to obtain in the 360-degree domain.", "weaknesses": "1. Complexity: The proposed method introduces additional complexity, such as perspective distillation and synthetic stereo pair generation, which may require more computational resources and training time.\n\n2. Dataset dependency: The effectiveness of the proposed method heavily relies on the availability and quality of the benchmark datasets used for evaluation, which may affect its generalizability to real-world scenarios.\n\n3. Limited scope: The paper focuses specifically on 360 monocular depth estimation, which may limit its applicability to other depth estimation tasks or domains, such as 360 monocular depth completion. \n\n4. Insufficient related work: Adding the latest panoramic depth estimation and panoramic depth completion methods is preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback. We appreciate your recognition of our method's strengths and innovative techniques. We'll address your concerns and questions point by point:\n\n### `Q1. Applicability to non-dual-projection fusion models:`\nWe acknowledge this limitation in our current evaluation. To address this, we've conducted additional experiments with HoHoNet[33] (horizontal compression-based) and EGFormer[50] (transformer-based):\n| Method          | Train               | Test | Abs Rel ↓ | δ₁ ↑   | δ₂ ↑   | δ₃ ↑   |\n|-----------------|---------------------|------|-----------|--------|--------|--------|\n| HoHoNet         | M-all                | SF   | 0.095     | 0.906  | 0.975  | 0.991  |\n| HoHoNet (our)   | M-all, ST-all (p)    | SF   | 0.088     | 0.920  | 0.979  | 0.992  |\n| EGformer        | M-subset             | SF   | 0.169     | 0.764  | 0.924  | 0.972  |\n| EGformer (our)  | M-subset, ST-all (p) | SF   | 0.148     | 0.814  | 0.946  | 0.982  |\n\nThese results further demonstrate our method's effectiveness across different architectures. We will include these analyses in the paper and discuss the broader applicability of our approach. Due to the limited time available for rebuttal and the lengthy process of reproducing EGformer[50], we ran it on a subset of Matterport3D, with the size of $1/5$, and will add the results from the full set to our final version. For PanoFormer, we would like to clarify that it was initially included in our model list, but we removed it due to incorrect implementation and evaluation in their official code, which led to a marginal difference between the official code and the results reported in their paper.\n\n### `Q2. Cross-domain experiments (synthetic to real):` \nThank you for the insightful suggestion. We agree that training on synthetic data for real-world scenarios is an important research direction. Our method focuses on leveraging the large amount of unlabeled real-world data, whereas synthetic data often includes depth ground truth and emphasizes domain adaptation. We will add a literature review on these two distinct research topics.\n\n### `Q3. Median Alignment and Fair Comparisons:`\nWe apologize for any confusion regarding median alignment. All methods discussed in our paper (Tables 2, 3, and 4 in the original paper) were re-trained using affine-invariant loss for relative depth training, with the same evaluation criteria applied for a fair comparison. The metric depth values presented in the paper are listed only for reference and are not used for direct comparison.\n\n### `Q4. Additional Related Work:`\nWe have conducted additional experiments, as shown in the table in `Q1`, and will include these results in the final version.\n\n### `Q5. Complexity and Computational Resources:`\nWe would like to clarify that our method does not require stereo pair generation nor synthetic generation. While our method does introduce additional complexity during training, this is a one-time cost. During the inference stage, the runtime and complexity are the same as those of the chosen SOTA 360 methods.\n\n### `Q6. Dataset Dependency and Generalizability:`\nOur proposed method demonstrates strong generalization ability across algorithms, as evidenced by the interchangeable use of SOTA 360 depth models in the table in `Q1` and in Table 3 of the original paper. Our method particularly excels in in-the-wild real-world scenarios, showing significant improvements in generalization ability on diverse data, as illustrated in Figure 2 of the PDF and Figure 6 of the original paper.\n\nThese results demonstrate our method's ability to generalize to unseen datasets, supporting its effectiveness in real-world scenarios.\n\nWe sincerely appreciate your valuable feedback, which has helped us identify areas for improvement and additional experiments. We believe these changes and clarifications significantly strengthen our paper and address the main concerns raised in your review. Thank you for your time and expertise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LDPjqsBwch", "reviewer": "Reviewer_Kp4U", "summary": "The authors propose a technique that utilizes unlabeled 360-degree data to improve previous methods, which includes two main stages: offline mask generation for invalid regions and an online semi-supervised joint training regime.  Experimental results indicate that the proposed method outperforms previous methods on the Matterport3D and Stanford2D3D datasets.", "questions": "Please see the weakness.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and clearly structured;\n\nThe performance is promising;\n\nThe experiments are thorough and the proposed method is validated on multiple datasets.", "weaknesses": "My major concern is that the technical contribution seems limited. The core idea of utilizing unlabeled data has been proposed by DepthAnything. The proposed method is more like an application of this idea to the 360 data;\n\nAs shown in Tables 2 and 3, previous 360 depth estimation methods typically use the metric loss for training. While the affine-invariant approach employed by the authors enables training on multiple datasets, it also impedes real-world applications due to the lose of metric scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for recognizing the strengths of our paper. We appreciate your feedback and would like to address your concerns:\n\n### `Q1. Technical contribution and novelty:`\nWhile we acknowledge that utilizing unlabeled data is not a new concept, our work makes several novel contributions specific to 360-degree depth estimation:\n\n- **a)**  We introduce a novel cube projection technique with random rotation preprocessing, which significantly reduces cube artifacts and improves depth estimation quality for 360-degree images with cross camera projection knowledge distillation.\n\n- **b)**  We develop a tailored masking strategy using SAM for 360-degree images, addressing the sky and watermark regions that are undesire in real-world scenes but don’t often appear in datasets.\n\n- **c)**  Our method is architecture-agnostic and can be applied to various 360-degree depth estimation models, as demonstrated in our experiments with UniFuse, BiFuse++, and additional models such as HoHoNet[33] (horizontal compression-based) and EGformer[50] (transformer-based):\n    | Method          | Train               | Test | Abs Rel ↓ | δ₁ ↑   | δ₂ ↑   | δ₃ ↑   |\n    |-----------------|---------------------|------|-----------|--------|--------|--------|\n    | HoHoNet         | M-all                | SF   | 0.095     | 0.906  | 0.975  | 0.991  |\n    | HoHoNet (our)   | M-all, ST-all (p)    | SF   | 0.088     | 0.920  | 0.979  | 0.992  |\n    | EGformer        | M-subset             | SF   | 0.169     | 0.764  | 0.924  | 0.972  |\n    | EGformer (our)  | M-subset, ST-all (p) | SF   | 0.148     | 0.814  | 0.946  | 0.982  |\n\n\n    These results show that our method generalizes well to different architectures, highlighting its broader impact on the field.\n\n### `Q2. Affine-invariant loss and metric scale:` \nWe understand your concern about the loss of metric scale. The affine-invariant loss, as you mentioned, is designed for cross-dataset training to leverage more data for real-world applications, which aligns with our approach of using both labeled and unlabeled datasets. Several previous works, such as MiDaS[4] and Depth Anything[45], also employ this training technique. To address the issue of metric scale, we will perform metric depth fine-tuning with our pretrained model in the final version. For real-world applications, we demonstrate exceptional improvement on in-the-wild scenes, as shown in Figure 2 of the PDF and Figure 6 of the original paper.\n\n### **Additionally, we'd like to highlight some key aspects of our work that underscore its significance:**\n\n1. **Bridging the gap between perspective and 360-degree depth estimation:** Our method effectively transfers knowledge from well-established perspective depth models to the 360-degree domain, addressing the scarcity of labeled 360-degree data.\n\n2. **Improved generalization:** We've conducted zero-shot experiments on unseen datasets, demonstrating our method's ability to generalize in Table 3 in the main paper. Our method especially shines in in-the-wild scenarios which can be observed in Figure 6 in the main paper and Figure 2 of the pdf.\n\n3. **Scalability:** Our approach allows for the efficient utilization of large amounts of unlabeled 360-degree data, which is increasingly available but often underutilized in current methods.\n\nWe believe these points, combined with the additional experiments and clarifications, address your concerns about the technical contribution and real-world applicability of our work. Our method not only improves upon existing 360-degree depth estimation techniques but also introduces novel concepts that can be broadly applied in the field.\n\nThank you again for your valuable feedback. We are confident that addressing these points will strengthen our paper and highlight its contributions to the field of 360-degree depth estimation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their thoughtful feedback. We are encouraged that reviewers found our work to be innovative, well-motivated, and impactful for 360° depth estimation. We appreciate the constructive comments and will address the main points below.

**Positive aspects highlighted by reviewers:**
1. Novel and effective use of perspective models for 360° depth estimation (R\_3NfA, R\_ewNL)
2. Well-motivated approach with consistent improvements (R\_3NfA, R\_F7bZ)
3. Clear presentation and thorough experiments (R\_3NfA, R\_Kp4U)
4. Practical utility demonstrated in real-world scenarios (R\_F7bZ)

**We have responded to each reviewer individually to address any comments. We would like to give a brief summary.**
1. Applicability: We test our approach on additional models (HorizonNet[33] and EGFormer[50]) to demonstrate broader applicability beyond dual-projection models.
2. Comparison to related work: We acknowledged similarities with FoVA-Depth but highlighted differences in their approach.
3. Additional experiments: We add new baselines with directly projection. We will also evaluate metric depth finetuning in the final version.
4. Limitations: We will add 360 data scarcity as a limitation and clarify potential generalization to other omnidirectional camera models.
5. Corrections: We fix inconsistencies, errors, and typos in the final version.
6. Semi-supervised ratios: We test different ratios of labeled to unlabeled data and will include results in supplementary materials.
7. Marginal improvements: Our improvements may seem small in some datasets quantitatively, but it improves consistently across methods and datasets. Moreover, on the Stanford2D3D zero-shot test, our method improves AbsRel from 0.09 to 0.082 for BiFuse++, a 8.9$\%$ relative improvement. Such improvements are considered substantial in computer vision, especially for zero-shot scenarios.

Again, we thank all reviewers and area chairs!

Best,

Authors

Meta-review: The paper proposes a simple yet effective general-purpose framework to deal with training data scarcity in 360-degree depth estimation by exploiting the knowledge of a SOTA pre-trained conventional (perspective) monocular depth estimation network. Reviewers acknowledged the proposal's relevance but raised concerns, especially regarding novelty and the experimental evaluation with only two networks, and requested several clarifications. In the rebuttal/discussion, the authors provided satisfactory responses and convincing experimental results on two additional networks (HoHoNet and EGformer), leading all reviewers towards acceptance. Although some key concepts were already proposed for another task (stereo), as noticed by reviewer 3Nfa, the AC finds the overall contribution valuable for the monocular setup and recommends acceptance. However, compared to the original submission, the paper needs a significant revision to include additional experiments and clarifications that emerged after the rebuttal.
Decision: Accept (poster)