Title: PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural rendering

neural radiance field

novel view synthesis

panorama

360-degree image
Abstract: Achieving an immersive experience enabling users to explore virtual environments with six degrees of freedom (6DoF) is essential for various applications such as virtual reality (VR). Wide-baseline panoramas are commonly used in these applications to reduce network bandwidth and storage requirements. However, synthesizing novel views from these panoramas remains a key challenge. Although existing neural radiance field methods can produce photorealistic views under narrow-baseline and dense image captures, they tend to overfit the training views when dealing with wide-baseline panoramas due to the difficulty in learning accurate geometry from sparse $360^{\circ}$ views. To address this problem, we propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas, which construct spherical radiance fields incorporating $360^{\circ}$ scene priors. Unlike generalizable radiance fields trained on perspective images, PanoGRF avoids the information loss from panorama-to-perspective conversion and directly aggregates geometry and appearance features of 3D sample points from each panoramic view based on spherical projection. Moreover, as some regions of the panorama are only visible from one view while invisible from others under wide baseline settings, PanoGRF incorporates $360^{\circ}$ monocular depth priors into spherical depth estimation to improve the geometry features. Experimental results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-art generalizable view synthesis methods for wide-baseline panoramas (e.g., OmniSyn) and perspective images (e.g., IBRNet, NeuRay).

PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas

Abstract

Achieving an immersive experience enabling users to explore virtual environments with six degrees of freedom (6DoF) is essential for various applications such as virtual reality (VR). Wide-baseline panoramas are commonly used in these applications to reduce network bandwidth and storage requirements. However, synthesizing novel views from these panoramas remains a key challenge. Although existing neural radiance field methods can produce photorealistic views under narrow-baseline and dense image captures, they tend to overfit the training views when dealing with wide-baseline panoramas due to the difficulty in learning accurate geometry from sparse 360 ◦ views. To address this problem, we propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas, which construct spherical radiance fields incorporating 360 ◦ scene priors. Unlike generalizable radiance fields trained on perspective images, PanoGRF avoids the information loss from panorama-to-perspective conversion and directly aggregates geometry and appearance features of 3D sample points from each panoramic view based on spherical projection. Moreover, as some regions of the panorama are only visible from one view while invisible from others under wide baseline settings, PanoGRF incorporates 360 ◦ monocular depth priors into spherical depth estimation to improve the geometry features. Experimental results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-art generalizable view synthesis methods for wide-baseline panoramas (e.g., OmniSyn) and perspective images (e.g., IBRNet, NeuRay). Poject Page: https://thucz.github.io/PanoGRF/ .

1 Introduction

The rise of 360 ◦ cameras and virtual reality headsets has fueled the popularity of 360 ◦ images among photographers and tourists. Commercial VR platforms, such as Matterport 2 , enable users to experience virtual walks in 360 ◦ scenes by interpolating between panoramas [17]. Wide-baseline panoramas are frequently employed on these platforms for capture and network transmission to reduce storage space and bandwidth requirements. Consequently, synthesizing novel views from wide-baseline panoramas is an essential task for providing a seamless six degrees of freedom (6DoF) experience to users.

However, synthesizing novel views from a pair of wide baseline panoramas encounters two primary challenges. First , the input views are sparse, causing existing state-of-the-art methods such as NeRF [21] to struggle in learning accurate geometry due to the shape-radiance ambiguity [37],

1

leading to overfitting of the input views. Second , certain regions in the scene may only be visible in one view but not in another, making it challenging to provide a correct geometry prior to NeRF using only multi-view stereo. State-of-the-art generalizable NeRF methods [35, 32, 20] address the overfitting problem by incorporating scene priors into NeRF; however, they are designed for perspective images and require conversion of panoramas into perspective views when applied, resulting in information loss and suboptimal performance due to the limited field of view. Although spherical radiance fields [11] have been proposed to render panoramic images based on spherical projection, they still suffer from the overfitting problem without considering the 360 ◦ scene prior. Other approaches designed for 360 ◦ view synthesis, such as multi-sphere images (MSI) [2] and mesh-based methods [17], exhibit limited expressiveness and rendering quality compared to NeRF due to the use of discretized scene representations.

In this work, we present a method that addresses the overfitting issues in spherical radiance fields by incorporating 360 ◦ scene priors. Unlike existing generalizable methods that require panoramato-perspective conversion, our approach retains the panoramic representation. Furthermore, we incorporate 360 ◦ monocular depth to alleviate the view occlusion problem.

To address the first challenge, we present a solution in the form of generalizable spherical radiance fields . To render the panorama at a new position, spherical radiance fields (Spherical NeRF) cast a ray for each pixel using spherical projection, sample points along the ray, and aggregate the colors of these points based on their density values following NeRF [21]. In our method, named PanoGRF, we incorporate 360 ◦ scene priors into Spherical NeRF. Specifically, we extract appearance and geometry features from input panoramas and estimated spherical depths through convolutions, respectively. PanoGRF accurately aligns the queried 3D points with the corresponding pixels in panoramas using spherical projection. This alignment strategy leverages the full field-of-view characteristic of panoramas and eliminates the information loss of panorama-to-perspective conversion. The local appearance and geometry features at these corresponding pixels are then aggregated and serve as the conditional input for Spherical NeRF.

To tackle the second challenge, we enhance the accuracy of depth estimation in 360 ◦ multi-view stereo by integrating a 360 ◦ monocular depth estimation network. In 360 ◦ multi-view stereo, depth inaccuracies can arise due to view inconsistencies caused by occluded regions. Inspired by the perspective method proposed in [3], we sample depth candidates around the estimated 360 ◦ monocular depth, assuming a Gaussian distribution. These depth candidates are then utilized in sphere sweeps during 360 ◦ multi-view matching. By incorporating monocular depth candidates, we can improve the accuracy of depth estimation in regions with view inconsistencies. This improvement contributes to the robustness of the geometry features employed in PanoGRF and ultimately leads to superior view synthesis performance.

In summary, we make the following contributions: 1) We propose the design of generalizable spherical radiance fields to learn 360 ◦ scene priors that alleviate the overfitting problem when dealing with wide-baseline panoramas. 2) We incorporate 360 ◦ monocular depth into the spherical multi-view stereo to mitigate the view inconsistency problem caused by occluded regions, which results in more robust geometry features, ultimately improving the rendering performance. 3) We achieve the stateof-the-art novel view synthesis performance on Matterport3D [4], Replica [28] and Residential [8] under the wide-baseline setting.

2 Related Work

In this section, we briefly review the methods of perspective view synthesis and 360 ◦ view synthesis.

2.1 Perspective View Synthesis

Recently, NeRF (neural radiance fields [21]) has become the mainstream scene representation due to its photo-realistic rendering quality for perspective view synthesis. However, given sparse input views, NeRF's per-scene optimization approach is prone to overfitting the training views and fails to learn the correct geometry due to the shape-radiance ambiguity, leading to significant floaters in novel views. Several subsequent methods [12, 6, 15, 22, 5, 34] have attempted to address this issue by incorporating depth constraints from COLMAP [26] or other regularization terms, but they still rely on per-scene optimization and lack the ability to generalize to new scenes. In contrast,

GeometryFeatures

Features of I1

Mono-guided

Local Features

Conv

360°Depth

pi

Estimator

Volume

Local Features

360°Viewpoint

Rendering

Feature

Input Views

Extractor

Novel View

I;(j = 1,2)

AppearanceFeatures

Features of I2

Spherical RadianceFields

PixelNeRF [35], GRF [30] and IBRNet [32] condition NeRF rendering with pixel-aligned appearance features to learn scene priors from large-scale datasets, enabling direct inference with input images. NeuRay [20] aggregates visibility information (obtained from multi-view stereo) and appearance features from each view to generate the density and color of 3D sample points. However, when dealing with panoramas, these methods require converting the panoramas to perspective images, leading to information loss and suboptimal performance due to limited field-of-view in each perspective view. In contrast, PanoGRF operates directly on panoramas using spherical projection, eliminating the need for the panorama-to-perspective conversion.

2.2 360 ◦ View Synthesis

Multi-sphere images (MSI) are widely adopted for 360 ◦ view synthesis. It is introduced by [2] to represent scenes with the input of omnidirectional stereo video. SOMSI [8] improved the expressive ability of MSI by incorporating high-dimensional feature layers and achieved real-time performance. However, MSI's capability is limited to narrow-baseline scenarios due to the back surface rendering issues as discussed in [18], which attempted to expand the renderable viewpoint range of MSI through the interpolation of two MSI instances. But their pure convolutional architecture only works at low resolutions. OmniSyn [17] constructs two meshes for input panoramas using 360 ◦ multi-view stereo. It warps the meshes to the target view and fuses the colors attached to the meshes with an in-painting network. This approach frequently produces ghosting artifacts due to surface-based warping and blending. 360 ◦ Roam [11] divides NeRF into multiple blocks to achieve real-time rendering of panoramas, akin to KiloNeRF [23]. But it fits a single scene without considering scene priors, making it unsuitable for wide-baseline panoramas. In contrast to the aforementioned methods, we introduce generalizable spherical radiance fields to learn both geometric and appearance priors from 360 ◦ datasets, enabling better generalization to unseen 360 ◦ scenes.

However, 360 ◦ multi-view stereo alone cannot provide a robust geometric prior, as it fails to address the occlusion issue in the wide-baseline setting. Recently, numerous researchers have focused on 360 ◦ monocular depth estimation [13, 19, 1] to mitigate spherical distortion by harnessing the fusion of equirectangular and perspective projections. Inspired by MaGNet [3], we employ 360 ◦ monocular depth prior to guide the construction of 360 ◦ cost volume, which enhances the quality of geometric features and boosts rendering performance.

3 Method

PanoGRF facilitates novel view synthesis from wide-baseline panoramas as shown in Fig. 1. Before introducing PanoGRF, we first review NeRF and its spherical variant in Sec. 3.1. We demonstrate the two important parts of PanoGRF, the generalizable spherical radiance fields, and the mono-guided spherical depth estimator in Sec. 3.2 and Sec. 3.3 respectively.

3.1 Spherical Radiance Fields (Spherical NeRF)

To render a pixel, NeRF [21] casts a ray from a given viewpoint and parameterizes the ray as p ( t ) = o + t d , t ∈ R + , where o is the camera center and d is the ray direction. N points are sampled along the ray with increasing t , we denote p i ≡ p ( t i ) , i = 1 , ..., N . NeRF utilizes an MLP to map the position p i together with viewing direction d into color c i and density σ i . The pixel color of the ray p can then be computed by:

where c ∈ R 3 is the rendered color of p , and c i ∈ R 3 is the color of the sampled point p i . w i is the blending weight of the point p i , which indicates the probability that a ray travels from the origin to t i without hitting any particle and terminates in the range ( t i , t i +1 . It is computed by w i = ∏ i -1 k =1 (1 -α k ) α i , where α i is the alpha values in the depth range ( t i , t i +1 ) , α i = 1 -exp( -σ i δ i ) and δ i = t i +1 -t i . The color c of a ray p is optimized by a photometric loss:

where c gt represents the ground truth color.

Ray-casting based on spherical projection The panorama uses the panoramic pixel grid coordinate system, while the coordinate systems in NeRF are all Cartesian. By employing spherical projection [8, 11], a pixel ( u, v ) in the panorama is firstly transformed into spherical polar coordinate ( ϕ, θ ) and subsequently into cartesian coordinate ( x, y, z ) . Detailed transformation formulas can be seen in the supplementary material. The ray direction emitted from the pixel ( u, v ) can be computed by d = R [ x, y, z ] T , where R ∈ R 3 × 3 is the panorama's camera-to-world rotation matrix.

3.2 Generalizable Spherical Radiance Fields

Under a wide baseline, Spherical NeRF tends to overfit training views and struggle to generate plausible novel views. To address this limitation, we draw inspiration from the generalizable NeRF approaches [35, 32, 20] designed for perspective images. In our work, we propose to incorporate 360 ◦ scene priors into Spherical NeRF by aggregating local features from input panoramas.

Alignment based on spherical projection Panoramas can be converted into perspective images, and then generalizable NeRF methods for perspective images can be applied to novel 360 ◦ view synthesis. However, when projecting the 3D sample point p i onto the source perspective view, it may fall outside the image borders or even be located behind the source perspective camera (with a z -depth < 0) due to the limited field-of-view. This can introduce errors in the aggregation of features and result in poor rendering results. To overcome this problem, we directly align p i with the corresponding pixels in panoramas using spherical projection, enabling us to leverage the full fieldof-view characteristic of panoramas. We first transform p i into the camera's Cartesian coordinate system ( x, y, z ) of the j -th panoramic view I j . Subsequently, ( x, y, z ) is converted into spherical polar coordinate ( θ, ϕ, t ) ( t ∈ R + indicates the spherical depth of p i for I j ) and finally transformed into the panoramic grid coordinate ( u, v ) . The specific formulas for these transformations can be found in the supplementary material.

Appearance and geometry feature aggregation We follow NeuRay [20] to aggregate appearance and geometry features. But differently, we take panoramas and spherical depth (radial distance from the camera center) as input instead of perspective images and z -depth. PanoGRF respectively extracts appearance feature W j from the j -th panoramic view ( j = 1 , 2 ) and geometry feature G j from

360°MonocularDepthEstimation

ReferenceView

360°MonocularDepth

CostVolume(u, v, i)ref =Similarity ( ,

360°Monocular

DepthEstimator

：Reference&SourceViews

SourcePose

Sampling

(u, U)sre

CNN

Sphere Sweep

Decode

(u,v)ref

Pi，(i1,2,,D)

Feature Maps

360° Depth

360°Multi-viewStereo

the spherical depth using convolutions. Details for getting the spherical depth will be introduced in Sec. 3.3. For a given 3D sample point p i , we project it onto the panoramic grid coordinate ( u, v ) , and extract the local appearance feature f i,j = W j ( u, v ) and the local geometry feature g i,j = G j ( u, v ) at ( u, v ) . The local geometry feature g i,j is decoded into the visibility v i,j (demonstrated below) using an MLP. The local appearance feature f i,j and visibility v i,j of the j -th panoramic view are aggregated for p i with an aggregation network F :

According to the local geometry feature g of an input panorama, we predict the visibility function v ( t ) to indicate the probability that a point at spherical depth t (instead of z -depth in NeuRay) is visible for the input panorama. v ( t ) ( v ( t ) ∈ [0 , 1] ) is represented as v ( t ) = 1 -o ( t ) , where o ( t ) is the occlusion probability. To parameterize o ( t ) , we employ a mixture of N l logistic distributions:

where µ k , σ k and m k are the mean, standard variance, and blending weight of the k -th logistic distribution respectively. ∑ N l i m k = 1 and S ( · ) denotes a sigmoid function. The parameters [ µ k , σ k , m k ] are decoded from the local geometry feature vector g by an MLP. For each 3D sample point p i , we compute its spherical depth t i for j -th input view based on spherical projection and obtain the visible probability v i,j = v j ( t i ) of p i for the j -th panoramic view. Then we aggregate v i,j into f i by Eq. 3 and decode f i into the color and density of p i .

3.3 Mono-guided Spherical Depth Estimator

To obtain the visibility in Sec. 3.2, we need to predict the spherical depth of each input view. The process is illustrated in Fig. 2.

360 ◦ multi-view stereo (MVS) With the input of multi-view panoramas, we estimate the spherical depth of the reference view using 360 ◦ multi-view stereo similar to previous methods [17, 14]. Firstly, we extract the image features of the reference and source views with an image encoder. For each pixel ( u, v ) ref of the reference view, assuming its depth is t i , we find its corresponding pixel ( u, v ) src in

the source view according to spherical projection. We uniformly sample D depth candidates for t i (with i = 1 , 2 , ..., D ) without considering the monocular depth prior, which will be introduced later. Next, we compute the similarity between the local features at ( u, v ) ref and ( u, v ) src , considering it as the value of the cost volume at ( u, v, i ) . Assuming the length of the local feature vector is F , this process results in a 4D cost volume V ∈ R H 4 × W 4 × D × F after D sphere sweeps, where H 4 and W 4 respectively represent the height and width of the feature maps for the reference view. Subsequently, the dimension of the cost volume is reduced to H 4 × W 4 × D through a 3D CNN. Lastly, the processed cost volume is decoded into 360 ◦ depth using several convolution layers.

Mono-guided spherical depth sampling Under the wide-baseline setting, some areas might be visible from one view but occluded from another, in which case 360 ◦ MVS may struggle to produce accurate depth estimates. To address this challenge, we leverage the 360 ◦ monocular depth [13] to guide the depth sampling of 360 ◦ MVS, inspired by perspective methods [36, 31, 3].

For each pixel ( u, v ) in the panorama, we denote its estimated spherical depth from the monocular depth network as µ u,v . We generate depth candidates in the vicinity of the monocular depth using a Gaussian distribution assumption. Specifically, a search space for each pixel in the panorama is defined as [ µ u,v -βσ, µ u,v + βσ ] where β and σ serve as hyperparameters and σ represents the standard deviation. This search space is divided into N mono bins, ensuring that each bin has the same probability mass. We select the midpoint of each bin as the depth candidate. Consequently, the k -th depth candidate for pixel ( u, v ) is defined as t u,v,k = µ u,v + b k σ, k = 1 , 2 , ..., N mono . b k represents the offset corresponding to the k -th bin. Similar to [3], we calculate b k as the following:

where P ∗ = erf( β √ 2 ) denotes the probability mass covered by the search space [ µ u,v -βσ, µ u,v + βσ ] , Φ -1 ( · ) is the quantile function of standard normal distribution, erf is the error function. Considering errors may occur in the 360 ◦ monocular depth, we use a mixture of N mono monocular depth candidates and N uni uniform depth candidates sampled from a uniform distribution. By constructing D = N uni + N mono sphere sweeps, we obtain a new spherical cost volume. The monocular depth guidance enables us to extract more reliable geometry features employed in PanoGRF.

4 Experiment

4.1 Metrics and Datasets

PSNR, SSIM [10], LPIPS [38] and WS-PSNR [29] are used as evaluation metrics. We conduct experiments on Matterport3D [4], Replica [28], and Residential [8]. For Matterport3D and Replica, we leverage HabitatAPI [25] to generate the 256 × 256 perspective images (representing the six sides of cube-maps) and stitch them into a 512 × 1024 panorama. We use panoramic sequences with a length of 3. The middle view is rendered based on the first and last views. We conduct comparative experiments on Matterport3D under fixed camera baselines of 1.0, 1.5, and 2.0 meters, where the camera baseline refers to the distance between the camera centers of the first and last views. The baselines used for Residential and Replica are approximately 0.3 and 1.0 meters, respectively.

4.2 Implementation Details

We set N mono = 5 , N uni = 59 , and N l = 2 . D = N mono + N uni is 64. σ used in mono-guided depth sampling is set to 0.5 and β is set to 3 . Additional details regarding network architecture can be found in the supplementary material.

4.3 Comparisons

We compared PanoGRF with S-NeRF (spherical variant of NeRF [21]), IBRNet [32], NeuRay [20], and OmniSyn [17]. We trained S-NeRF from scratch, as it is a per-scene optimization method. Other methods are pre-trained on Matterport3D and tested on unseen testing scenes. We fed the original cube maps of panoramas (in Matterport3D and Replica) to the perspective methods (IBRNet and NeuRay). As there are only panoramas in ERP (equirectangular projection) format for Residential,

cube maps were converted from panoramas. To render a panoramic view for evaluation, we cast rays in IBRNet and NeuRay with spherical projection (refer to Sec. 3.1) and aggregate features with their original perspective projection. We use NeuRay ∗ and IBRNet ∗ to denote variants of NeuRay and IBRNet which render panoramas. We did not compare with methods based on multi-sphere images (MSI) [2, 8] as they can only render novel views within the smallest sphere of MSI, which are unsuitable for wide-baseline panoramas. In the supplementary materials, we provide additional analysis on per-scene fine-tuning of PanoGRF and also compare it with Cross Attention Renderer [7].

Sample 1 (Replica)

Sample 2 (Matterport3D)

IBRNet

∗

[32]

OmniSyn [17]

NeuRay

∗

[20]

PanoGRF

Ground Truth

Analysis On Matterport3D and Replica, we quantitatively compared PanoGRF with the baseline methods. The results can be found in Table. 1 and Table. 2. Almost all the metrics show that PanoGRF outperforms other methods significantly. We show the qualitative comparisons in Fig. 3. IBRNet* and NeuRay* aggregate features based on perspective projection. Due to the limited field of view, 3D sample points are often projected somewhere behind the source perspective camera ( z -depth<0) or outside the image border during pixel alignment. In this case, the aggregated features are incorrect, causing bad rendering results. As for OmniSyn, its rendering outputs exhibit ghosting artifacts, particularly notable at the boundaries of the two sofas in Sample 1 of Fig. 3. Unlike these methods, PanoGRF is a generalizable spherical NeRF which is more appropriate for panoramic images due to the use of spherical coordinates. It achieves superior accuracy in synthesizing object boundaries and demonstrates better pixel alignment, as evident from the results showcasing the sofas in Sample 1 and the desk in Sample 2 of Fig. 3). On Residential, we also compared PanoGRF with S-NeRF, IBRNet*, NeuRay*. The results of S-NeRF show severe floaters (See the ceilings of Sample 1 in Fig. 4). As

Sample 1 (Residential)

Sample 2 (Residential)

S-NeRF [21]

IBRNet

∗

[32]

NeuRay

∗

[20]

PanoGRF

Ground Truth

converting panoramas into perspective views brings information loss, the perspective generalization methods IBRNet* and NeuRay* have notable artifacts. In contrast, PanoGRF demonstrates excellent generalization performance on the Residential dataset, as shown in Fig. 4. Additional qualitative comparisons can be seen in the supplementary material.

4.4 Ablation Studies

We conducted ablation studies to evaluate the effectiveness of the key components in mono-guided spherical depth estimator: 360 ◦ monocular depth and multi-view stereo. We provide qualitative comparisons of 360 ◦ depth estimation quality on Matterport3D in Fig. 5, and analyze the view synthesis quality on Replica in Table. 3. More results can be found in the supplementary material.

360 ◦ monocular depth We removed the 360 ◦ monocular depth guidance and kept the total number of depth candidates unchanged ( N uni = 64 ) in the depth sampling. Under three camera baselines, all the metrics dropped after removing the 360 ◦ monocular depth. Especially, under the camera baseline of 2.0 meters, WS-PSNR dropped by 1.33dB without monocular depth. This removal resulted in error-prone object boundaries, such as the bedside and wooden pillar (shown in Fig. 6). The guidance of monocular depth can give more accurate depth candidates and mitigates the view inconsistency problem, which 360 ◦ multi-view stereo cannot handle alone.

360 ◦ multi-view stereo We removed 360 ◦ multi-view stereo and used 360 ◦ monocular depth directly as the input of the geometry feature extraction in PanoGRF. In this case, WS-PSNR dropped

more than 1.0 dB under the camera baselines of 1.5 and 2.0 meters. And LPIPS became 0.165 from 0.134 under the baseline of 1.0 meters. Without multi-view stereo, multi-view consistency of geometry is not guaranteed, degrading the quality of novel view synthesis results. In Fig. 6, we observed the presence of ghosting artifacts near the left wooden pillar when relying solely on 360 ◦ monocular depth.

5 Conclusion

In this paper, we propose PanoGRF, generalizable spherical radiance fields for wide-baseline panoramas. We aggregate the appearance and geometry features from input panoramas through spherical projection to avoid panorama-to-perspective conversion. To address the view inconsistency problem, we use 360 ◦ monocular depth to guide the spherical depth sampling and obtain a more accurate geometric prior for the spherical radiance fields. Experiments on multiple datasets verify that PanoGRF can render high-quality novel views given wide-baseline panoramas.

Limitations Similar to other generalizable radiance fields, PanoGRF suffers from the issue of rendering speed. Additionally, since we only train PanoGRF on indoor data due to the lack of large-scale outdoor 360 ◦ datasets, its generalization performance can be limited when applied to outdoor scenes with significantly different depth scales.

Societal impact This method may be used to synthesize fake or deceptive panoramas, combined with generative methods.

Mono only

MVS only

full

Ground Truth

References

Hao Ai, Zidong Cao, Yan-Pei Cao, Ying Shan, and Lin Wang. Hrdfuse: Monocular 360deg depth estimation by collaboratively learning holistic-with-regional depth distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13273-13282, 2023.

Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin. Matryodshka: Real-time 6dof video view synthesis using multi-sphere images. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I , pages 441-459. Springer, 2020.

Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Multi-view depth estimation by fusing single-view depth probability with multi-view geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2842-2851, 2022.

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV) , 2017.

Zheng Chen, Chen Wang, Yuan-Chen Guo, and Song-Hai Zhang. Structnerf: Neural radiance fields for indoor scenes with structural hints. arXiv preprint arXiv:2209.05277 , 2022.

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. CoRR , abs/2107.02791, 2021.

Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Tewodros Habtegebrial, Christiano Gava, Marcel Rogge, Didier Stricker, and Varun Jampani. Somsi: Spherical novel view synthesis with soft occlusion multi-sphere images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15725-15734, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition , pages 2366-2369. IEEE, 2010.

Huajian Huang, Yingshu Chen, Tianjian Zhang, and Sai-Kit Yeung. 360roam: Real-time indoor roaming using geometry-aware 360 ◦ radiance fields. arXiv preprint arXiv:2208.02705 , 2022.

Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5885-5894, 2021.

Hualie Jiang, Zhe Sheng, Siyu Zhu, Zilong Dong, and Rui Huang. Unifuse: Unidirectional fusion for 360 panorama depth estimation. IEEE Robotics and Automation Letters , 6(2):15191526, 2021.

Sameh Khamis, Sean Fanello, Christoph Rhemann, Adarsh Kowdle, Julien Valentin, and Shahram Izadi. Stereonet: Guided hierarchical refinement for real-time edge-aware depth prediction. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 573-590, 2018.

Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. CoRR , abs/2112.15399, 2021.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, and Ruofei Du. Omnisyn: Synthesizing 360 videos with wide-baseline panoramas. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) , pages 670-671. IEEE, 2022.

Jisheng Li, Yuze He, Jinghui Jiao, Yubin Hu, Yuxing Han, and Jiangtao Wen. Extending 6-dof vr experience via multi-sphere images interpolation. In Proceedings of the 29th ACM International Conference on Multimedia , pages 4632-4640, 2021.

Yuyan Li, Yuliang Guo, Zhixin Yan, Xinyu Huang, Ye Duan, and Liu Ren. Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2801-2810, 2022.

Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7824-7833, 2022.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. CoRR , abs/2112.00724, 2021.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14335-14345, 2021.

Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12892-12901, 2022.

Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Johannes L. Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 4104-4113, 2016.

Stefan Schubert, Peer Neubert, Johannes Pöschmann, and Peter Protzel. Circular convolutional neural networks for panoramic images and laser data. In 2019 IEEE intelligent vehicles symposium (IV) , pages 653-660. IEEE, 2019.

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019.

Yule Sun, Ang Lu, and Lu Yu. Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE signal processing letters , 24(9):1408-1412, 2017.

Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15182-15192, 2021.

Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. arXiv preprint arXiv:2303.16196 , 2023.

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4690-4699, 2021.

Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. SynSin: End-to-end view synthesis from a single image. In CVPR , 2020.

Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. arXiv preprint arXiv:2303.07418 , 2023.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2021.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206.00665 , 2022.

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

A More Results of Qualitative Comparisons

Qualitative comparisons with baseline methods on Replica and Matterport3D can be seen in Fig. 7 and Fig. 8, respectively.

OmniSyn

IBRNet

NeuRay

PanoGRF

Ground Truth

OmniSyn

IBRNet

NeuRay

PanoGRF

Ground Truth

B Spherical Projection

Equirectangular-to-spherical The transformation from the equirectangular image coordinate system to the polar coordinate system is defined as:

where ϕ , θ represent the latitude and longitude of the sphere, u , v represent the rows and columns of the panorama, H and W represent the height and width of the panorama respectively.

Spherical-to-cartesian The transformation from the polar coordinate system to the 3D Cartesian coordinate system is:

Cartesian-to-spherical The camera cartesian coordinate ( x, y, z ) is transformed into the polar coordinate ( θ, ϕ, t ) ( t ∈ R + denotes its spherical depth in view I j ) by:

NeuRay

PanoGRF

Ground Truth

Spherical-to-equirectangular The spherical polar coordinate ( θ, ϕ, t ) is turned into the equirectangular image coordinate ( u, v ) by the inverse process of Eq. 6.

C More Ablation Studies for 360 ◦ View Synthesis

We conducted ablation studies on Matterport3D, and the results are shown in Table. 4. In the "w/o appearance feature" ablation study, we replaced the appearance feature vector with a zero vector to disable the appearance feature while keeping other modules unchanged. We found that the model without appearance features loses its ability to infer the color of novel views entirely, as the generalizable renderer heavily relies on appearance cues from input views. In the "w/o geometry feature" ablation study, we replaced the geometry feature vector with a zero vector to disable the geometry feature while keeping other modules unchanged. We observed that although the model can still infer normal results, its performance is significantly worse than the original (full) model.

D Comparisons with NeuRay [20] Given Multi-view Inputs

Our method is not limited to two panoramas. For instance, when rendering a test view in the renderer module, we use N input panoramas as reference images, and the renderer does not need to be modified. In the 360 ◦ spherical depth estimator module, for each reference image, we use the other N -1 input panoramas as source images. We average the multiple cost-volumes obtained during 360 ◦ multi-view matching process between the reference image and each source image. The rest is unchanged. In this way, our method can be applied to the multi-view panoramic inputs.

To further verify the effectiveness of our method, we conducted comparative experiments with NeuRay using multiple panoramic image as inputs. We placed the four input viewpoints at the

NeuRay

PanoGRF

Ground Truth

NeuRay

PanoGRF

Ground Truth

corners of a horizontal square and tested the rendering performance at the center viewpoint and other viewpoints located at -0.4, -0.2, -0.1, -0.05, 0.05, 0.1, 0.2, and 0.4 meters in the vertical direction from the center viewpoint. The diagonal length of the square is 1.0 meters. Table. 10 and Fig. 9 present the quantitative and qualitative comparison results between PanoGRF and NeuRay. As shown, PanoGRF still largely outperforms NeuRay with multiple panoramic inputs.

This experiment is added during the rebuttal period. Due to the limited time, we trained PanoGRF only for 20k iterations and NeuRay for 80k iterations. The learning decay strategies are similar to the setting of two views.

E Comparisons with NeuRay [20] beyond Camera Baseline and Failure Case

We conducted an additional experiment where novel views were generated at positions 0.25 and 0.5 meters above the middle point between two input viewpoints. The input viewpoints are 2.0 meters apart. We compared the quantitative and qualitative results of our method with NeuRay's as shown in Table. 11 and Fig. 10. PanoGRF consistently surpassed NeuRay's performance, indicating its capacity to yield superior results beyond the camera baseline.

We also present the failure case of PanoGRF under such condition in Fig. 11. In some viewpoints, a previously occluded area may be visible and since this area has not been seen in either of the two

existing viewpoints, the synthesis of this area is not effective. The area, which has not been seen in either of the existing viewpoints may be able to be filled in by combining with the existing diffusion generative approach. This is the next direction we plan to investigate.

F Experiments for Mono-guided Spherical Depth Estimator

F.1 Ablation Studies for Mono-guided Spherical Depth Estimator

To further validate the effectiveness of the key components, namely 360 ◦ multi-view stereos and 360 ◦ monocular depth, we conducted ablation studies specifically focused on spherical depth estimation. For evaluation purposes, we selected three commonly used metrics: L 1 , L 2 , and RMSE. Additionally, we also used WSL 1 , WSL 2 , and WS-RMSE as metrics, which incorporate weighted latitudes of equirectangular images to simulate WS-PSNR [29]. This approach aims to mitigate the impact of equirectangular projection distortion. We selected the first 1000 panorama pairs from Matterport3D as our test data, with a camera baseline of 1 meter. For the evaluation, we considered depth values within the range of [0 . 1 , 10] as valid.

The quantitative results of our ablation studies are presented in Table 5, while the qualitative results can be observed in Fig. 12. The experiments clearly demonstrate the importance of each module in achieving accurate depth estimation. Removing either the 360 ◦ multi-view stereo or the 360 ◦ monocular depth significantly reduces the depth accuracy. Using only 360 ◦ monocular depth does not guarantee multi-view consistency, resulting in potential discrepancies between the predicted scale and the ground truth in certain regions. The occlusion problem poses a challenge for using only 360 ◦ MVS, particularly at the boundaries of objects. Consequently, the depth predictions in these regions are inaccurate and lack detail.

F.2 Different Backbones for 360 ◦ Multi-view Stereo

Introducing 360 ◦ monocular depth to 360 ◦ multi-view stereo does result in an increase in the number of model parameters. However, it is important to note that the improvement in depth accuracy is not attributed to the increase in parameters. In our experiments, we increased the model size of 360 ◦ MVSNet by using larger backbones, specifically ResNet [9].

From Table. 6, we found that L 2 and RMSE of 360 ◦ MVSNet(ResNet-101) are still far inferior to those of 360 ◦ MVSNet(ResNet-18) together with the 360 ◦ monocular depth network [13]. This observation suggests that simply increasing the model size does not effectively address the view inconsistency problem in the wide-baseline setting. On the other hand, the introduction of 360 ◦ monocular depth provides a qualitative improvement to the accuracy of 360 ◦ multi-view stereo. By incorporating monocular depth information, the model gains additional cues that help mitigate the view inconsistency issue and improve depth estimation performance. Furthermore, when larger backbones were used as replacements, it was found that the performances of 360 ◦ MVS+Mono deteriorated. This could be attributed to the excessive model parameters leading to overfitting.

F.3 Different Hyperparameters for Mono-guided Spherical Depth Estimator

We conducted evaluations to assess the impact of different values for σ (standard deviation) and N mono (number of monocular depth candidates) on mono-guided spherical depth sampling. The results are presented in Table 7 and Table 8.

RGB

Mono only

MVS only

full

Ground Truth

Cross Attention Renderer

PanoGRF

Ground Truth

Cross Attention Renderer

PanoGRF

Ground Truth

8

()

The reliability of 360° monocular depth estimation is not perfect. Therefore, our paper employs uniform sampling to compensate for the remaining depth candidates. In the ablation experiments, we consistently maintain N mono + N uni = 64 , where N uni represents the sample number of uniform distribution. We discovered that the configuration yields the best results with regard to the metrics of WS-L1, WS-L2, and WS-RMSE.

G Comparisons with Cross Attention Renderer [7]

The Cross Attention Renderer (CAR) 3 is a method that operates on a wide-baseline perspective pair. We divided the two panoramas into cube maps and utilized the corresponding sides of the cube maps as inputs for CAR. Specifically, we rendered the corresponding side of the cube maps at the middle viewpoint. For example, we input the left side of the cube maps and generate the left side of the cube maps at the intermediate viewpoint as the output. We repeated this process for all six corresponding pairs of cube maps. In contrast, our method directly takes two panoramas as input and performs ray casting based on perspective projection. We then render the results for each side of the cube maps at the intermediate viewpoint, preserving the panoramic nature of the input.

We conducted quantitative comparative experiments on Matterport3D and Replica. The qualitative comparisons with CAR on Replica and Matterport3D can be seen in Fig. 13 and Fig. 14. The results clearly demonstrate that PanoGRF significantly outperforms CAR in terms of rendering quality. CAR suffers from limitations associated with its input field-of-view (FoV), particularly in edge regions that are only visible from a single perspective view. CAR relies on a pure stereo-matching method for geometric estimation, which leads to suboptimal rendering performance, as evidenced by the results presented in Table. 9. In contrast, PanoGRF is specifically designed to handle full FoV inputs, and it mitigates the issue of view inconsistency by incorporating the 360 ◦ monocular depth network.

H Fine-tuning of PanoGRF

We conducted per-scene fine-tuning for NeuRay [20] and PanoGRF on the first test scene of Matterport3D with baselines of 1.0 and 0.2 meters, respectively. The general renderers were fine-tuned for 10k iterations, and the quantitative results are presented in Table.12. Under the baseline of 1.0 meters, the general renderer of PanoGRF, denoted as PanoGRF-gen, achieved the best performance. NeuRay-ft, the fine-tuned renderer of NeuRay, underwent fine-tuning using RGB loss and depth consistency loss, following the methodology described in their original paper [20]. PanoGRF-ft was fine-tuned with only RGB loss. However, the results of NeuRay-ft and PanoGRF-ft were inferior to NeuRay-gen and PanoGRF-gen, respectively. This suggests that fine-tuning under a wide-baseline setting does not improve the performance of general renderers. It is likely that fine-tuning with a wide baseline leads to overfitting. Even with the introduction of a depth uncertainty loss [24] by supervising the renderer depth of PanoGRF with predicted spherical depths during the fine-tuning process, the results of PanoGRF-ft remained inferior to those of the general renderer of PanoGRF. On the other hand, the performance of PanoGRF-ft under the baseline of 0.2 meters was slightly better than that of PanoGRF-gen. However, adding the depth loss [24] was still unable to improve PanoGRF-gen when fine-tuning. Inaccurate predicted depths in certain regions may be the cause, misleading the estimation of NeRF's geometry and thereby reducing the rendering performance.

Additionally, NeuRay-ft was still inferior to NeuRay-gen under the narrow baseline. This may be attributed to the limited field-of-view, which can result in the aggregation of incorrect features. When projecting 3D sample points onto other source perspective views (cube-maps), these points may fall outside the image borders of the source perspective view or be located behind the source perspective camera.

I Quantitative Comparisons with Baseline Methods for Narrow-baseline Panoramas

We compared PanoGRF with baseline methods under the baseline of 0.2 and 0.5 meters on Matterport3D. As shown in Table. 13, the quantitative results demonstrate that PanoGRF consistently outperforms all the baseline methods under the baseline of 0.2 and 0.5 meters. These findings indicate that our method is applicable to both wide-baseline and narrow-baseline panoramas. In comparison to generalizable methods designed for perspective views, our method is particularly well-suited for synthesizing panoramic views by leveraging the aggregated features based on spherical projection.

J More Details of PanoGRF

J.1 Renderer

J.1.1 Training

PanoGRF employs the Adam optimizer [16] with an initial learning rate of 4.0e-4. The pre-training process of PanoGRF was conducted on an A100 GPU for 100k iterations, which required approximately two days. The learning rate is halved every 20k iterations, and a batch size of 512 was used during training.

To streamline the experiments, we train PanoGRF and all the baselines under a 1.0m baseline for comparisons and test them under various baselines. In the ablation studies, the spherical depth estimators (Mono, MVS, and fusion) are pre-trained under a fixed 1.0m baseline. The general renderers in the ablation studies are pre-trained under corresponding baselines when tested under various baselines. We discovered that if we pre-trained the general renderers under a 1.0m baseline and tested them under larger baselines, there was no noticeable difference in performance between them.

J.1.2 Architecture

We adopted a coarse-to-fine sampling approach, similar to NeRF [21], and sampled 64 points in both phases. We followed a similar architecture as NeuRay [20] to build our renderers. The coarse and fine renderers share the same image encoder, geometry feature extractor, and visibility encoder. But they have different distribution decoders and aggregation networks F , similar to NeuRay. During training, the weights of the 360 ◦ spherical depth estimator are fixed due to GPU memory limitations. Our image encoder, visibility encoder, distribution decoder, and aggregation network are implemented similarly to NeuRay, except for the padding mode. In the convolution layers of the image encoder and visibility encoder, we employ circular padding horizontally and zero padding vertically instead of direct zero padding. This is done to adapt to equirectangular image inputs and simulate Circular

Spherical Depth

Conv 3 × 3(32)

ResBlock(32)

Conv 1 x 1(32)

Image Encoder

Image Feature

十

Conv 3 × 3(32)

ResUNet

ResBlock(32)

Conv 1 × 1(32)

Geometry

Feature

CNNs [27]. The circular padding helps account for the continuity of pixels at the leftmost and rightmost edges of equirectangular images, which are neighbors, while the top and bottom edges are not. The distribution network consists of 5 sub-networks, each with 3 fully connected layers. The appearance feature extractor is a ResNet [9] which contains 13 residual blocks and outputs the appearance feature map with 32 channels. The architecture details of the geometry feature extractor are shown in Fig. 15. The image encoder in the geometry feature extractor contains 14 residual blocks. The spherical depth input for the geometry feature extractor is first downsampled to 1/4 of its original resolution and then fed into the extractor to reduce GPU occupancy.

J.2 Spherical Depth Estimator

J.2.1 Training

We initially train the 360 ◦ monocular depth network and then freeze its weights when training the remaining components of 360 ◦ MVSNet. The Adam optimizer is employed with a fixed learning rate of 0.0001. Both the 360 ◦ monocular depth network and 360 ◦ MVSNet are trained for 100k iterations, which required approximately one day on a single V100 GPU. The batch size is set to 2.

J.2.2 Architecture

The 360 ◦ monocular depth network [13] and 360 ◦ MVSNet utilize ResNet-18 as the feature extractor. The 3D CNN regularization network is composed of 3 downsampling and 3 upsampling blocks, similar to [17]. The depth decoder consists of 2 convolution blocks. The feature map obtained from the middle layer in the 360 ◦ monocular depth network is concatenated with the regularized spherical cost volume and then decoded into 360 ◦ depth by the depth decoder. In the multi-view matching process, we compute the similarity by subtracting the feature vectors.

K Datasets

For Matterport3D, we split the training and testing set following SynSin [33]. The first 10 scenes of the test set are used for evaluation. In the case of the Replica dataset, we render a total of 18 scenes for evaluation. Additionally, we utilize the Residential dataset provided in [8], which comprises three scenes. For this dataset, we select the first and last panorama as the input views.

L Training Details of Baseline Methods

We trained NeuRay [20] and IBRNet [32] for 400k and 250k iterations, respectively. Spherical NeRF (S-NeRF) [21] underwent training for 2000 epochs, equivalent to approximately 256k iterations with the batch size of 4096. For OmniSyn [17], the in-painting network was trained for 50k iterations, which took approximately 3 days on a TitanRTX GPU. The depth estimator of OmniSyn was trained for 100k iterations. Due to limitations in GPU memory, OmniSyn was trained at a resolution of 256 × 256 , and its output was resized to 512 × 1024 for evaluation purposes. CAR [7] was trained for 300k iterations.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey describe a novel method for panoramic view synthesis given wide-baseline panoramas as input.  They first obtain depth maps for the input panoramas use a custom spherical depth estimation method guided by a pre-trained monocular depth estimator.  They then extract geometry and appearance features from the input panoramas.  To render a new panorama, they sample points along the viewing rays,  project them into the feature grids, aggregate features using an estimated visibility term, decode the aggregated features into color+density, and compute final color using volume rendering.  They outperform recent generalizable view synthesis methods on Matterport3D, Replica, and Residential datasets.\n\nSTRENGTHS:\nThey combine many ideas from prior work on perspective and spherical view synthesis but synthesizing these ideas into a novel approach.  The closest work is probably NeuRay, which is a generalizable radiance field representation for perspective images.  The main differences to NeuRay are that they work in spherical coordinates and they use monocular depth estimation to contribute to the visibility prediction and depth-guided sampling.  \n\nTheir evaluation is thorough and includes a comparison to a reasonable selection of recent view synthesis methods, including some designed for panoramas and some adapted to panoramas by applying the method to each side of a cube map.  Quantitatively they show a large improvement in rendering quality and this is also evident from the example images in the figures and the videos in the supplementary.  \n\nThey also include an ablation study in which they verify the usefulness of including 360 monocular depth and multi-view stereo in the method.\n\nAnother contribution is the insight (reported in the supplemental) that fine-tuning did not lead to an improvement in rendering quality in their experiments, contrary to what was reported by NeuRay for example.  Instead fine-tuning led to overfitting.  They hypothesize that this is due to the use of wide-baseline images.  This is an interesting result and points to potential future work to overcome this limitation.\n\nWEAKNESSES:\nIn many ways the work is an adaptation of NeuRay to the spherical case.  It could be argued that the work is somewhat incremental in that sense.  However, their results for panoramic view synthesis are excellent and there is not a lot of previous work on wide-baseline panoramic view synthesis.\n\nBecause the name includes GRF, I would think it appropriate to mention and cite GRF which is an earlier general radiance field approach which is a precursor to NeuRay:\n\nTrevithick, A., & Yang, B. (2021). GRF: Learning a general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15182-15192).\n\nAnother weakness is that the provided video only shows planar movement.\n\nQUESTIONS:\nThe videos only demonstrate camera movement along a line with no vertical movement.  Does the method support complete 6DOF motion or is limited to planar motion?\n\nLIMITATIONS:\nThe discussion of limitations is rather brief.  They could explore failure cases more and highlight cases where the view synthesis result could be improved.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackels large-baseline (up to 2 meters) multi-view stereo for panorama images. Extending NeuRay framework, the density and raidiance field are estimated based on the feature extracted from nearby views following panorama projection. To predict stereo depth as geometric feature, this work propose to use 360 monocular depth estimation model to sample more sphere sweeps close to the surface. All the features are extracted or estimated by 2D/3D CNN and can be trained from many scenes for generalization.\n\nSTRENGTHS:\nThe quantitative and qualitative results are better than the compared baselines.\n\nWEAKNESSES:\n**Adapting from perspective cubemap to equirectangular projection is not a big contribution.**\nOne of the main claimed disadvantage of some of the baseline methods (i.e., IBRNet and NeuRay in this work) is that their original codebases work on perspective instead of panoramic imagery (L36-38). However, adapting perspective projection to equirectangular projection is trivial. Some common adaptation like circular padding is easy to add as well. Many deep-learning based models also show promising quality by adapting CNN on equirectangular panorama like LayoutNet[Zou et al] and DuLa-Net[Yang et al], just name a few. With these simple adaptation, the baseline methods can also be directly trained on equirectangular panorama without the claimed disadvantage.\n\n**Some design choices are not well justified and discussed.** Please refer to question 1,2,3 for detail.\n\n**Paper writing**\n- Sec.3.1 L122-123. $w_i$ is not the accumulated transmittance. Only the $\\prod_{k=1}^{i-1} (1 - \\alpha_k)$ is.\n- The $w$ is duplicated in Eq.1 and Eq.4 but they are totally different things. I suggest using different notations.\n- Table.1 and Table.2. It's better to directly use a table footnote describing the \"*\" means the models are trained with perspective cubemap projection and evaluated with equirectangular projection.\n- L140. \"When\" should be \"when\".\n\nQUESTIONS:\n1. Why constraint the proposed method on aggregating only 2 panorama views (Eq.3)? The original NeuRay have tried many different number of working views and find 8 nearyby views achieves saturated quality. Experiment to justify the design choice in this work is missing.\n2. The search space of the stereo depth candidacy is scene dependent and crucial. How is the $\\beta$ in L215 determined? If applying the proposed method to other datasets, is there any rule of thumb to determine the $\\sigma$ based on the scene scales?\n3. In supp's Table.3. It's strange that the more fine-grained is the sampling (i.e., more $N_{mono}$) the worse results the proposed method can achieve. The paper should discuss about this experiment. Besides, what if we using even less $N_{mono}$? Given that the best result is achieved by the minimum $N_{mono}=5$ setting in Table.3, I think the ablation to find the optimum $N_{mono}$ is incomplete.\n\nLIMITATIONS:\nThe discussed limitation looks reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method called PanoGRF for synthesizing novel panoramas using two wide-baseline panoramas, with the incorporation of 360 scene priors into Spherical NeRF to generate new views. The method involves extracting appearance and geometry features from the input panoramas and estimating spherical depths through convolutions. To enhance the accuracy of 360 panoramas MVS depth estimation, the authors integrated a 360 monocular depth estimation network. Specifically, they employed a Gaussian distribution to sample depth candidates around the estimated 360 monocular depth. Extensive experiments were conducted to demonstrate the effectiveness of the proposed method in synthesizing novel views from wide-baseline panorama stereos.\n\nSTRENGTHS:\nThe authors have performed thorough experiments to showcase the effectiveness of the proposed method, making the study comprehensive. The results obtained from the performance evaluation and ablation study are noteworthy, highlighting that the proposed method achieves state-of-the-art performance. The contributions of the proposed components in achieving this success are significant.\n\nMoreover, the paper is excellently written and presents the concepts in a clear and easily understandable manner. The figures are well-drawn, effectively conveying the main ideas and providing visual clarity to the readers.\n\nWEAKNESSES:\nThe level of novelty in this paper is relatively limited since many of the components utilized have been proposed by previous methods. For instance, the appearance and geometry feature aggregation techniques are adopted from NeuRay, while the 360 MVS approach, which involves cost volume and mono-guided depth sampling, can be attributed to [14, 12, 28]. Furthermore, it is not particularly challenging to adapt these techniques to panoramas by employing spherical projection.\n\nIt is worth noting that the comparison with IBRNet and NeuRay may not be entirely fair due to the differing assumptions regarding the number of input images. Both IBRNet and NeuRay assume multiple images as inputs, while this paper solely employs two images in its experiments.\n\nQUESTIONS:\nAlthough all experiments in the paper demonstrate the method's capability to generate views along the baseline of two cameras, the design of the proposed method suggests the potential to generate views outside of these limited regions. It would be valuable to investigate how the method performs in synthesizing views beyond the baseline. By conducting additional experiments to test the boundary and assess the performance of the proposed method in generating views outside the baseline, a more comprehensive understanding of its capabilities can be gained. This would be a beneficial addition to further validate and explore the boundaries of the proposed method.\n\nLIMITATIONS:\nThe authors have mentioned the limitations and societal impact of the paper in lines 273-278. It well covered the possible limitations and potential societal impact of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper introduces PanoGRF, a method for generalizable novel view synthesis of sparse panorama images with wide baselines. This PanoGRF is basically built upon the perspective view synthesis method, NeuRay. Previous generalizable NeRF methods are mainly designed for perspective images are may introduce extra errors between the projection of perspective views and panorama views.

To perform generalizable view synthesis given multiple panoramas, the authors 1) design CNN modules on panoramas to extract the geometry and appearance fields, constructing the cost volumes for density and color estimation; 2) design a 360-panorama depth estimation method based on both multi-view image feature correlations and cues of monocularly estimated depth maps. The predicted depth maps are for the upcoming geometry feature extraction.

STRENGTHS:
- The paper is well written and easy to understand.
- Lots of supplementary materials are given to help the readers fully appreciate the experimental results and algorithm designs.
- The proposed method demonstrates state-of-the-art view synthesis quality on generalizable view synthesis task from panorama images.

WEAKNESSES:
- The overall idea for generalizable view synthesis is not new and is basically built upon the existing NeuRay work, therefore it is a bit incremental for the research community.
- The qualitative results are limited in indoor scenes as discussed in the limitations. It would be better if PanoGRF can be trained and evaluated on outdoor unbounded scenes.

QUESTIONS:
The paper is written very clearly, and I find no significant questions to ask.

LIMITATIONS:
The limitations are adequately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper presents PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas, which introduces mono-guided 360 ◦ depth estimation and leverages each panoramic view based on spherical projection. The experiments show that the proposed method significantly outperforms state-of-the-art generalizable view synthesis methods for wide-baseline panoramas (e.g., OmniSyn) and perspective images (e.g., IBRNet, NeuRay).

STRENGTHS:
## Pros
1. Novelty: It's interesting to directly aggregate geometry and appearance features of 3D sample points from each panoramic view based on spherical projection. It's novel to leverage mono-guided 360 ◦ depth estimation to improve the geometry features. Although it can be argued that it's just the combination of Neuray + mono-mvs + spherical radiance fields.
2. performance: The results are remarkable compared with the sota methods.
3. writing: it's well-written and easy to follow.

WEAKNESSES:
Cons: I do not find some major concerns. The insight is easy to follow and the experiments are adequate. But I have some subjections as follows:
1. it seems to use false citation: S-NeRF[17].
2. It may lack  some ablation studies: w/o aggregation geometry/appearance feature

QUESTIONS:
please see the weakness.

LIMITATIONS:
yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05