PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas
Authors: Zheng Chen
Yan-Pei Cao
Yuan-Chen Guo
Chen Wang
Ying Shan
Song-Hai Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: neural rendering
neural radiance field
novel view synthesis
panorama
360-degree image
TL;DR: high-quality novel view synthesis with wide-baseline panoramas

Abstract: Achieving an immersive experience enabling users to explore virtual environments with six degrees of freedom (6DoF) is essential for various applications such as virtual reality (VR). Wide-baseline panoramas are commonly used in these applications to reduce network bandwidth and storage requirements. However, synthesizing novel views from these panoramas remains a key challenge. Although existing neural radiance field methods can produce photorealistic views under narrow-baseline and dense image captures, they tend to overfit the training views when dealing with wide-baseline panoramas due to the difficulty in learning accurate geometry from sparse $360^{\circ}$ views. To address this problem, we propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas, which construct spherical radiance fields incorporating $360^{\circ}$ scene priors. Unlike generalizable radiance fields trained on perspective images, PanoGRF avoids the information loss from panorama-to-perspective conversion and directly aggregates geometry and appearance features of 3D sample points from each panoramic view based on spherical projection. Moreover, as some regions of the panorama are only visible from one view while invisible from others under wide baseline settings, PanoGRF incorporates $360^{\circ}$ monocular depth priors into spherical depth estimation to improve the geometry features. Experimental results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-art generalizable view synthesis methods for wide-baseline panoramas (e.g., OmniSyn) and perspective images (e.g., IBRNet, NeuRay).

PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas
Zheng Chen'; Yan-Pei Cao?, Yuan-Chen Guo', Chen Wang', Ying Shan', Song-Hai Zhang' TTinghuua University 2 ARC Lab, Tencent PCG
China
{chenz20, @wwac19mmlalllll..oomom. wtw.llsssstingua..e.. cw. www.vanggoottlook com yingashan@teeceet. com shzStsinghua, edu
eaoyanpei@gmail com
Abstract
Achieving an immersive experience enabling users explore virtual enviromments with six degrees of freedom (6DoF) is essential for various applications such as virtual reality (VR). Wide-baseline panoramas are commonly used these applications to reduce network bandwidth and storage requirements. However, synthesizing novel views from these panoramas remains a key challenge. Al- though existing neural radiance field methods can produce photorealistie views under narrow-baseline and dense image captures, they tend to overfit the train- ing views when dealing with
panoramas due to the difficulty in learning accurate geometry from sparse 360 views. To address this problem, we propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas, which construct spherical radiance fields incorporating 360 scene pri- Unlike generalizable radiance fields trained perspective images, PanoGRF avoids the information loss from manorama-to-perspectin conversion and directly aggregates geometry and appearance features of 3D sample points from each panoramic view based on spherical projection. Moreover, as some regions of the panorama are only visible from one view while invisible from others under wide baseline settings, PanoGRF incorporates 360 monocular depth priors into spher- ical depth estimation improve the geometry features. Experimental results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-ar generalizable view synthesis methods for wide-baseline panoramas (e.g., OmniSyn) and perspective images (e.g., IBRNet, NeuRay). Poject Page: https://thuez: wtt.t....................o
Introduction
The rise of 360 cameras and virtual reality headsets has fueled the popularity of 360 images among photographers and tourists. Commercial VR platforms, such Matterport 2 enable users to experience virtual walks in 360 scenes by interpolating between panoramas [17]. Wide-baseline panoramas frequently employed on these platforms for capture and network transmission to reduce storage space and bandwidth requirements. Consequently, synthesizing novel views from wide-baseline panoramas is essential task for providing a seamless six degrees of freedom (6DoF) experience to users.
However, synthesizing novel views from pair of wide baseline panoramas encounters two primary challenges. First, the input views are sparse, causing existing state-of-the-art methods such as NeRF 21 to struggle in learning accurate geometry due to the shape-radiance ambiguity [37].
Work done during an internship at ARC Lab, Tencent PCG Matterport: https://maaterport.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

leading to overfitting of the imput views. Second, certain regions in the scene may only be visible in one view but in another, making it challenging to provide correct geometry prior NeRF using only multi-view stereo. State-of-the-art generalizable NeRF methods 35 32 20 address the overfitting problem by incorporating scene priors into NeRF; however, they are designed for perspective images and require conversion of panoramas into perspective views when applied, resulting in information loss and suboptimal performance due to the limited field of view. Although spherical radiance fields [T] have been proposed to render panoramic images based on spherical projection, they still suffer from the overfitting problem without considering the 360 scene prior. Other approaches designed for 360 view synthesis, such multi-sphere images (MSI) [2] and mesh- methods [17], exhibit limited expressiveness and rendering quality compared to NeRF due to the use of discretized scene representations.
In this work, we present a method that addresses the overfitting issues spherical radiance fields by incorporating 360 scene priors. Unlike existing generalizable methods that require panorama- to-perspective conversion, our approach retains the panoramic representation. Furthermore, we incorporate 360 monocular depth to alleviate the view occlusion problem.
To address the first challenge, we present a solution in the form of generalizable spherical radiance fields. To render the panorama at a new position, spherical radiance fields (Spherical NeRF) cast a ray for each pixel using spherical projection, sample points along the ray, and aggregate the colors of these points based on their density values following NeRF [21]. In our method, named PanoGRF, we incorporate 360 scene priors into Spherical NeRF. Specifically, we extract appearance and geometry features from input panoramas and estimated spherical depths through convolutions, respectively. PanoGRF accurately aligns the queried 3D points with the corresponding pixels in panoramas using spherical projection. This alignment strategy leverages the full field-of-view characteristic of panoramas and eliminates the information loss of petoorama-to-erspeectte conversion. The local appearance and geometry features at these corresponding pixels are then aggregated and serve as the conditional input for Spherical NeRF.
To tackle the second challenge, we enhance the accuracy of depth estimation in 360 multi-view stereo by integrating 360 monocular depth estimation network. In 360 multi-view stereo, depth inaccuracies can arise due to view inconsistencies caused by occluded regions. Inspired by the per- spective method proposed in B we sample depth candidates around the estimated 360 monocular depth, assuming a Gaussian distribution. These depth candidates are then utilized in sphere sweeps during 360 multi-view matching. By incorporating monocular depth candidates, we can improve the accuracy of depth estimation in regions with view inconsistencies. This improvement contributes to the robustness of the geometry features employed in PanoGRF and ultimately leads to superior view synthesis performance.
In summary. we make the following contributions: 1) We propose the design of generalizable spherical radiance fields to learn 360' scene priors that alleviate the overfitting problem when dealing with wide-baseline panoramas. 2) We incorporate 360 monocular depth into the spherical multi-view stereo to mitigate the view inconsistency problem caused by occluded regions, which results in more robust geometry features, ultimately improving the rendering performance. 3) We achieve the state-
of-the-art novel view synthesis performance on Matterport3D [4, Replica (28 and Residential g under the vide-baseline setting.
2 Related Work
In this section, we briefly review the methods of perspective view synthesis and 360 view synthesis.
2.1 Perspective View Synthesis
Recently, NeRF (neural radiance fields 21 has become the mainstream scene representation due to its photo-realistic rendering quality for perspective view synthesis. However, given sparse input views, NeRF's per-scene optimization approach is prone to overfitting the training views and fails to learn the correct geometry due to the shape-radiance ambiguity, leading to significant floaters in novel views. Several subsequent methods [12 6 15 22
34 have attempted to address this issue by incorporating depth constraints from COLMAP 26 or other regularization terms, but they
still rely on per-scene optimization and lack the ability to generalize to new scenes. In contrast,

---

Geumetry Features
Features h
Moso-gaided 360' Depti Estimator
Local Features
Pi
Lood Feaurres
Volume Rendaring
360 Viewpoint
Fealure Extractor
Inpat Views 40 1.2)
Novel View
Appearance Fentures
Features of Spherical Radiance Fields
Figure 1: The overview of PanoGRF. Initially, employ convolutional neural networks (CNNs) to extract appearance features and geometry features from the input views. The geometry feature is obtained from the predicted spherical depth generated by proposed Mono-guided 360 Depth Estimator (Sec. 3.3). Next, we cast rays based on spherical projection to render novel 360 view (Sec. 3.Ip. Along each ray, every 3D sample point p; is projected onto the corresponding panoramic grid coordinate (u, v). The local geometry feature gi,j at (u, v) is decoded into Vi,j (the visible probability of p, for the j-th view). The local appearance feature fa. (u, v) is aggregated into fa along with Ui (.2). Subsequently, the aggregated feature fi is decoded to determine the color and density of p,. Finally, novel 360 view is synthesized through volume rendering (Sec. 3.1
PixeINeRF 35, GRF 30) and IBRNet 32 condition NeRF rendering with pixel-aligned appearance features to learn scene priors from large-scale datasets, enabling direct inference with input images. NeuRay 20 aggregates visibility information (obtained from multi-view stereo) and appearance features from each view to generate the density and color 3D sample points. However, when dealing with panoramas, these methods require converting the panoramas to perspective images, leading to information loss and suboptimal performance due to limited field-of-view in each perspective view. In contrast, PanoGRF operates directly on panoramas using spherical projection, eliminating the need for the panorama-to-perspeetiii conversion.
2.2 360 View Synthesis
Multi-sphere images (MSI) are widely adopted for 360 view synthesis. It is introduced by [2] to represent scenes with the input of omnidirectional stereo video. SOMSI (8) improved the expressive ability of MSI by incorporating high-dimensional feature layers and achieved real-time performance. However, MSI's capability is limited to narrow- baarelee scenarios due to the back surface rendering issues as discussed in [18, which attempted expand the renderable viewpoint range of MSI through the interpolation of two MSI instances. But their pure convolutional architecture only works at low resolutions. OmniSyn I7 constructs two meshes for input panoramas using 360 multi-view stereo. It warps the meshes the target view and fuses the colors attached to the meshes with an in-painting network. This approach frequently produces ghosting artifacts due to surface-based warping and blending. 360" Roam [II] divides NeRF into multiple blocks to achieve real-time rendering of panoramas, akin to KiloNeRF [23]. But it fits a single scene without considering scene priors, making it unsuitable for wide-bascline panoramas. In contrast the aforementioned methods, introduce generalizable spherical radiance fields to learn both geometric and appearance priors from 360 datasets, enabling better generalization unseen 360 scenes.
However, 360 multi-view stereo alone cannot provide robust geometric prior, fails to address the occlusion issue in the vide-baseling setting. Recently, numerous researchers have focused on to mitigate spherical distortion by harnessing the fusion of equirectangular and perspective projections. Inspired by MaGNet 3. we employ 360" monocular depth prior to guide the construction of 360 cost volume, which enhances the quality of geometric features and boosts rendering performance.
360 monocular depth estimation [13 I9
3

---

3 Method
PanoGRF facilitates novel view synthesis from wide-baseline panoramas as shown in Fig. 1 Before
introducing PanoGRF, we first review NeRF and its spherical variant in Sec. 3.1 We demonstrate the two important parts of PanoGRF, the generalizable spherical radiance fields, and the mono- guided spherical depth estimator in Sec. 3.2 and Sec. 3.3 respectively.
3.1 Spherical Radiance Fields (Spherical NeRF)
To render a pixel, NeRF casts ray from a given viewpoint and parameterizes the ray as p(t) = o td, & E IR+, where o is the camera center and is the ray direction. N points are sampled along the ray with increasing t, we denote p; p(t;), = I.... N. NeRF utilizes an MLP map the position p; together with viewing direction d into color and density o; The pixel color of the ray p can then be computed by:
Σ W.C.
(1)
where e R& is the rendered color of p. and C; E R is the color of the sampled point p:. w; is the blending weight of the point p. which indicates the probability that ray travels from the origin to t without hitting any particle and terminates in the range (t4, f++1. is computed by w, = I (1 - ax )os, where a is the alpha values in the depth range (4i, 1),,, ou exp(-o;;)
and 6, 1 ty. The color of a ray p optimized by photometric loss:
L
Σ ||c cgell?
(2)
where Cyt represents the ground truth color.
Ray-casting based on spherical projection The panorama uses the panoramic pixel grid coordinate system, while the coordinate systems in NeRF are all Cartesian. By employing spherical projection (8] 111, a pixel (u, v) in the panorama is firstly transformed into spherical polar coordinate (ò, 0) and subsequently into cartesian coordinate (a, 2). Detailed transformation formulas can be seen in the supplementary material. The ray direction emitted from the pixel (u, v) can be computed by d = R(x, y, z] where R E 3x33 is the panorama's 's camera-to-word rotation matrix.
3.2
Generalizable Spherical Radiance Fields
Under wide bascline, Spherical NeRF tends to overfit training views and struggle to generate plausible novel views. To address this limitation, we draw inspiration from the generalizable NeRF approaches B5 32 20 designed for perspective images. In work, we propose to incorporate 360 scene priors into Spherical NeRF by aggregating local features from input panoramas.
Alignment based spherical projection Panoramas can be converted into perspective images, and then generalizable NeRF methods for perspective images can be applied to novel 360 view synthesis. However, when projecting the 3D sample point p, onto the source perspective view, it may fall outside the image borders or even be located behind the source perspective camera (with z-depth < 0) due to the limited field-of-view. This can introduce errors the aggregation of features and result in poor rendering results. To overcome this problem, we directly align p; with the corresponding pixels panoramas using spherical projection, enabling us leverage the full field- of-view characteristic of panoramas. We first transform into the camera's 's Cartesian coordinate system (1, y, 2) of the j-th panoramic view 1,. Subsequently, (z, y, 2) is converted into spherical polar coordinate (0, Φb t) (t E TR+ indicates the spherical depth of p; for I,) and finally transformed into the panoramic grid coordinate (u, v). The specific formulas for these transformations can be found in the supplementary material.
Appearance and geometry feature aggregation We follow NeuRay 20 to aggregate appearance and geometry features. But differently, we take panoramas and spherical depth (radial distance from the camera center) as input instead of perspective images and z-depth. PanoGRF respectively extracts appearance feature W, from the j-th panoramic view (j 1,2) and geometry feature G, from

---

360" Munocular Depth Estimation
Reference View
Monocular Depth
CostVolume(r, D. D Similarity
360 Monocular
Depth Estimator
Giusssaa Sampling
Reference & Source Views
Smurce Pose
Sphere Sweep
(tr Plwweg
CNN
Decode
Feature Maps
(u, M)re
(1 .).
Referenie Pose
360 Depth
360 Multi-view Stereo
Figure 2: The process of mono-guided 360 depth estimation. We first extract the image features of reference and source views with convolutions. Using spherical projection, corresponding pixel (u, sre in the source view for each pixel (u, v)ref of the reference view, with the depth hypothesis of 4, The similarity between the local features (u, v)ref and (u, v) is computed as the value of the cost volume at (u, v, i). Depth sampling of l is guided by the 360 monocular depth using Gaussian distribution assumption. The cost volume obtained after D sphere sweeps. Lastly, the cost volume is decoded into 360 depth using convolutions.
determine the
the spherical depth using convolutions. Details for getting the spherical depth will be introduced in Sec. 3.3 For a given 3D sample point pt. we project it onto the panoramic grid coordinate (u. v), and
extract the local appearance feature fi.j = W (u, v) and the local geometry feature gtj = (u, v)
at (u, v). The local geometry feature J., is decoded into the visibility vi.j (demonstrated below) using an MLP. The local appearance feature frg and visibility Us.Ú of the j-th panoramic view are aggregated for p: with aggregation network J::
fi = F(f Vigli = 1,2)).
(3)
According the local geometry feature g of an input panorama, predict the visibility function u(t) to indicate the probability that point at spherical depth (instead of z-depth in NeuRay) is visible for the input panorama. v(t) (u(t) [0, 1) is represented as vtt) o(t), where o(t) is the occlusion probability, To parameterize (f), we employ a mixture of Ni logistic distributions:
olt; pi, Th, m2)
Σ my (m.S((t-a)/o)),
where (4, and my are the mean, standard variance, and blending weight of the k-th logistic distribution respectively. Σ my 1 and S(.) denotes sigmoid function. The parameters [Uk, ok: my] are decoded from the local geometry feature vector g by an MLP. For each 3D sample point p:, compute its spherical depth t, for j-th input view based on spherical projection and obtain the visible probability Vi,y = uy (t,) of p, for the j-th panoramic view. Then we aggregate Uj. into f. by Eq. and decode f: into the color and density of p;.
and
3.3 Mono-guided Spherical Depth Estimator
To obtain the visibility in Sec. 3.2 we need predict the spherical depth of each input view. The process is illustrated in Fig.
360 multi-view stereo (MVS) With the input multi-view panoramas, we estimate the spherical depth of the reference view using 360 multi-view stereo similar to previous methods [1 14]. Firstly, we extract the image features of the reference and source views with an image encoder. For each pixel (u, t)ref of the reference view, assuming its depth is tt, we find its corresponding pixel (u, v) are in
5

---

the source view according to spherical projection. We uniformly sample D depth candidates for t; (with 1, 2, ..., D) without considering the monocular depth prior, which will be introduced later. Next, we compute the similarity between the local features at (u, t)ref and (u, s)ee. considering it as the value of the cost volume at (u, v, i). Assuming the length of the local feature vector F, H this process results in a 4D cost volume V E IR * xDxF after D sphere sweeps, where T and W respectively represent the height and width of the feature maps for the reference view. Subsequently, и W. x I x D through a 3D CNN. Lastly, the processed
I
the dimension of the cost volume is reduced to I cost volume is decoded into 360 depth using several convolution layers.
Mono-guided spherical depth sampling Under the wide-baseline setting, some areas might be visible from one view but occluded from another, which case 360 MVS may struggle to produce accurate depth estimates. To address this challenge, we leverage the 360 monocular depth [3] to guide the depth sampling of 360 MVS, inspired by perspective methods 36 31
For each pixel (u, v) in the panorama, we denote its estimated spherical depth from the monocular depth network Mu,.- We generate depth candidates in the vicinity of the monocular depth using a Gaussian distribution assumption. Specifically, a search space for each pixel in the panorama is defined [ph,F Ba, Mu,e [30] where 33 and serve hyperparameters and represents the standard deviation. This search space is divided into Nono bins, ensuring that each bin has the same probability mass. We select the midpoint of each bin as the depth candidate. Consequently, the k-th depth candidate for pixel (u, v) is defined as bu Ma,, + bbO k: = 1, 2, ...y Nmono- by represents the offset corresponding to the k-th bin. Similar to
calculate by the following:
by 2
1
④
1-1 Nmono
P
1 Pt
N Mono 1
h
P' 1- 1-11)
2 )
(5)
where P = crf( denotes the probability mass covered by the search space [,u, Bo, ua, + Bo),
/2
(-) is the quantile function of standard normal distribution, erf is the error function. Considering errors may occur in the 360 monocular depth, we use a mixture of Nmono monocular depth candidates and Nani uniform depth candidates sampled from uniform distribution. By constructing
D == Nuni + NNmono sphere sweeps, we obtain a new spherical cost volume. The monocular depth guidance enables us to extract more reliable geometry features employed in PanoGRF.
4 Experiment
4.1 Metrics and Datasets
PSNR, SSIM [10, LPIPS 38 and WS-PSNR 229 are used as evaluation metrics. We conduct experiments on Matterport3D 4. Replica (28), and Residential [8) For Matterport3D and Replica, we leverage HabitatAPI (25 to generate the 256 x 256 perspective images (representing the six sides of cube-maps) and stitch them into a 512 1024 panorama. We panoramic sequences with length of 3. The middle view is rendered based on the first and last views. We conduct comparative experiments on Matterport3D under fixed camera baselines of 1.0, 1.5, and 2.0 meters, where the camera baseline refers to the distance between the camera centers of the first and last views. The baselines used for Residential and Replica are approximately 0.3 and 1.0 meters, respectively.
4.2
Implementation Details
We set /Nmono 5, Nuns = 59, and N₁ = D =NNono + Nuni is 64. used in mono- guided depth sampling is set to 0.5 and 3 is set to 3. Additional details regarding network architecture can be found in the supplementary material.
4.3 Comparisons
We compared PanoGRF with S-NeRF (spherical variant of NeRF (21),, IBRNet [32,, NeuRay 20). and OmniSyn [77. We trained S-NeRF from seratch, as is a per-scene optimization method. Other methods are pre-trained on Matterport3D and tested on unseen testing scenes. We fed the original cube maps of panoramas (in Matterport3D and Replica) to the perspective methods (IBRNet and NeuRay). As there are only panoramas in ERP (cquirectangular projection) format for Residential,

---

cube maps were converted from panoramas. To render panoramic view for evaluation, we cast rays in IBRNet and NeuRay with spherical projection (refer to Sec, (.I] and aggregate features with their original perspective projection. We use NeuRay" and IBRNet" to denote variants of NeuRay and IBRNet which render panoramas. We did not compare with methods based on multi-sphere images (MSI) 2 g they can only render novel views within the smallest sphere of MSI, which are unsuitable for wide-baseline panoramas, In the supplementary materials, we provide additional analysis on per-scene fine-tuning of PanoGRF and also compare it with Cross Attention Renderer 7
Table Quantitative comparisons with baseline methods on Matterport3I. The best results are in bold.
baseline
1.0m
1.5m
2.0m
method
WS-PSNR1 SSIM LPIPS WS-PSNR1 SSIM LPIPS| WS-PSNR1 SSIM LPIPS
S-NeRF 15.25
0.579
0.546
14.16
0.563
0.580
13.13
0.523
0.607
OmniSyn
22.90
0.850
0.244
20.31
0.790
0.317
18.91
0.761
0.354
IBRNet" 25.72
0.855
0.258
21.69
0.751
0.382
20.04
0.706
0.431
NeuRay" 24.92
0.832
0.260
21.92
0.766
0.347
19.85
0.715
0.407
PanoGRF 27.12
0.876
0.195
23.38
0.811
0.282
20.96
0.761
0.352
means models are trained with cubemap projection and evaluated with equirectangular projection.
Somple (Replica)
Sample 2 (MatterportL))
IBRNet
12
OmniSyn
[tt]
NeuRay"
20
PanoGRF
Ground Truth
Figure Qualitative comparison on Replica and Matterport3D between IBRNet". OmniSyn,
NeuRay" and PanoGRF.
Analysis On Matterport3D and Replica, we quantitatively compared PanoGRF with the baseline methods. The results be found in Table. [add Table. Almost all the metrics show that PanoGRF outperforms other methods significantly. We show the qualitative comparisons in Fig. 3 IBRNet* and NeuRay aggregate features based on perspective projection. Due to the limited field of view, 3D sample points are often projected somewhere behind the source perspective camera (z-depth<O) or outside the image border during pixel alignment. In this case, the aggregated features are incorrect, causing bad rendering results. As for OmniSyn, its rendering outputs exhibit ghosting artifacts, particularly notable the boundaries of the two sofas in Sample of Fig. B Unlike these methods, PanoGRF is generalizable spherical NeRF which is more appropriate for panoramic images due to use spherical coordinates. achieves superior accuracy synthesizing object boundaries and demonstrates better pixel alignment, as evident from the results showcasing the sofas in Sample and the desk in Sample 2 of Fig. 3. On Residential, we also compared PanoGRF with S-NeRF, IBRNet*. NeuRay*, The results of S-NeRF show severe floaters (See the ceilings of Sample in Fig. 4). As

---

Sample (Residential)
Sample (Residential)
S-NeRF
21
IBRNet
NeuRay'
20
PanoGRF
Ground Truth
Figure Qualitative comparison on Residential between S-NeRF, IBRNet". NeuRay" and PanoGRF.
converting panoramas into perspective views brings information loss, the perspective generalization methods IBRNet* and NeuRay have notable artifacts. In contrast, PanoGRF demonstrates excellent generalization performance on the Residential dataset, as shown Fig. Additional qualitative comparisons can be seen in the supplementary material.
Table 2: Quantitative comparison with baseline methods on Replica and Residential Dataset
Dataset
Replica (1.0m)
Residential (about 0.3m)
method
PSNR
WS-PSNRT
SSIM
LPIPS
PSNRT
WS-PSNRT
SSIM
LPIPS
S-NeRF 16.91
16.10
0.723
0.443
23.06
22.47
0.741
0.435
OmniSyn 23.91
23.17
0.898
0.189
IBRNet 23.35
22.65
0,854
0,291
22.95
22.47
0.735
0.498
NeuRay* 26.62
25.90
0.899
0.187
23.01
22.38
0.753
0.427
PanoGRF 30.08
29.22
0.937
0.134
31.64
31.03
0.909
0.207
4.4 Ablation Studies
We conducted ablation studies to evaluate the effectiveness of the key components in mono-guided spherical depth estimator: 360 monocular depth and multi-view stereo. We provide qualitative comparisons of 360 depth estimation quality on Matterport3D Fig. 5 and analyze the view synthesis quality Replica in Table. More results can be found in the supplementary material.
360 monocular depth We removed the 360 monocular depth guidance and kept the total number of depth candidates unchanged (N uut 64) in the depth sampling. Under three camera baselines, all the metrics dropped after removing the 360 monocular depth. Especially. under the camera baseline of 2.0 meters, WS-PSNR dropped by 1.33dB without monocular depth. This removal resulted in error-prone object boundaries, such as the bedside and wooden pillar (shown in Fig. 6). The guidance of monocular depth can give more accurate depth candidates and mitigates the view inconsistency problem, which 360 multi-view stereo cannot handle alone.
360 multi-view stereo We removed 360 multi-view stereo and used 360 monocular depth directly as the input of the geometry feature extraction in PanoGRF, In this case, WS-PSNR dropped

---

more than 1.0 dB under the camera baselines of 1,5 and 2.0 meters. And LPIPS became 0.165 from 0.134 under the baseline of 1.0 meters. Without multi-view stereo, multi-view consistency of geometry is not guaranteed, degrading the quality of novel view synthesis results. In Fig. 6 we observed the presence of ghosting artifacts near the left wooden pillar when relying solely on 360 monocular depth.
Table 3:
Ablation studies on Replica
baseline
1.0m
1.5m
2.0m
method
WS-PSNRI SSIM LPIPS
WS-PSNR1
SSIM LPIPS!
WS-PSNR1
SSIM
LPIPS.
w/o Mono
28.71
0.935
0.136
25.53
0.898
0.188
23.33
0.864
0.249
w/o MVS
28.23
0.925
0.165
25.15
0.888
0.227
23.15
0.855
0.274
full 29.22
0.937
0.134
26.38
0.903
0.187
24.48
0.885
0.223
5
Conclusion
In this paper, we propose PanoGRF, generalizable spherical radiance fields for wide-baseline panora- mas. We aggregate the appearance and geometry features from input panoramas through spherical persppecttive conversion. To address the view inconsistency problem, use 360 monocular depth to guide the spherical depth sampling and obtain more accurate geo- metric prior for the spherical radiance fields. Experiments on multiple datasets verify that PanoGRF render high-quality novel views given wide-baseline panoramas.
projection to avoid
Limitations Similar to other generalizable radiance fields, PanoGRF suffers from the issue of rendering speed. Additionally, since we only train PanoGRF on indoor data due to the lack of large-scale outdoor 360 datasets, its generalization performance can be limited when applied to outdoor scenes with significantly different depth scales.
Societal impact This method may be used synthesize fake deceptive panoramas, combined with generative methods.
Mono only
MVS only
fall
Ground Truth
Figure 5: Qualitative results of ablation studies for depth estimation on Matterport3D. Mono only and MVS only respectively refer to the results obtained when using only 360 monocular depth and 360 multi-view stereo. The use of 360" monocular depth alone does not ensure multi-view consistency, resulting potential discrepancies between the predicted scale and the ground truth certain regions. Due to the occlusion problem, using only 360 MVS results in inaccurate and less detailed depth predictions, especially at the boundaries of objects.
6 Acknowledgements
This work was supported by the Natural Science Foundation of China (Project Number 62132012) and Teincnt
Tencent Joint Laboratory for Internet Innovation Technology.

---

w/o MVS
w/o Mono
fall
Ground Truth
Figure 6: Qualitative results of ablation studies for novel view synthesis on Replica.
References
[1] Hao Ai, Zidong Cao, Yan-Pei Cao, Ying Shan, and Lin Wang. Hrdfuse: Monocular 360deg depth estimation by collaboratively learning holistic-with-regiinnn depth distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19273-13282. 2023.
[2] Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin. Ma- tryodshka: Real-time 6dof video view synthesis using multi-sphere images. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part pages 441-459. Springer, 2020.
[3] Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Multi-view depth estimation by fusing single-view depth probability with multi-view geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2842-28511 2022.
[4] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[5] Zheng Chen, Chen Wang, Yuan-Chen Guo, and Song- Hai Zhang. Structnerf: Neural radiance fields for indoor scenes with structural hints. arXiv preprint arXiv:2209.05277, 2022.
[6] Kangle Deng, Andrew Liu, Jun-Yas Zhu, and Deva Ramanan, Depth-supervised nerf: Fewer views and faster training for free. CoRR, abs/2107.02791 2021.
[7] Yilun Du Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning render novel views from wide-baseline stereo pairs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[8] Tewodros Habtegebrial, Christiano Gava, Marcel Rogge, Didier Stricker, and Varun Jampani. Somsi: Spherical novel view synthesis with soft occlusion multi-sphere images. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15725-15734, 2022.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference paltern recognition, pages 2366-2369, IEEE, 2010.
[11] Huajian Huang. Yingshu Chen, Tianjian Zhang, and Sai-Kit Yeung. 360roam: Real-time indoor roaming using geometry-aware 360 radiance fields. arXiv preprini urXiv:2208.02700 2022.
[12] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEFF//VV International Conference on Computer Vision, pages 5885-5894, 2021.
10

---

[13] Hualie Jiang, Zhe Sheng, Siyu Zhu, Zilong Dong, and Rui Huang. Unifuse: Unidirectional fusion for 360 panorama depth estimation. IEEE Robotics and Automation Letters, 6(2):1519-
1526, 2021.
[14) Sameh Khamis, Sean Fanello, Christoph Rhemann, Adarsh Kowdle, Julien Valentin, and Shahram Izadi. Stereonet: Guided hierarchical refinement for real-time edge-aware depth prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 573-590, 2018.
[15] Mijcong Kim, Sconguk Sco, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. CoRR, abs/21 12.15399, 2021.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[17] David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, and Ruofei Du. Omnisyn: Synthesizing 360 videos with wide-baseline panoramas. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pages 670-671.
IEEE, 2022.
[18] Jisheng Li, Yuze He, Jinghui Jiao, Yubin Hu, Yuxing Han, and Jiangtao Wen. Extending 6-dof vr experience via multi-sphere images interpolation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4632-4640, 2021.
[19] Yuyan Li, Yuliang Guo, Zhixin Yan, Xinyu Huang, Ye Duan, and Liu Ren. Omnifusion: 360 monocular depth estimation via seometry-awand fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2801 2810, 2022.
[20] Yuan Liu, Sida Peng. Lingjie Liu, Qianqian Wang. Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7824-7833 2022.
[21] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[22] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. CoRR, abs/21 12.00724, 2021.
[23] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335-14345, 2021.
[24] Barbara Roessle, Jonathan Barron, Ben Mildenhall, Pratul P Srinivasan, and Mathhis NieBner. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12892-12901, 2022.
[25] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[26] Johannes L. Schönberger and Jan-Michael Frahm. Structure-from mrommm revisited. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pages 4104-41 13, 2016
[27] Stefan Schubert, Peer Neubert, Johannes Pöschmann, and Peter Protzel. Circular convolutional neural networks for panoramic images and laser data. In 2019 IEEE intelligent vehicles symposium (IV), pages 653-660, IEEE, 2019.
[28] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruy Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The 2019.
Replica dataset: A digital replica of indoor spaces. arXiv preprint ar
11

---

[29] Yule Sun, Ang Lu, and Lu Yu.
to-spherically-umifmm quality evaluation for omnidi-
rectional video. IEEE signal processing letters, 24(9):1408-1444 1412, 2017.
[30] Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15182-15192, 2021.
[31]
Guangeong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. arXiv preprint arXiv:2303.16196 2023.
[32] Qianqian Wang. Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[33] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. SynSin: End-to-end view synthesis from single image. In CVPR, 2020.
[34] Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. arXiv preprint arXiv:2303.07411 2023.
[35] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 4578-4587, 2021.
[36] Zehao Yu. Songyou Peng. Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint wrXiv2206.00066. 2022.
[37] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[38] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
12

---

Table 4: Additional ablation studies on Matterport3D
Dataset
Matterport3D (1.0m)
method
PSNR
WS-PSNRT
SSIM
LPIPS
w/o appearance
5.44
5.24
0.001
0.707
w/o geometry
26.25
25.25
0.839
0.263
full 28.10
27.10
0.876
0.195
A More Results of Qualitative Comparisons
Qualitative comparisons with baseline methods on Replica and Matterport3D can be seen in Fig.
and Fig. respectively.
OmniSyn
IBRNet
NeuRay
PanoGRF
Ground Truth
Figure 7: Qualitative comparisons with baseline methods on Replica.
13

---

OmniSyn
IBRNet
NeuRay
PanoGRF
Ground Truch
Figure 8: Qualitative comparisons with baseline methods on Matterport3D.
Table 5: Ablation studies for depth estimation on Matterport3D
setting
L1
L2 4
RMSEL
WS-L,
WS-L24
WS-RMSEL
MVS only
0.1731
0.5048
0.5831
0.1984
0.2806
0.4731
Mono only
0.2452
0.3175
0.4731
0.2445
0.2729
0.4522
full
0.1441
0.2047
0.3877
0.1502
0.1624
0.3546
Table 6: The impact of different backbones for depth estimation Matterport3D
setting(backbone)
L |
L2
RMSEL
WS-L
WS-L24
WS-RMSEL
parameters
MVS(resnet-18)
0.1731
0.5048
0.5831
0.1984
0.2806
0.4731
29.75
MVS(resnet-34)
0.1654
0.4820
0.5676
0.1844
0.2577
0.4493
39.39
MVS(resnet-50)
0.1598
0.4725
0.5630
0.1822
0.2446
0.4442
47.10
MVS(resnnt-100)
0.1642
0.4994
0.5717
0.1835
0.2519
0.4441
65.22
1
0.1441
0.2047
0.3877
0.1502
0.1624
0.3546
58,95
MVSSrrssnet334++Mono
0.1549
0.2231
0.4120
0.1684
0.1878
0.3844
68.59
MVS/resnnet-0)>-Monn
0.1519
0.2379
0.4188
0.1675
0.1955
0.3887
76.30
MVSSrresnet-10))+Mooo
0.1735
0.2673
0.4452
0.1831
0.2129
0.4023
94.42
14

---

The impact of different numbers of depth candidates Mono for depth estimation on Matterport3D
Table 7:
Mono
L1 +
L2 +
RMSEL
WS-L14
WS-L24
WS-RMSEL
0.1586
0.2498
0.4317
0.1745
0.1971
0.3937
0.1432
0.1993
0.3865
0.1529
0.1649
0.3580
5
0.1441
0.2047
0.3877
0.1502
0.1624
0.3546
0.1496
0.2252
0,4104
0.1640
0.1896
0.3832
9
0.1645
0.2912
0,4511
0.1752
0.2215
0.4059
0.1596
0.2379
0,4188
0.1675
0.1955
0.3887
32
0.1735
0.2673
0.4452
0.1831
0.2129
0,4023
48
0.1689
0.2618
0,4333
0.1776
0.2047
0.3941
64
0.1604
0.2432
0.4162
0.1669
0.2004
0.3853
Table 8: The impact of different values of a for depth estimation on Matterport3D
L1
L2 |
RMSEL
WS-L14
WS-L24
WS-RMSEL
0.1
0.1544
0.2515
0.4261
0.1672
0.1915
0.3830
0.5
0.1441
0.2047
0.3877
0.1502
0.1624
0.3546
1.0
0.1689
0.2686
0.4424
0.1803
0.2124
0.4029
1.5
0.1426
0.2457
0.4254
0.1563
0.1759
0.3723
B Spherical Projection
The transformation from the equirectangular image coordinate system to the polar coordinate system is defined as:
Equirretangular----hhherraal
Φ 2/ H 0 = W/ 2m 0.5m.
(6)
where á. 0 represent the latitude and longitude of the sphere, u, v represent the rows and columns of the panorama, H and W represent the height and width of the panorama respectively.
Spperrial The transformation from the polar coordinate system the 31D Cartesian coordinate system is:
sin ()) cos(9)
= cos ())
sin (Ø) si).
The camera cartesian coordinate (x: y, 2) transformed into the polar coordinate (8, (t E R+ denotes its spherical depth in view I,) by:
Cartesian-to-spheriaal
t 2 + 222 + 22
0 ar ctan
y1
é arccos
Table 9: Quantitative comparison with Cross Attention Renderer 7 on Matterport3D and Replica
Dataset
Matterport3D (1.0m)
Replica(1.0m)
method
PSNRT
SSIM
LPIPS
PSNRT
SSIMI
LPIPS
CAR 7
22.87
0.7679
0.3108
23.60
0.8594
0.2515
PanoGRF
27.78
0.8158
0.2444
29.87
0.9046
0.1604
15

---

Table 10: Comparisons with NeuRay 20
given multi- view inputs on Matterport3D
Dataset
Matterport3D (1.0m)
method
PSNR
WS-PSNRT
SSIM
LPIPS
NeuRay 20
27.82
26.74
0.8614
0.2312
PanoGRF
28.99
27.91
0.8762
0.2071
NeuRay
PanoGRF
Ground Truth
Figure 9: Qualitative comparisons between PanoGRF and NeuRay on Matterport3D with multi-view panoramic inputs.
Spphefial---eequircctaagaaar
The spherical polar coordinate (,, is turned into the equirectan-
gular image coordinate (u, v) by the inverse process of Eq. 6
C More Ablation Studies for 360 View Synthesis
We conducted ablation studies on Matterport33D and the results are shown in Table. In the "w/o appearance feature" ablation study, we replaced the appearance feature vector with a zero vector disable the appearance feature while keeping other modules unchanged. We found that the model without appearance features loses its ability to infer the color of novel views entirely, as the generalizable renderer heavily relies on appearance cues from input views. In the "w/o geometry feature" ablation study, we replaced the geometry feature vector with zero vector to disable the geometry feature while keeping other modules unchanged. We observed that although the model can still infer normal results, its performance is significantly worse than the original (full) model.
D Comparisons with NeuRay [20] Given Multi-view Inputs
Our method not limited to two panoramas. For instance, when rendering test view in the renderer module, we use N input panoramas reference images, and the renderer does not need to be modified. In the 360 spherical depth estimator module, for each reference image, we use the other N - 1 input panoramas source images. We average the multiple cost-volumes obtained during 360 multi-view matching process between the reference image and each source image. The rest is unchanged. In this way, our method can be applied to the multi-view panoramic inputs.
To further verify the effectiveness our method, we conducted comparative experiments with NeuRay using multiple panoramic image inputs. We placed the four input viewpoints at the
16

---

NeuRay
PanoGRF
Ground
Figure 10: Qualitative comparisons with NeuRay beyond the camera baseline on Matterport3D. We synthesized novel views at positions 0.25 meters above the middle point between two input viewpoints. The input viewpoints are 1.0 meters apart. Our method can achieve better results than NeuRay beyond the camera baseline.
NeuRay
PanoGRF
Ground
Figure 11: Failure case. The carpet area behind the bed was not visible the input viewpoints, but it becomes visible in higher viewpoints. NeuRay and PanoGRF tend to produce different and blurry results compared to the ground truth because they lack generative power. Combining existing diffusion generative models could potentially address this issue. We leave this future work.
corners of a horizontal square and tested the rendering performance at the center viewpoint and other viewpoints located at -0.4, -0.2, -0.1, -0.05, 0.05, 0.1, 0.2, and 0.4 meters in the vertical direction from the center viewpoint. The diagonal length of the square is 1.0 meters. Table. 10 and Fig. present the quantitative and qualitative comparison results between PanoGRF and NeuRay. As shown, PanoGRF still largely outperforms NeuRay with multiple panoramic inputs.
This experiment added during the rebuttal period. Due the limited time, we trained PanoGRF only for 20k iterations and NeuRay for 80k iterations. The learning decay strategies are similar to the setting of two views.
E
Comparisons with NeuRay [20] beyond Camera Baseline and Failure Case
We conducted an additional experiment where novel views were generated at positions 0.25 and 0.5 meters above the middle point between two input viewpoints. The input viewpoints are 2.0 meters apart. We compared the quantitative and qualitative results of our method with NeuRay's as shown Table. TT and Fig. IO, PanoGRF consistently surpassed NeuRay's performance, indicating its capacity yield superior results beyond the camera baseline.
some viewpoints, We also present the failure case of PanoGRF under such condition in Fig. | previously occluded may be visible and since this area has not been in either of the two
17

---

Table 11: Comparisons with NeuRay 20 beyond Camera Baseline on Matterport3D
Dataset
Matterport3D (1.0m)
distance
method
PSNR
WS-PSNRT
SSIM
LPIPS
0.25m
NeuRay
20.66
20.05
0.714
0.409
0.25m
PanoGRF
21.98
21.30
0.763
0.348
0.5m
NeuRay 20.39
19.94
0.711
0.420
0.5m
PanoGRF 21.99
21.42
0.769
0.349
existing viewpoints, the synthesis of this area is not effective. The area, which has been seen in either of the existing viewpoints may be able to be filled in by combining with the existing diffusion generative approach. This is the next direction we plan to investigate.
Experiments for Mono-guided Spherical Depth Estimator
F.1 Ablation Studies for Mono-guided Spherical Depth Estimator
To further validate the effectiveness of the key components, namely 360 multi-view stereos and 360 monocular depth, we conducted ablation studies specifically focused on spherical depth estimation. For evaluation purposes, we selected three commonly used metries: L1, L2, and RMSE. Additionally, we also used WS-L1, WS-L2, and WS-RMSE as metrics, which incorporate weighted latitudes of equirectangular images to simulate WS-PSNR [29]. This approach aims to mitigate the impact of equirectangular projection distortion. We selected the first 1000 panorama pairs from Matterport3D test data, with camera baseline of meter. For the evaluation, considered depth values within the range of [0.1, 1, 10] as valid.
The quantitative results of our ablation studies are presented in Table while the qualitative results can observed in Fig. 12 The experiments clearly demonstrate the importance of each module in achieving accurate depth estimation. Removing either the 360 multi-view stereo or the 360 monocular depth significantly reduces the depth accuracy. Using only 360 monocular depth does not guarantee multi-view consistency, resulting in potential discrepancies between the predicted scale and the ground truth in certain regions. The occlusion problem poses a challenge for using only 360 MVS, particularly at the boundaries of objects. Consequently, the depth predictions in these regions are inaccurate and lack detail.
F.2 Different Backbones for 360 Multi-view Stereo
Introducing 360 monocular depth 360 multi-view stereo does result in increase in the number of model parameters. However, is important to note that the improvement in depth accuracy is not attributed to the increase in parameters. In our experiments, we increased the model size of 360 MVSNet by using larger backbones, specifically ResNet [9].
From Table. 6. found that L2 and RMSE of 360 MVSNe((ResNet-IU) 101) are still far inferior to those of 360 MVSNet(RRsNNet-188 together with the 360 monocular depth network [13] This observation suggests that simply increasing the model size does not effectively address the view inconsistency problem the wide-baseline setting. On the other hand, the introduction of 360 monocular depth provides qualitative improvement the accuracy of 360 multi-view stereo. By incorporating monocular depth information, the model gains additional cues that help mitigate the view inconsistency issue and improve depth estimation performance. Furthermore, when larger backbones were used as replacements, it was found that the performances of 360 MVS+Mono deteriorated. This could be attributed the excessive model parameters leading overfitting.
F.3 Different Hyperparameters for Mono-guided Spherical Depth Estimator
We conducted evaluations to assess the impact of different values for (standard deviation) and N Nmono (number of monocular depth candidates) on mono-guided spherical depth sampling. The results presented in Table 7 and Table g
18

---

RGB
Mono only
MVS only
fall
Ground Truth
Qualitative results of ablation studies for depth estimation on Matterport3D. Mono only and MVS only respectively refer to the results obtained when using only 360 monocular depth and 360 miew stereo.
Figure 12:
19

---

Cross Attention Renderer
PanoGRF
Ground Truth
Figure 13: Qualitative comparisons with Cross Attention Renderer Replica
20

---

Cross Attention Renderer
PanoGRF
Ground Truch
Figure 14: Qualitative comparisons with Cross Attention Renderer Matterport3D
21

---

Table 12: The quantitative results of fine-tumin tinettuning of the general renderers. The best results are in bold.
baseline
1.0m
0.2m
method
setting
PSNR
WS-PSNR SSIMI LPIPS) PSNRT
WS-PSNRI SSIM LPIPS
NeuRay
gen
27.751
27.253
0.8470 0.2565 34.318
33.376
0.9409 0.1158
PanoGRF
gen
28.818
28.487
0.8778 0.1996 35.817
35.056
0.9554 0.0959
NeuRay(+Consist)
ft 26.755
26,250
0.8181
0.2820 33.354
32.359
0.9240 0.1304
PanoGRF ft 24.341
23,941
0,8175
0,2834 35.828
35.138
0.9578 0.0912
PanoGRF (+Depth)
ft 27.399
27.202
0,8556 0.2446 33.691
33.066
0.9333 0.1342
The reliability of 360 monocular depth estimation is not perfect. Therefore, our paper employs uniform sampling to compensate for the remaining depth candidates. In the ablation experiments, we consistently maintain Nmono + Nu = 64, where Nuni represents the sample number of uniform distribution. We discovered that the configuration yields the best results with regard to the metries of WS-L1, WS-L2, and WS-RMSE.
G Comparisons with Cross Attention Renderer 7
The Cross Attention Renderer (CAR) is method that operates on wide-baseline perspective pair. We divided the two panoramas into cube maps and utilized the corresponding sides of the cube maps as inputs for CAR. Specifically, we rendered the corresponding side of the cube maps at the middle viewpoint. For example, we input the left side of the cube maps and generate the left side of the cube maps at the intermediate viewpoint the output. We repeated this process for all six corresponding pairs of cube maps. In contrast, our method directly takes two panoramas as input and performs ray casting based on perspective projection. We then render the results for each side of the cube maps the intermediate viewpoint, preserving the panoramic nature of the input.
We conducted quantitative comparative experiments on Matterport3D and Replica. The qualitative comparisons with CAR on Replica and Matterport3D can be seen in Fig. 13 and Fig. [4] The results clearly demonstrate that PanoGRF significantly outperforms CAR in terms of rendering quality. CAR suffers from limitations associated with input field-of-view (FoV), particularly in edge regions that only visible from single perspective view. CAR relies pure stereo-matching matching method for geometric estimation, which leads to suboptimal rendering performance, as evidenced by the results presented in Table. / In contrast, PanoGRF is specifically designed to handle full FoV inputs, and it mitigates the issue of view inconsistency by incorporating the 360 monocular depth network.
H Fine-tuning of PanoGRF
We conducted per-scene fine-tuning for NeuRay 20 and PanoGRF on the first test scene of Mat- terport3D with baselines of 1.0 and 0.2 meters, respectively. The general renderers were fine-tuned for 10k iterations, and the quantitative results are presented in Table [2 Under the baseline of 1.0 meters, the general renderer of PanoGRF, denoted as PanoGRF-gen,, achieved the best performance. NeuRay-ft, the fine-tuned renderer of NeuRay, underwent fine-tuning using RGB loss and depth consistency loss, following the methodology described in their original paper 20) PanoGRF-ft was fine-tuned with only RGB loss. However, the results of NeuRay-ft and PanoGRF-It were inferior to NeuRay-gen and PanoGRF-gen, respectively. This suggests that fine-tuning under a wide-baseline setting does not improve the performance of general renderers. It is likely that fine-tuning with wide baseline leads overfitting. Even with the introduction of depth uncertainty loss 24] by supervising the renderer depth of PanoGRF with predicted spherical depths during the fine-tuning process, the results of PanoGRF-ft remained inferior those of the general renderer of PanoGRF. On the other hand, the performance of PanoGRF-ft under the baseline of 0.2 meters was slightly better than that of PanoGRF-gen.. However, adding the depth loss (2 was still unable improve PanoGRF- -en when fune-tuning. Inaccurate predicted depths in certain regions may be the cause, misleading the estimation of NeRF's geometry and thereby reducing the rendering performance.
Crros Attention Renderer: https://gthub.comyylllddduuuuu
22

---

Additionally, NeuRay-ft was still inferior to NeuRay- gen under the narrow baseline. This may be attributed the limited field-of-view, which can result the aggregation of incorrect features. When projecting 3D sample points onto other source perspective views (cube-maps), these points may fall outside the image borders of the source perspective view or located behind the source perspective camera.
I Quantitative Comparisons with Baseline Methods for Narrow-baseline Panoramas
Table 13: Quantitative comparisons with baseline methods on Matterport3D under the baseline of 0.2 and 0.5 meters. The best results are in bold.
baseline
0.2m
0.5m
method
PSNR1
WS-PSNRT
SSIM
LPIPS
PSNR1
WS-PSNR1
SSIM
LPIPS|
S-NeRF 20.79
19.52
0,6967
0.3756
17.95
16.81
0.6278
0.4856
OmniSyn
28.95
28.26
0,9132
0.1804
26.59
26.07
0.8897
0.2005
IBRNet 30.53
29.63
0.9271
0.1363
28.22
27.26
0.8844
0.1987
NeuRay 33.54
32.33
0.9485
0.1074
30.88
29.81
0.9196
0.1536
PanoGRF 34.29
33.27
0.9515
0.0977
31.41
30.46
0.9238
0.1318
We compared PanoGRF with baseline methods under the baseline 0.2 and meters on Mat- terport3ID. As shown in Table. 13] the quantitative results demonstrate that PanoGRF consistently outperforms all the baseline methods under the baseline 0.2 and 0.5 meters. These findings indicate that our method is applicable to both wide-baseline and narrow- narrow-baseline panoramas. In comparison to generalizable methods designed for perspective views, our method is particularly well-suited for synthesizing panoramic views by leveraging the aggregated features based spherical projection.
J
More Details of PanoGRF
J.1 Renderer
J.1.1 Training
PanoGRF employs the Adam optimizer (16] with initial learning rate of 4.0e-4. The pre-training process of PanoGRF was conducted on an A A00 GPU for 100k iterations, which required approxi- mately two days. The learning rate halved every 20k iterations, and batch size of 512 was used during training.
To streamline the experiments, train PanoGRF and all the baselines under 1.0m baseline for comparisons and test them under various baselines. In the ablation studies, the spherical depth estimators (Mono, MVS, and fusion) pre-trained under fixed 1.0m baseline. The general renderers in the ablation studies are pre-trained under corresponding baselines when tested under various baselines. We discovered that if pre-trained the general renderers under 1.0m baseline and tested them under larger baselines, there was no noticeable difference in performance between them.
J.1.2 Architecture
We adopted a coarse-to-fine sampling approach, similar to NeRF (21), and sampled 64 points in both phases. We followed similar architecture as NeuRay (20 build our renderers. The coarse and fine renderers share the same image encoder, geometry feature extractor, and visibility encoder. But they have different distribution decoders and aggregation networks F, similar NeuRay. During training, the weights of the 360 spherical depth estimator are fixed due to GPU memory limitations. Our image encoder, visibility encoder, distribution decoder, and aggregation network are implemented similarly to NeuRay, except for the padding mode. In the convolution layers of the image encoder and visibility encoder, we employ circular padding horizontally and zero padding vertically instead of direct zero padding. This is done to adapt to equirectangular image inputs and simulate Circular
23

---

Spherical Depth
Conv x 3(32)
ResBlock(32)
Conv x 1(32)
Image Encoder
Image Feature
ResUNet
Conv 3(32)
RexBlock(322
Conv 1(32)
Goometry Feature
Figure 15: Architecture of geometry feature extractor.
The circular padding helps account for the continuity of pixels the leftmost and rightmost edges of equirectangular images, which are neighbors, while the top and bottom edges are not. The distribution network consists of sub-networks, each with 3 fully connected layers. The appearance feature extractor is ResNet [9] which contains 13 residual blocks and outputs the appearance feature map with 32 channels. The architecture details of the geometry feature extractor shown in Fig. 15 The image encoder in the geometry feature extractor contains 14 residual blocks. The spherical depth input for the geometry feature extractor is first downsampled to 1/4 of its original resolution and then fed into the extractor to reduce GPU occupancy.
CNNs 277.
J.2 Spherical Depth Estimator
J.2.1 Training
We initially train the 360 monocular depth network and then freeze its weights when training the remaining components 360 MVSNet. The Adam optimizer is employed with a fixed learning rate of 0.0001. Both the 360 monocular depth network and 360 MVSNet trained for 100k iterations, which required approximately one day on single V100 GPU. The batch size is set to 2.
J.2.2 Architecture
The 360 monocular depth network 13 and 360" MVSNet utilize ResNet-18 as the feature extractor. The CNN regularization network composed
downsampling and 3 upsampling blocks, similar to [17]. The depth decoder consists of 2 convolution blocks. The feature map obtained from the middle layer in the 360 monocular depth network concatenated with the regularized spherical cost volume and then decoded into 360 depth by the depth decoder. In the multi-view matching process, we compute the similarity by subtracting the feature vectors.
K Datasets
For Matterport3D, we split the training and testing set following SynSin [33. The first scenes of the test set are used for evaluation. In the case of the Replica dataset, we render total of 18 scenes for evaluation. Additionally, we utilize the Residential dataset provided (8), which comprises three scenes. For this dataset, select the first and last panorama the input views.

---

L Training Details of Baseline Methods
We trained NeuRay 20 and IBRNet 32 for 400k and 250k iterations, respectively. Spherical NeRF (S-NeRF) 21 underwent training for 2000 epochs, equivalent to approximately 256k iterations with the batch size of 4096. For OmniSyn [17. the in-painting network was trained for 50k iterations. which took approximately 3 days on a TitanRTX GPU. The depth estimator of OmniSyn was trained 100k iterations. Due to limitations
GPU memory, OmniSyn was trained at a resolution of
256 x 256, and its output was resized to 512 x 1024 for evaluation purposes. CAR i was trained for 300k iterations.
25


Reviews:
[{"review_id": "EQGLNhYDIC", "reviewer": "Reviewer_zV2U", "summary": "This paper presents PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas, which introduces mono-guided 360 ◦ depth estimation and leverages each panoramic view based on spherical projection. The experiments show that the proposed method significantly outperforms state-of-the-art generalizable view synthesis methods for wide-baseline panoramas (e.g., OmniSyn) and perspective images (e.g., IBRNet, NeuRay).", "questions": "please see the weakness. ", "limitations": "yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Pros\n1. Novelty: It's interesting to directly aggregate geometry and appearance features of 3D sample points from each panoramic view based on spherical projection. It's novel to leverage mono-guided 360 ◦ depth estimation to improve the geometry features. Although it can be argued that it's just the combination of Neuray + mono-mvs + spherical radiance fields.\n2. performance: The results are remarkable compared with the sota methods.\n3. writing: it's well-written and easy to follow.", "weaknesses": "Cons: I do not find some major concerns. The insight is easy to follow and the experiments are adequate. But I have some subjections as follows:\n1. it seems to use false citation: S-NeRF[17].\n2. It may lack  some ablation studies: w/o aggregation geometry/appearance feature\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zV2U:\nWe sincerely appreciate your review and comments. We are delighted to receive your positive feedback and score. We hope the following responses will address your concerns effectively.\n\n1. Weakness 1:\n> it seems to use false citation: S-NeRF[17].\n    \n    We apologize for causing your misunderstanding. In fact, S-NeRF represents spherical variant of NeRF which casts rays by spherical projection to adapt to panoramic inputs as described in Sec.3.1 of our paper. So we cite the original paper of NeRF. To avoid confusion, we will change the description from \"S-NeRF [17]\" to \"S-NeRF (spherical variant of NeRF [17])\" in the revision of our paper. Thanks for pointing out this issue.\n\n2. Weakness 2:\n> It may lack some ablation studies: w/o aggregation geometry/appearance feature.\n    \n   Thank you for your suggestion. We conducted ablation studies on Matterport3D, and the results are shown in the table below. In the \"w/o appearance feature\" ablation study, we replaced the appearance feature vector with a zero vector to disable the appearance feature while keeping other modules unchanged. We found that the model without appearance features loses its ability to infer the color of novel views entirely, as the generalizable renderer heavily relies on appearance cues from input views. In the \"w/o geometry feature\" ablation study, we replaced the geometry feature vector with a zero vector to disable the geometry feature while keeping other modules unchanged. We observed that although the model can still infer normal results, its performance is significantly worse than the original (full) model. Due to the length constraints of the main text, we will include these ablation studies in the supplementary material.\n\n    ||PSNR|WS-PSNR|SSIM|LPIPS|\n    |:----:|:----:|:----:|:----:|:----:|\n    |w/o appearance|5.44|5.24|0.001|0.707|\n    |w/o geometry|26.25|25.25|0.839|0.263|\n    |full|28.10|27.12|0.876|0.195|\n\nBest,\n\nAuthors\n"}, {"review_id": "q0KFvRrv8K", "reviewer": "Reviewer_un5g", "summary": "This paper introduces PanoGRF, a method for generalizable novel view synthesis of sparse panorama images with wide baselines. This PanoGRF is basically built upon the perspective view synthesis method, NeuRay. Previous generalizable NeRF methods are mainly designed for perspective images are may introduce extra errors between the projection of perspective views and panorama views.\n\nTo perform generalizable view synthesis given multiple panoramas, the authors 1) design CNN modules on panoramas to extract the geometry and appearance fields, constructing the cost volumes for density and color estimation; 2) design a 360-panorama depth estimation method based on both multi-view image feature correlations and cues of monocularly estimated depth maps. The predicted depth maps are for the upcoming geometry feature extraction.\n\n ", "questions": "The paper is written very clearly, and I find no significant questions to ask.\n", "limitations": "The limitations are adequately discussed.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to understand.\n- Lots of supplementary materials are given to help the readers fully appreciate the experimental results and algorithm designs.\n- The proposed method demonstrates state-of-the-art view synthesis quality on generalizable view synthesis task from panorama images.", "weaknesses": "- The overall idea for generalizable view synthesis is not new and is basically built upon the existing NeuRay work, therefore it is a bit incremental for the research community.\n- The qualitative results are limited in indoor scenes as discussed in the limitations. It would be better if PanoGRF can be trained and evaluated on outdoor unbounded scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and comments. We are pleased to receive positive feedback on our performance and writing. We hope the following responses will address your concerns effectively.\n\n* For Weakness 1: \n>The overall idea for generalizable view synthesis is not new and is basically built upon the existing NeuRay work, therefore it is a bit incremental for the research community.\n\n    We think it is actually not incremental. Please refer to the detailed contribution clarification in our global response. \n    \n    Novel view synthesis from wide baseline panoramas is a very challenging task. As we mentioned in the \"global\" rebuttal, both NeRF and perspective generalization methods cannot achieve good results under such settings. It is non-trivial to directly apply perspective methods to wide-baseline panoramas because of different 3D representation and severe occlusion problem. We observed their limitations and proposed crucial solutions: we no longer rely on ordinary perspective inputs and planar depth (z-depth), but directly input panoramic images and spherical depth. We alignment appearance feature and geometry feature in the spherical representation. To provide a more robust geometry feature, we introduce 360&deg; monocular depth into 360&deg; MVSNet further increasing the upper limit of PanoGRF. PanoGRF is inspired by these perspective generalization methods but it is not incremental.  The comprehensive experiments validate that our solutions are vital for wide-baseline panoramas. More importantly, no similar work has been done on panoramic images.\n\nNovel view synthesis from wide-baseline panoramas is an extremely challenging task. As mentioned in our global\" response, both NeRF and perspective generalization methods struggle to achieve good results under such settings. Directly applying perspective methods to wide-baseline panoramas is non-trivial due to differences in 3D representation and the severe occlusion problem. We observed their limitations and proposed crucial solutions: instead of relying on ordinary perspective inputs and planar depth (z-depth), we directly input panoramic images and spherical depth. We align appearance and geometry features in the spherical representation. To provide a more robust geometry feature, we introduce 360&deg; monocular depth into 360&deg; MVSNet, further increasing the upper limit of PanoGRF. Although PanoGRF is inspired by perspective generalization methods, it is not incremental. Our comprehensive experiments validate that our solutions are vital for wide-baseline panoramas. More importantly, no similar work has been done on panoramic images.\n* For Weakness2: \n>The qualitative results are limited in indoor scenes as discussed in the limitations. It would be better if PanoGRF can be trained and evaluated on outdoor unbounded scenes.\n   \n    Indeed, the current absence of large-scale outdoor panoramic datasets prevents PanoGRF from being trained on outdoor scenes. Furthermore, the significant depth scale differences between indoor and outdoor settings limit PanoGRF's generalization capabilities for outdoor scenes, which also applies to other generalizable methods for perspective images. But if large-scale outdoor panoramic dataset is available, PanoGRF can also be trained and evaluated on outdoor scenes. We consider addressing this issue as part of our future work.\n\nSincerely,\n\nAuthors\n"}, {"review_id": "VdAYlAJxMm", "reviewer": "Reviewer_kb7G", "summary": "This paper presents a method called PanoGRF for synthesizing novel panoramas using two wide-baseline panoramas, with the incorporation of 360 scene priors into Spherical NeRF to generate new views. The method involves extracting appearance and geometry features from the input panoramas and estimating spherical depths through convolutions. To enhance the accuracy of 360 panoramas MVS depth estimation, the authors integrated a 360 monocular depth estimation network. Specifically, they employed a Gaussian distribution to sample depth candidates around the estimated 360 monocular depth. Extensive experiments were conducted to demonstrate the effectiveness of the proposed method in synthesizing novel views from wide-baseline panorama stereos.", "questions": "Although all experiments in the paper demonstrate the method's capability to generate views along the baseline of two cameras, the design of the proposed method suggests the potential to generate views outside of these limited regions. It would be valuable to investigate how the method performs in synthesizing views beyond the baseline. By conducting additional experiments to test the boundary and assess the performance of the proposed method in generating views outside the baseline, a more comprehensive understanding of its capabilities can be gained. This would be a beneficial addition to further validate and explore the boundaries of the proposed method.", "limitations": "The authors have mentioned the limitations and societal impact of the paper in lines 273-278. It well covered the possible limitations and potential societal impact of the paper.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "The authors have performed thorough experiments to showcase the effectiveness of the proposed method, making the study comprehensive. The results obtained from the performance evaluation and ablation study are noteworthy, highlighting that the proposed method achieves state-of-the-art performance. The contributions of the proposed components in achieving this success are significant.\n\nMoreover, the paper is excellently written and presents the concepts in a clear and easily understandable manner. The figures are well-drawn, effectively conveying the main ideas and providing visual clarity to the readers.", "weaknesses": "The level of novelty in this paper is relatively limited since many of the components utilized have been proposed by previous methods. For instance, the appearance and geometry feature aggregation techniques are adopted from NeuRay, while the 360 MVS approach, which involves cost volume and mono-guided depth sampling, can be attributed to [14, 12, 28]. Furthermore, it is not particularly challenging to adapt these techniques to panoramas by employing spherical projection.\n\nIt is worth noting that the comparison with IBRNet and NeuRay may not be entirely fair due to the differing assumptions regarding the number of input images. Both IBRNet and NeuRay assume multiple images as inputs, while this paper solely employs two images in its experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your effort for reviewing our paper. We hope the following responses could solve your major concerns.\n\n* For weakness 1: \n>The level of novelty...\n\n    \n    PanoGRF is not a simple combination of NeuRay and the 360&deg; novel view synthesis task. For a more detailed clarification, please refer to our global response.\n\n    The novel view synthesis task for wide-baseline panoramic images is indeed highly challenging. First, input views are sparse. Second, the occlusion issue is severe. To address the first challenge, we develop generalizable spherical radiance fields that incorporate 360&deg; data priors into the spherical NeRF, preventing the overfitting problem. For the second challenge, we introduce 360&deg; monocular depth to guide the spherical depth sampling of 360&deg;MVSNet.\n\n    It is also important to note that PanoGRF is not simply the combination of NeuRay and spherical projection. We built PanoGRF based on our key observations. As we metioned in the global response, NeuRay can only work on perspective images. If the perspective slices of the panoramic image are input, incorrect features are easily aggregated during the feature aggregation process due to the limited field of view of the perspective image. To solve this problem, we use spherical projection for feature alignment, maintaining a spherical image representation, and fully leveraging the omnidirectional field of view provided by panoramas. Secondly, NeuRay relies on planar depth, while PanoGRF relies on spherical depth. For generalizable renderers of panoramic images, spherical depth can provide more robust geometric features, avoid feature alignment issue similar to that mentioned earlier, and offer a full field of view and better continuity. We also observed that ordinary 360MVSNet based on merely multi-view matching cannot handle the serious occlusion problem under a wide baseline. In this regard, we proposed a feasible solution. Specifically, we use 360 monocular depth to guide the deep sampling process of spherical sweeps to introduce strong single-view priors.\n\n    Furthermore, our method is simple and highly effective, as demonstrated in the extensive comparative experiments (Sec.4.3 of the main text, Sec.E and Sec.F of the supplementary materials). These experiments validate that our solutions are crucial for wide-baseline panoramas. \n\n* For Weakness 2: \n>...the comparison with IBRNet and NeuRay may not be entirely fair...\n\n     We would like to clarify that we utilize two wide-baseline panoramic views as inputs because our research focuses on the camera baseline size rather than the number of panoramic images. By ensuring that all experiments have two panoramic image inputs, we facilitate the execution of the experiments. Correspondingly, we split each panorama into cubemaps, with six sides per panorama, and then use all the cubemaps as inputs for NeuRay/IBRNet. Consequently, NeuRay/IBRNet receives twelve perspective images as inputs. This comparison remains fair under these input conditions.\n\n    However, our method is not limited to two panoramas. For instance, when rendering a test view in the renderer module, we use $N$ input panoramas as reference images, and the renderer does not need to be modified. In the 360&deg; spherical depth estimator module, for each reference image, we use the other $N-1$ input panoramas as source images. We average the multiple cost-volumes obtained during 360&deg; multi-view matching process between the reference image and each source image. The rest is unchanged. In this way, our method can be applied to the multi-view panoramic inputs.\n\n\n    To further verify the effectiveness of our method, we conducted comparative experiments with NeuRay using multiple panoramic image as inputs. We placed the four input viewpoints at the corners of a horizontal square and tested the rendering performance at the center viewpoint and other viewpoints located at -0.4, -0.2, -0.1, -0.05, 0.05, 0.1, 0.2, and 0.4 meters in the vertical direction from the center viewpoint. The diagonal length of the square is 1.0 meters. The table below presents the quantitative comparison results between PanoGRF and NeuRay. As shown, PanoGRF still largely outperforms NeuRay with multiple panoramic inputs. The qualitative results can be found in the attached PDF of the global response.\n\n\n    |method|PSNR|WS-PSNR|SSIM|LPIPS|\n    |:----:|:----:|:----:|:----:|:----:|\n    |NeuRay|27.82|26.74|0.8614|0.2312|\n    |PanoGRF|**28.99**|**27.91**|**0.8762**|**0.2071**|\n    \n\n\n* For Question: \n>...how the method performs in synthesizing views beyond the baseline...\n    \n    Thanks for your valuable suggestion! We conducted an additional experiment where novel views were generated at positions 0.25 and 0.5 meters above the middle point between two input viewpoints. The input viewpoints are 1.0 meters apart.  We compared the quantitative results of our method with NeuRay's as shown in the table below. PanoGRF consistently surpassed NeuRay's performance, indicating its capacity to yield superior results beyond the camera baseline. For the qualitative comparison, please refer to the attached PDF file of our \"global\" rebuttal. We also present the failure case of PanoGRF under such condition in the attached PDF file. In some viewpoints, a previously occluded area may be visible and since this area has not been seen in either of the two existing viewpoints, the synthesis of this area is not effective. The area, which has not been seen in either of the existing viewpoints may be able to be filled in by combining with the existing diffusion generative approach. This is the next direction we plan to investigate.\n\n\n    |distance|method|PSNR|WS-PSNR|SSIM|LPIPS|\n    |:----:|:----:|:----:|:----:|:----:|:----:|\n    |0.25m|NeuRay|20.66|20.05|0.714|0.409|\n    |0.25m|PanoGRF|**21.98**|**21.30**|**0.763**|**0.348**|\n    |0.5m|NeuRay|20.39|19.94|0.711|0.420|\n    |0.5m|PanoGRF|**21.99**|**21.42**|**0.769**|**0.349**|\n\nBest,\n\nAuthors"}, {"review_id": "uKbN8ZngGy", "reviewer": "Reviewer_74Ux", "summary": "This work tackels large-baseline (up to 2 meters) multi-view stereo for panorama images. Extending NeuRay framework, the density and raidiance field are estimated based on the feature extracted from nearby views following panorama projection. To predict stereo depth as geometric feature, this work propose to use 360 monocular depth estimation model to sample more sphere sweeps close to the surface. All the features are extracted or estimated by 2D/3D CNN and can be trained from many scenes for generalization.", "questions": "1. Why constraint the proposed method on aggregating only 2 panorama views (Eq.3)? The original NeuRay have tried many different number of working views and find 8 nearyby views achieves saturated quality. Experiment to justify the design choice in this work is missing.\n2. The search space of the stereo depth candidacy is scene dependent and crucial. How is the $\\beta$ in L215 determined? If applying the proposed method to other datasets, is there any rule of thumb to determine the $\\sigma$ based on the scene scales?\n3. In supp's Table.3. It's strange that the more fine-grained is the sampling (i.e., more $N_{mono}$) the worse results the proposed method can achieve. The paper should discuss about this experiment. Besides, what if we using even less $N_{mono}$? Given that the best result is achieved by the minimum $N_{mono}=5$ setting in Table.3, I think the ablation to find the optimum $N_{mono}$ is incomplete.\n", "limitations": "The discussed limitation looks reasonable.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The quantitative and qualitative results are better than the compared baselines.", "weaknesses": "**Adapting from perspective cubemap to equirectangular projection is not a big contribution.**\nOne of the main claimed disadvantage of some of the baseline methods (i.e., IBRNet and NeuRay in this work) is that their original codebases work on perspective instead of panoramic imagery (L36-38). However, adapting perspective projection to equirectangular projection is trivial. Some common adaptation like circular padding is easy to add as well. Many deep-learning based models also show promising quality by adapting CNN on equirectangular panorama like LayoutNet[Zou et al] and DuLa-Net[Yang et al], just name a few. With these simple adaptation, the baseline methods can also be directly trained on equirectangular panorama without the claimed disadvantage.\n\n**Some design choices are not well justified and discussed.** Please refer to question 1,2,3 for detail.\n\n**Paper writing**\n- Sec.3.1 L122-123. $w_i$ is not the accumulated transmittance. Only the $\\prod_{k=1}^{i-1} (1 - \\alpha_k)$ is.\n- The $w$ is duplicated in Eq.1 and Eq.4 but they are totally different things. I suggest using different notations.\n- Table.1 and Table.2. It's better to directly use a table footnote describing the \"*\" means the models are trained with perspective cubemap projection and evaluated with equirectangular projection.\n- L140. \"When\" should be \"when\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 74Ux: \n\nThanks for the review and comments.  We hope the following responses could solve your concerns.\n\n* For Weakness 1: \n>Adapting from perspective cubemap to equirectangular projection...\n\n     We illustrate the key differences between PanoGRF and NeuRay from many aspects in the \"global\" response. Please refer to the more detailed clarification in our global response.\n\n    The novel view synthesis task for wide-baseline panoramic images is challenging. First, input views are sparse. Second, the occlusion issue is severe. To tackle the first challenge, we develop generalizable spherical radiance fields that incorporate 360&deg; data priors into the spherical NeRF. For the second challenge, we introduce 360&deg; monocular depth into PanoGRF.\n    \n    We build up PanoGRF based on our key observations. NeuRay can only work on perspective images. If the perspective slices of the panoramic image are input, incorrect features are easily aggregated during the feature aggregation process due to the limited field of view of the perspective image. To solve this problem, we use spherical projection for feature alignment, maintaining a spherical image representation. Secondly, NeuRay relies on planar depth, while PanoGRF relies on spherical depth. Spherical depth can provide more robust geometric features, avoid feature alignment issue similar to that mentioned earlier, and offer a full field of view and better continuity. We also observed that ordinary 360&deg; MVSNet based on merely multi-view matching cannot handle the serious occlusion problem under a wide baseline. We use 360 monocular depth to guide the deep sampling process of spherical sweeps to introduce strong single-view priors. Furthermore, our method is simple and highly effective. \n    \n    Adapting perspective to equirectangular is not trival. Many papers, such as [1, 2], worked on this. Besides, we also designed the additional targeted methods for improvements, as mentioned above. We also have attempted to improve the performance of CNN in PanoGRF to alleviate spherical distortion by introducing part of the methods [1, 2]. But these methods make almost no difference and increase the complexity of PanoGRF for 360&deg; view synthesis task. What truly works in the synthesis of wide-baseline panoramic views are our proposed solutions.\n\n* For Weakness 2: \n>...Please refer to question 1,2,3 for detail.\n    \n    We answer your questions in subsequent paragraphs.\n\n* For Weakness 3: \n>Paper writing...\n    \n    Thanks for your correction. According to your suggestions, we will correct these mistakes in the main body of our paper.\n\n* For Question 1: \n>Why constraint the proposed method on aggregating only 2 panorama views...\n\n    We utilize two wide-baseline panoramic views as inputs because our research focuses on the camera baseline size rather than the number of panoramic images. Therefore, we conducted comparative experiments under varying camera baselines. \n\n    However, our method is not limited to two panoramas. To further verify the effectiveness of our method, we conducted comparative experiments with NeuRay using multiple panoramic image as inputs. We placed the four input viewpoints at the corners of a horizontal square and tested the rendering performance at the center viewpoint, as well as other viewpoints located at -0.4, -0.2, -0.1, -0.05, 0.05, 0.1, 0.2, and 0.4 meters in the vertical direction from the center viewpoint. The diagonal length of the square is 1.0 meters. The table below presents the quantitative comparison results. As shown, PanoGRF still largely outperforms NeuRay with multiple panoramic inputs. The qualitative results can be found in the attached PDF of the global response.\n\n    |method|PSNR|WS-PSNR|SSIM|LPIPS|\n    |:----:|:----:|:----:|:----:|:----:|\n    |NeuRay|27.82|26.74|0.8614|0.2312|\n    |PanoGRF|**28.99**|**27.91**|**0.8762**|**0.2071**|\n\n* For Question 2: \n>...How is the \\beta in L215 determined...\n\n\n    $\\beta$ is taken to be 3 because [$\\mu$-3*$\\sigma$, $\\mu$+3*$\\sigma$] is a frequently used interval in Gaussian sampling in the vicinity of the expectation value.\n    \n    The value $\\sigma$=0.5 is obtained from experiments on Matterport3D, where we tested the values 0.1, 0.5, 1.0, and 1.5, with 0.5 proving to be the most effective. In general cases, this value does not require adjustment for indoor scenes. However, if the depth scale of a scene is particularly large, this value may need adaptation. For instance, $\\sigma$ can be set as 1/20 of the maximum depth of the scene.\n\n* For Question 3: \n>In supp's Table.3. It's strange that ...\n\n    Thanks for your suggestion! We added the experiment as follows: \n    \n    |$N_{mono}$|L1|L2|RMSE|WS-L1|WS-L2|WS-RMSE|\n    |:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n    |1|0.1586|0.2498|0.4317|0.1745|0.1971|0.3937|\n    |3|**0.1432**|**0.1993**|**0.3865**|0.1529|0.1649|0.3580|\n    |5|0.1441|0.2047|0.3877|**0.1502**|**0.1624**|**0.3546**|\n    |7|0.1496|0.2252|0.4104|0.1640|0.1896|0.3832|\n    |9|0.1645|0.2912|0.4511|0.1752|0.2215|0.4059|\n\n    The reliability of 360&deg; monocular depth estimation is not perfect. Therefore, our paper employs uniform sampling to compensate for the remaining depth candidates. In the ablation experiments, we consistently maintain $N_{mono}+N_{uni}=64$, where $N_{uni}$ represents the sample number of uniform distribution. We discovered that the configuration $N_{mono}=5$ yields the best results with regard to the metrics of WS-L1, WS-L2, and WS-RMSE.\n\nSincerely,\n\nAuthors\n\n[1] Jiang, H., Sheng, Z., Zhu, S., Dong, Z., & Huang, R. (2021). Unifuse: Unidirectional fusion for 360 panorama depth estimation. IEEE Robotics and Automation Letters, 6(2), 1519-1526.\n\n[2] Li, Y., Guo, Y., Yan, Z., Huang, X., Duan, Y., & Ren, L. (2022). Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2801-2810).\n"}, {"review_id": "7j2GHbXdZf", "reviewer": "Reviewer_WLFo", "summary": "They describe a novel method for panoramic view synthesis given wide-baseline panoramas as input.  They first obtain depth maps for the input panoramas use a custom spherical depth estimation method guided by a pre-trained monocular depth estimator.  They then extract geometry and appearance features from the input panoramas.  To render a new panorama, they sample points along the viewing rays,  project them into the feature grids, aggregate features using an estimated visibility term, decode the aggregated features into color+density, and compute final color using volume rendering.  They outperform recent generalizable view synthesis methods on Matterport3D, Replica, and Residential datasets.", "questions": "The videos only demonstrate camera movement along a line with no vertical movement.  Does the method support complete 6DOF motion or is limited to planar motion?", "limitations": "The discussion of limitations is rather brief.  They could explore failure cases more and highlight cases where the view synthesis result could be improved.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "They combine many ideas from prior work on perspective and spherical view synthesis but synthesizing these ideas into a novel approach.  The closest work is probably NeuRay, which is a generalizable radiance field representation for perspective images.  The main differences to NeuRay are that they work in spherical coordinates and they use monocular depth estimation to contribute to the visibility prediction and depth-guided sampling.  \n\nTheir evaluation is thorough and includes a comparison to a reasonable selection of recent view synthesis methods, including some designed for panoramas and some adapted to panoramas by applying the method to each side of a cube map.  Quantitatively they show a large improvement in rendering quality and this is also evident from the example images in the figures and the videos in the supplementary.  \n\nThey also include an ablation study in which they verify the usefulness of including 360 monocular depth and multi-view stereo in the method.\n\nAnother contribution is the insight (reported in the supplemental) that fine-tuning did not lead to an improvement in rendering quality in their experiments, contrary to what was reported by NeuRay for example.  Instead fine-tuning led to overfitting.  They hypothesize that this is due to the use of wide-baseline images.  This is an interesting result and points to potential future work to overcome this limitation.", "weaknesses": "In many ways the work is an adaptation of NeuRay to the spherical case.  It could be argued that the work is somewhat incremental in that sense.  However, their results for panoramic view synthesis are excellent and there is not a lot of previous work on wide-baseline panoramic view synthesis.\n\nBecause the name includes GRF, I would think it appropriate to mention and cite GRF which is an earlier general radiance field approach which is a precursor to NeuRay:\n\nTrevithick, A., & Yang, B. (2021). GRF: Learning a general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15182-15192).\n\nAnother weakness is that the provided video only shows planar movement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer WLFo: \n\n\nThank you for the review and comments. We are glad to see your positive comments and score. We hope the following responses could solve your concerns.\n* For Weakness1: \n>In many ways the work is an adaptation of NeuRay to the spherical case. It could be argued that the work is somewhat incremental in that sense. However, their results for panoramic view synthesis are excellent and there is not a lot of previous work on wide-baseline panoramic view synthesis.\n\n    We appreciate your thoughtful comments. Although our method is inspired by using NeuRay to the spherical case, it goes much more beyond that. To address the occlusion issue, we additionally employ 360&deg; monocular depth to guide the spherical depth sampling in sphere sweeps of 360&deg; MVSNet. More insights behind our design can be seen in the global response.\n\n* For Weakness 2: \n>Because the name includes GRF, I would think it appropriate to mention and cite GRF which is an earlier general radiance field approach which is a precursor to NeuRay...\n\n    Thanks for your nice suggestion. We will mention and cite GRF in the related work. \n* For Question: \n>The videos only demonstrate camera movement along a line with no vertical movement. Does the method support complete 6DOF motion or is limited to planar motion?\n\n    Yes, PanoGRF support complete 6DOF motion and is not limited to planar motion. To validate this point,  we conducted an additional experiment where we synthesized novel views at positions 0.25 and 0.5 meters above the middle point between two input viewpoints. The input viewpoints are 1.0 meters apart. We compared the quantitative results of our method with NeuRay's using various indicators, as shown in the table. Our results consistently outperformed NeuRay's, demonstrating that our method can achieve better results than NeuRay beyond the camera baseline. For the qualitative comparison, please refer to the attached PDF of our \"global\" rebuttal.\n\n    |distance|method|PSNR|WS-PSNR|SSIM|LPIPS|\n    |:----:|:----:|:----:|:----:|:----:|:----:|\n    |0.25m|NeuRay|20.66|20.05|0.714|0.409|\n    |0.25m|PanoGRF|**21.98**|**21.30**|**0.763**|**0.348**|\n    |0.5m|NeuRay|20.39|19.94|0.711|0.420|\n    |0.5m|PanoGRF|**21.99**|**21.42**|**0.769**|**0.349**|\n\n* About limitation: \n>The discussion of limitations is rather brief. They could explore failure cases more and highlight cases where the view synthesis result could be improved.\n\n    Great thanks for your suggestion! The qualitative result of the failure case can be seen in the attached PDF of our \"global\" rebuttal. In some viewpoints, a previously occluded area may be visible, and since this area has not been seen in either of the two existing viewpoints, the synthesis of this area is not effective. The area, which has not been seen in either of the existing viewpoints, may be able to be filled in by combining with the existing diffusion generative approach. This is the next direction we plan to investigate. \n\nBest,\n\nAuthors\n"}]

Author rebuttal:
Dear Reviewers:

Thank you for acknowledging the strong performance of this work. As Reviewer un5g said, quantitatively PanoGRF shows a large improvement in rendering quality and this is also evident from the example images in the figures and the videos in the supplementary. We clarify the contributions of our paper as follows to highlight our unique insights and demonstrate that our method is not just a simple composition of exsiting techniques.

The novel view synthesis task for wide-baseline panoramic images is highly challenging. Firstly, the sparsity of input views makes it difficult for NeRF to learn the correct geometry. Secondly, certain areas of the scene may be visible from only one view and occluded in others, making it difficult to provide accurate geometry using 360&deg; multi-view stereo alone. To tackle the first challenge, we develop generalizable spherical radiance fields that incorporate 360&deg; data priors into the spherical NeRF, preventing the overfitting problem. For the second challenge, we introduce 360&deg; monocular depth to guide the spherical depth sampling of 360&deg; MVSNet.



* Perspective generalization methods (such as NeuRay) inherently has some drawbacks as they only can be applied to perspective slicings obtained from panoramas. PanoGRF is dedicated for panoramic inputs based on our key observations.

    * __feature aggregation__: We observed that when employing common perspective view synthesis methods, these approaches encounter a significant issue during feature aggregation: due to the limited field of view of the perspective view, a 3D sample point is often projected outside the perspective view or behind the camera (z-depth<0). Aggregating features in this manner is incorrect. Recognizing the limitations of perspective generalization methods led us to the idea of using spherical projection for feature alignment, maintaining a spherical image representation, and fully leveraging the omnidirectional field of view provided by panoramas. We have demonstrated the importance of this approach through comparative experiments in Sec.4.3 of the main text, Sec.E and Sec.F of the supplementary materials.

    * __depth estimation__: NeuRay relies on planar depth (z-depth) while PanoGRF relies on spherical depth. It is more robust to use spherical depth predicted by 360&deg; MVSNet than planar depth predicted by ordinary MVSNet. The process of the multi-view feature matching process in perspective images suffers from a similar feature alignment problem as mentioned earlier. Moreover, splitting panoramas into cubemaps to compute z-depth disrupts the neighborhood relationship between cubemaps, which can affect accuracy. Besides, we observed that the occlusion issue is quite severe under the wide-baseline setting. Existing 360&deg; methods [1, 2, 3, 4, 5] cannot handle wide-baseline 360&deg; depth estimation in general cases. Methods [1, 2, 3] overlooked the occlusion problem under wide baselines. Methods [4, 5] considered occlusion but utilized a special camera rig to capture wide field-of-view images, making them unsuitable for wide-baseline 360° depth estimation tasks with general 360° cameras. We propose a feasible and effective solution to address this issue, which we introduce later.


	We believe that our observations can contribute to and benefit related research within the community.
* Relying solely on 360&deg; multi-view stereo is insufficient to handle the occlusion issue. To address this problem, we employ a 360&deg; monocular depth network to guide the depth sampling of 360&deg; MVSNet during sphere sweeps and contribute to the visibility prediction of the generalizable renderer.  We conducted ablation studies with various camera baselines to verify that 360&deg; MVSNet and 360&deg; Monocular Depth, despite their individual shortcomings, can complement each other and play a crucial role in the task of 360&deg; novel view synthesis under wide baselines. 




In summary, we propose easy-to-implement and highly effective solutions to the challenges we encountered, and have conducted comprehensive comparisons and ablation studies to demonstrate the effectiveness of our method. Our method significantly outperforms baseline methods (including NeuRay) on several panorama datasets. Considering that Reviewer zV2U, un5g and WLFo have all agreed that our paper has great merits, we believe our research findings are worth sharing with the community. 



Best,

Authors

[1] Wang, N. H., Solarte, B., Tsai, Y. H., Chiu, W. C., & Sun, M. (2020, May). 360sd-net: 360 stereo depth estimation with learnable cost volume. In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 582-588). IEEE.

[2] Xie, S., Wang, D., & Liu, Y. H. (2023). OmniVidar: Omnidirectional Depth Estimation From Multi-Fisheye Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21529-21538).

[3] Chiu, C. Y., Wu, Y. T., Shen, I., & Chuang, Y. Y. (2023). 360MVSNet: Deep Multi-View Stereo Network With 360deg Images for Indoor Scene Reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 3057-3066).

[4] Won, C., Ryu, J., & Lim, J. (2019). Omnimvs: End-to-end learning for omnidirectional stereo matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8987-8996).

[5] Won, C., Ryu, J., & Lim, J. (2020). End-to-end learning for omnidirectional stereo matching with uncertainty prior. IEEE transactions on pattern analysis and machine intelligence, 43(11), 3850-3862.

Meta-review: This paper presents PanoGRF for synthesizing novel panoramas using two wide-baseline panoramas, with the incorporation of 360-depth priors into Spherical NeRF to generate new views. The paper received mixed ratings. Reviewers kb7G and 74Ux recommend rejection due to limited novelty. In their point of view, the proposed method is simply an adaptation of NeuRay to the spherical case and uses mono-depth to mitigate the wide baseline challenge. However, reviewer zV2U thinks the novelty is enough, and reviewer un5g thinks the task of novel views synthesis from wide-baseline images itself is novel. Moreover, reviewers WLFo, un5g, and zV2U agree that the results are impressive. The AC read the paper and discussion carefully. The proposed method is indeed tackling a very practical task with little prior work. Hence, it is beneficial to both the research and industry community. The qualitative and quantitative results of the proposed method are also impressive. Hence, AC recommends acceptance.
Decision: Accept (poster)