SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zheng Chen $^{1}$ , Yan-Pei Cao $^{2}$ , Yuan-Chen Guo $^{1}$ , Chen Wang $^{1}$ , Ying Shan $^{2}$ , Song-Hai Zhang $^{1}$
$^{1}$ Tsinghua University  $^{2}$ ARC Lab, Tencent PCG
{chenz20,guoc19}@mails.tsinghua.edu.cn caoyanpei@gmail.com
cw.chenwang@outlook.com yingsshan@tencent.com shz@tsinghua.edu.cn
Achieving an immersive experience enabling users to explore virtual environments with six degrees of freedom (6DoF) is essential for various applications such as virtual reality (VR). Wide-baseline panoramas are commonly used in these applications to reduce network bandwidth and storage requirements. However, synthesizing novel views from these panoramas remains a key challenge. Although existing neural radiance field methods can produce photorealistic views under narrow-baseline and dense image captures, they tend to overfit the training views when dealing with wide-baseline panoramas due to the difficulty in learning accurate geometry from sparse  $360^{\circ}$  views. To address this problem, we propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas, which construct spherical radiance fields incorporating  $360^{\circ}$  scene priors. Unlike generalizable radiance fields trained on perspective images, PanoGRF avoids the information loss from panorama-to-perspective conversion and directly aggregates geometry and appearance features of 3D sample points from each panoramic view based on spherical projection. Moreover, as some regions of the panorama are only visible from one view while invisible from others under wide baseline settings, PanoGRF incorporates  $360^{\circ}$  monocular depth priors into spherical depth estimation to improve the geometry features. Experimental results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-art generalizable view synthesis methods for wide-baseline panoramas (e.g., OmniSyn) and perspective images (e.g., IBRNet, NeuRay). Project Page: https://thucz.github.io/PanoGRF/.
The rise of  $360^{\circ}$  cameras and virtual reality headsets has fueled the popularity of  $360^{\circ}$  images among photographers and tourists. Commercial VR platforms, such as Matterport 2, enable users to experience virtual walks in  $360^{\circ}$  scenes by interpolating between panoramas [17]. Wide-baseline panoramas are frequently employed on these platforms for capture and network transmission to reduce storage space and bandwidth requirements. Consequently, synthesizing novel views from wide-baseline panoramas is an essential task for providing a seamless six degrees of freedom (6DoF) experience to users.
However, synthesizing novel views from a pair of wide baseline panoramas encounters two primary challenges. First, the input views are sparse, causing existing state-of-the-art methods such as NeRF [21] to struggle in learning accurate geometry due to the shape-radiance ambiguity [37],
*Work done during an internship at ARC Lab, Tencent PCG
$^{2}$ Matterport: https://matterport.com
leading to overfitting of the input views. Second, certain regions in the scene may only be visible in one view but not in another, making it challenging to provide a correct geometry prior to NeRF using only multi-view stereo. State-of-the-art generalizable NeRF methods [35, 32, 20] address the overfitting problem by incorporating scene priors into NeRF; however, they are designed for perspective images and require conversion of panoramicas into perspective views when applied, resulting in information loss and suboptimal performance due to the limited field of view. Although spherical radiance fields [11] have been proposed to render panoramic images based on spherical projection, they still suffer from the overfitting problem without considering the  $360^{\circ}$  scene prior. Other approaches designed for  $360^{\circ}$  view synthesis, such as multi-sphere images (MSI) [2] and mesh-based methods [17], exhibit limited expressiveness and rendering quality compared to NeRF due to the use of discretized scene representations.
In this work, we present a method that addresses the overfitting issues in spherical radiance fields by incorporating  $360^{\circ}$  scene priors. Unlike existing generalizable methods that require panorama-to-perspective conversion, our approach retains the panoramic representation. Furthermore, we incorporate  $360^{\circ}$  monocular depth to alleviate the view occlusion problem.
To address the first challenge, we present a solution in the form of generalizable spherical radiance fields. To render the panorama at a new position, spherical radiance fields (Spherical NeRF) cast a ray for each pixel using spherical projection, sample points along the ray, and aggregate the colors of these points based on their density values following NeRF [21]. In our method, named PanoGRF, we incorporate  $360^{\circ}$  scene priors into Spherical NeRF. Specifically, we extract appearance and geometry features from input panoramas and estimated spherical depths through convolutions, respectively. PanoGRF accurately aligns the queried 3D points with the corresponding pixels in panoramas using spherical projection. This alignment strategy leverages the full field-of-view characteristic of panoramas and eliminates the information loss of panorama-to-perspective conversion. The local appearance and geometry features at these corresponding pixels are then aggregated and serve as the conditional input for Spherical NeRF.
To tackle the second challenge, we enhance the accuracy of depth estimation in  $360^{\circ}$  multi-view stereo by integrating a  $360^{\circ}$  monocular depth estimation network. In  $360^{\circ}$  multi-view stereo, depth inaccuracies can arise due to view inconsistencies caused by occluded regions. Inspired by the perspective method proposed in [3], we sample depth candidates around the estimated  $360^{\circ}$  monocular depth, assuming a Gaussian distribution. These depth candidates are then utilized in sphere sweeps during  $360^{\circ}$  multi-view matching. By incorporating monocular depth candidates, we can improve the accuracy of depth estimation in regions with view inconsistencies. This improvement contributes to the robustness of the geometry features employed in PanoGRF and ultimately leads to superior view synthesis performance.
In summary, we make the following contributions: 1) We propose the design of generalizable spherical radiance fields to learn  $360^{\circ}$  scene priors that alleviate the overfitting problem when dealing with wide-baseline panoramas. 2) We incorporate  $360^{\circ}$  monocular depth into the spherical multi-view stereo to mitigate the view inconsistency problem caused by occluded regions, which results in more robust geometry features, ultimately improving the rendering performance. 3) We achieve the state-of-the-art novel view synthesis performance on Matterport3D [4], Replica [28] and Residential [8] under the wide-baseline setting.
In this section, we briefly review the methods of perspective view synthesis and  $360^{\circ}$  view synthesis.
2.1 Perspective View Synthesis
Recently, NeRF (neural radiance fields [21]) has become the mainstream scene representation due to its photo-realistic rendering quality for perspective view synthesis. However, given sparse input views, NeRF's per-scene optimization approach is prone to overfitting the training views and fails to learn the correct geometry due to the shape-radiance ambiguity, leading to significant floaters in novel views. Several subsequent methods [12, 6, 15, 22, 5, 34] have attempted to address this issue by incorporating depth constraints from COLMAP [26] or other regularization terms, but they still rely on per-scene optimization and lack the ability to generalize to new scenes. In contrast,
Figure 1: The overview of PanoGRF. Initially, we employ convolutional neural networks (CNNs) to extract appearance features and geometry features from the input views. The geometry feature is obtained from the predicted spherical depth generated by our proposed Mono-guided  $360^{\circ}$  Depth Estimator (Sec. 3.3). Next, we cast rays based on spherical projection to render a novel  $360^{\circ}$  view (Sec. 3.1). Along each ray, every 3D sample point  $\pmb{p}_i$  is projected onto the corresponding panoramic grid coordinate  $(u,v)$ . The local geometry feature  $\pmb{g}_{i,j}$  at  $(u,v)$  is decoded into  $v_{i,j}$  (the visible probability of  $\pmb{p}_i$  for the  $j$ -th view). The local appearance feature  $\pmb{f}_{i,j}$  at  $(u,v)$  is aggregated into  $\pmb{f}_i$  along with  $v_{i,j}$  (Sec. 3.2). Subsequently, the aggregated feature  $\pmb{f}_i$  is decoded to determine the color and density of  $\pmb{p}_i$ . Finally, a novel  $360^{\circ}$  view is synthesized through volume rendering (Sec. 3.1).
PixelNeRF [35], GRF [30] and IBRNet [32] condition NeRF rendering with pixel-aligned appearance features to learn scene priors from large-scale datasets, enabling direct inference with input images. NeuRay [20] aggregates visibility information (obtained from multi-view stereo) and appearance features from each view to generate the density and color of 3D sample points. However, when dealing with panoramas, these methods require converting the panoramas to perspective images, leading to information loss and suboptimal performance due to limited field-of-view in each perspective view. In contrast, PanoGRF operates directly on panoramas using spherical projection, eliminating the need for the panorama-to-perspective conversion.
2.2  $360^{\circ}$  View Synthesis
Multi-sphere images (MSI) are widely adopted for  $360^{\circ}$  view synthesis. It is introduced by [2] to represent scenes with the input of omnidirectional stereo video. SOMSI [8] improved the expressive ability of MSI by incorporating high-dimensional feature layers and achieved real-time performance. However, MSI's capability is limited to narrow-baseline scenarios due to the back surface rendering issues as discussed in [18], which attempted to expand the renderable viewpoint range of MSI through the interpolation of two MSI instances. But their pure convolutional architecture only works at low resolutions. OmniSyn [17] constructs two meshes for input panoramas using  $360^{\circ}$  multi-view stereo. It warps the meshes to the target view and fuses the colors attached to the meshes with an in-painting network. This approach frequently produces ghosting artifacts due to surface-based warping and blending.  $360^{\circ}$  Roam [11] divides NeRF into multiple blocks to achieve real-time rendering of panoramas, akin to KiloNeRF [23]. But it fits a single scene without considering scene priors, making it unsuitable for wide-baseline panoramas. In contrast to the aforementioned methods, we introduce generalizable spherical radiance fields to learn both geometric and appearance priors from  $360^{\circ}$  datasets, enabling better generalization to unseen  $360^{\circ}$  scenes.
However,  $360^{\circ}$  multi-view stereo alone cannot provide a robust geometric prior, as it fails to address the occlusion issue in the wide-baseline setting. Recently, numerous researchers have focused on  $360^{\circ}$  monocular depth estimation [13, 19, 1] to mitigate spherical distortion by harnessing the fusion of equirectangular and perspective projections. Inspired by MaGNet [3], we employ  $360^{\circ}$  monocular depth prior to guide the construction of  $360^{\circ}$  cost volume, which enhances the quality of geometric features and boosts rendering performance.
PanoGRF facilitates novel view synthesis from wide-baseline panoramicas as shown in Fig. 1. Before introducing PanoGRF, we first review NeRF and its spherical variant in Sec. 3.1. We demonstrate the two important parts of PanoGRF, the generalizable spherical radiance fields, and the mono-guided spherical depth estimator in Sec. 3.2 and Sec. 3.3 respectively.
3.1 Spherical Radiance Fields (Spherical NeRF)
To render a pixel, NeRF [21] casts a ray from a given viewpoint and parameterizes the ray as  $\pmb{p}(t) = \pmb{o} + t\pmb{d}, t \in \mathbb{R}^{+}$ , where  $\pmb{o}$  is the camera center and  $\pmb{d}$  is the ray direction.  $N$  points are sampled along the ray with increasing  $t$ , we denote  $\pmb{p}_i \equiv \pmb{p}(t_i), i = 1, \dots, N$ . NeRF utilizes an MLP to map the position  $\pmb{p}_i$  together with viewing direction  $\pmb{d}$  into color  $\pmb{c}_i$  and density  $\sigma_i$ . The pixel color of the ray  $\pmb{p}$  can then be computed by:
$$
\boldsymbol {c} = \sum_ {i = 1} ^ {N} w _ {i} \boldsymbol {c} _ {i}, \tag {1}
$$
where  $\pmb{c} \in \mathbb{R}^3$  is the rendered color of  $\pmb{p}$ , and  $\pmb{c}_i \in \mathbb{R}^3$  is the color of the sampled point  $\pmb{p}_i$ .  $w_i$  is the blending weight of the point  $\pmb{p}_i$ , which indicates the probability that a ray travels from the origin to  $t_i$  without hitting any particle and terminates in the range  $(t_i, t_{i+1})$ . It is computed by  $w_i = \prod_{k=1}^{i-1} (1 - \alpha_k) \alpha_i$ , where  $\alpha_i$  is the alpha values in the depth range  $(t_i, t_{i+1})$ ,  $\alpha_i = 1 - \exp(-\sigma_i \delta_i)$  and  $\delta_i = t_{i+1} - t_i$ . The color  $\pmb{c}$  of a ray  $\pmb{p}$  is optimized by a photometric loss:
$$
\mathcal {L} = \sum \| \boldsymbol {c} - \boldsymbol {c} _ {g t} \| ^ {2}, \tag {2}
$$
where  $c_{gt}$  represents the ground truth color.
Ray-casting based on spherical projection The panorama uses the panoramic pixel grid coordinate system, while the coordinate systems in NeRF are all Cartesian. By employing spherical projection [8, 11], a pixel  $(u,v)$  in the panorama is firstly transformed into spherical polar coordinate  $(\phi ,\theta)$  and subsequently into cartesian coordinate  $(x,y,z)$ . Detailed transformation formulas can be seen in the supplementary material. The ray direction emitted from the pixel  $(u,v)$  can be computed by  $d = \mathbf{R}[x,y,z]^T$ , where  $\mathbf{R}\in \mathbb{R}^{3\times 3}$  is the panorama's camera-to-world rotation matrix.
3.2 Generalizable Spherical Radiance Fields
Under a wide baseline, Spherical NeRF tends to overfit training views and struggle to generate plausible novel views. To address this limitation, we draw inspiration from the generalizable NeRF approaches [35, 32, 20] designed for perspective images. In our work, we propose to incorporate  $360^{\circ}$  scene priors into Spherical NeRF by aggregating local features from input panoramas.
Alignment based on spherical projection Panoramas can be converted into perspective images, and then generalizable NeRF methods for perspective images can be applied to novel  $360^{\circ}$  view synthesis. However, when projecting the 3D sample point  $\pmb{p}_i$  onto the source perspective view, it may fall outside the image borders or even be located behind the source perspective camera (with a  $z$ -depth  $< 0$ ) due to the limited field-of-view. This can introduce errors in the aggregation of features and result in poor rendering results. To overcome this problem, we directly align  $\pmb{p}_i$  with the corresponding pixels in panoramas using spherical projection, enabling us to leverage the full field-of-view characteristic of panoramas. We first transform  $\pmb{p}_i$  into the camera's Cartesian coordinate system  $(x,y,z)$  of the  $j$ -th panoramic view  $I_j$ . Subsequently,  $(x,y,z)$  is converted into spherical polar coordinate  $(\theta,\phi,t)$ $(t\in \mathbb{R}^{+}$  indicates the spherical depth of  $\pmb{p}_i$  for  $I_j$ ) and finally transformed into the panoramic grid coordinate  $(u,v)$ . The specific formulas for these transformations can be found in the supplementary material.
Appearance and geometry feature aggregation We follow NeuRay [20] to aggregate appearance and geometry features. But differently, we take panoramicas and spherical depth (radial distance from the camera center) as input instead of perspective images and  $z$ -depth. PanoGRF respectively extracts appearance feature  $\mathbf{W}_j$  from the  $j$ -th panoramic view ( $j = 1, 2$ ) and geometry feature  $\mathbf{G}_j$  from
Figure 2: The process of mono-guided  $360^{\circ}$  depth estimation. We first extract the image features of reference and source views with convolutions. Using spherical projection, we determine the corresponding pixel  $(u,v)_{src}$  in the source view for each pixel  $(u,v)_{ref}$  of the reference view, with the depth hypothesis of  $t_i$ . The similarity between the local features at  $(u,v)_{ref}$  and  $(u,v)_{src}$  is computed as the value of the cost volume at  $(u,v,i)$ . Depth sampling of  $t_i$  is guided by the  $360^{\circ}$  monocular depth using a Gaussian distribution assumption. The cost volume is obtained after  $D$  sphere sweeps. Lastly, the cost volume is decoded into  $360^{\circ}$  depth using convolutions.
the spherical depth using convolutions. Details for getting the spherical depth will be introduced in Sec. 3.3. For a given 3D sample point  $\pmb{p}_i$ , we project it onto the panoramic grid coordinate  $(u, v)$ , and extract the local appearance feature  $\pmb{f}_{i,j} = \mathbf{W}_j(u, v)$  and the local geometry feature  $\pmb{g}_{i,j} = \mathbf{G}_j(u, v)$  at  $(u, v)$ . The local geometry feature  $\pmb{g}_{i,j}$  is decoded into the visibility  $v_{i,j}$  (demonstrated below) using an MLP. The local appearance feature  $\pmb{f}_{i,j}$  and visibility  $v_{i,j}$  of the  $j$ -th panoramic view are aggregated for  $\pmb{p}_i$  with an aggregation network  $\mathcal{F}$ :
$$
\boldsymbol {f} _ {i} = \mathcal {F} \left(\left\{\boldsymbol {f} _ {i, j}, v _ {i, j} \mid j = 1, 2 \right\}\right). \tag {3}
$$
According to the local geometry feature  $\pmb{g}$  of an input panorama, we predict the visibility function  $v(t)$  to indicate the probability that a point at spherical depth  $t$  (instead of  $z$ -depth in NeuRay) is visible for the input panorama.  $v(t)$  ( $v(t) \in [0,1]$ ) is represented as  $v(t) = 1 - o(t)$ , where  $o(t)$  is the occlusion probability. To parameterize  $o(t)$ , we employ a mixture of  $N_{l}$  logistic distributions:
$$
o (t; \mu_ {k}, \sigma_ {k}, m _ {k}) = \sum_ {k} ^ {N _ {l}} m _ {k} S ((t - \mu_ {k}) / \sigma_ {k}), \tag {4}
$$
where  $\mu_{k},\sigma_{k}$  and  $m_{k}$  are the mean, standard variance, and blending weight of the  $k$  -th logistic distribution respectively.  $\sum_{i}^{N_{l}}m_{k} = 1$  and  $S(\cdot)$  denotes a sigmoid function. The parameters  $[\mu_k,\sigma_k,m_k]$  are decoded from the local geometry feature vector  $\pmb{g}$  by an MLP. For each 3D sample point  $\pmb{p}_i$ , we compute its spherical depth  $t_i$  for  $j$  -th input view based on spherical projection and obtain the visible probability  $v_{i,j} = v_{j}(t_{i})$  of  $\pmb{p}_i$  for the  $j$  -th panoramic view. Then we aggregate  $v_{i,j}$  into  $\pmb{f}_i$  by Eq. 3 and decode  $\pmb{f}_i$  into the color and density of  $\pmb{p}_i$ .
3.3 Mono-guided Spherical Depth Estimator
To obtain the visibility in Sec. 3.2, we need to predict the spherical depth of each input view. The process is illustrated in Fig. 2.
$360^{\circ}$  multi-view stereo (MVS) With the input of multi-view panoramas, we estimate the spherical depth of the reference view using  $360^{\circ}$  multi-view stereo similar to previous methods [17, 14]. Firstly, we extract the image features of the reference and source views with an image encoder. For each pixel  $(u,v)_{ref}$  of the reference view, assuming its depth is  $t_i$ , we find its corresponding pixel  $(u,v)_{src}$  in
the source view according to spherical projection. We uniformly sample  $D$  depth candidates for  $t_i$  (with  $i = 1,2,\ldots,D$ ) without considering the monocular depth prior, which will be introduced later. Next, we compute the similarity between the local features at  $(u,v)_{ref}$  and  $(u,v)_{src}$ , considering it as the value of the cost volume at  $(u,v,i)$ . Assuming the length of the local feature vector is  $F$ , this process results in a 4D cost volume  $\mathbf{V} \in \mathbb{R}_{\frac{H}{4} \times \frac{W}{4} \times D \times F}$  after  $D$  sphere sweeps, where  $\frac{H}{4}$  and  $\frac{W}{4}$  respectively represent the height and width of the feature maps for the reference view. Subsequently, the dimension of the cost volume is reduced to  $\frac{H}{4} \times \frac{W}{4} \times D$  through a 3D CNN. Lastly, the processed cost volume is decoded into  $360^{\circ}$  depth using several convolution layers.
Mono-guided spherical depth sampling Under the wide-baseline setting, some areas might be visible from one view but occluded from another, in which case  $360^{\circ}$  MVS may struggle to produce accurate depth estimates. To address this challenge, we leverage the  $360^{\circ}$  monocular depth [13] to guide the depth sampling of  $360^{\circ}$  MVS, inspired by perspective methods [36, 31, 3].
For each pixel  $(u,v)$  in the panorama, we denote its estimated spherical depth from the monocular depth network as  $\mu_{u,v}$ . We generate depth candidates in the vicinity of the monocular depth using a Gaussian distribution assumption. Specifically, a search space for each pixel in the panorama is defined as  $[\mu_{u,v} - \beta \sigma, \mu_{u,v} + \beta \sigma]$  where  $\beta$  and  $\sigma$  serve as hyperparameters and  $\sigma$  represents the standard deviation. This search space is divided into  $N_{mono}$  bins, ensuring that each bin has the same probability mass. We select the midpoint of each bin as the depth candidate. Consequently, the  $k$ -th depth candidate for pixel  $(u,v)$  is defined as  $t_{u,v,k} = \mu_{u,v} + b_k\sigma, k = 1,2,\dots,N_{mono}$ .  $b_k$  represents the offset corresponding to the  $k$ -th bin. Similar to [3], we calculate  $b_k$  as the following:
$$
b _ {k} = \frac {1}{2} \left[ \Phi^ {- 1} \left(\frac {k - 1}{N _ {\text {m o n o}}} P ^ {*} + \frac {1 - P ^ {*}}{2}\right) + \Phi^ {- 1} \left(\frac {k}{N _ {\text {m o n o}}} P ^ {*} + \frac {1 - P ^ {*}}{2}\right) \right], \tag {5}
$$
where  $P^{*} = \mathrm{erf}\left(\frac{\beta}{\sqrt{2}}\right)$  denotes the probability mass covered by the search space  $[\mu_{u,v} - \beta \sigma, \mu_{u,v} + \beta \sigma]$ ,  $\Phi^{-1}(\cdot)$  is the quantile function of standard normal distribution, erf is the error function. Considering errors may occur in the  $360^{\circ}$  monocular depth, we use a mixture of  $N_{mono}$  monocular depth candidates and  $N_{uni}$  uniform depth candidates sampled from a uniform distribution. By constructing  $D = N_{uni} + N_{mono}$  sphere sweeps, we obtain a new spherical cost volume. The monocular depth guidance enables us to extract more reliable geometry features employed in PanoGRF.
4.1 Metrics and Datasets
PSNR, SSIM [10], LPIPS [38] and WS-PSNR [29] are used as evaluation metrics. We conduct experiments on Matterport3D [4], Replica [28], and Residential [8]. For Matterport3D and Replica, we leverage HabitatAPI [25] to generate the  $256 \times 256$  perspective images (representing the six sides of cube-maps) and stitch them into a  $512 \times 1024$  panorama. We use panoramic sequences with a length of 3. The middle view is rendered based on the first and last views. We conduct comparative experiments on Matterport3D under fixed camera baselines of 1.0, 1.5, and 2.0 meters, where the camera baseline refers to the distance between the camera centers of the first and last views. The baselines used for Residential and Replica are approximately 0.3 and 1.0 meters, respectively.
We set  $N_{mono} = 5$ ,  $N_{uni} = 59$ , and  $N_{l} = 2$ .  $D = N_{mono} + N_{uni}$  is 64.  $\sigma$  used in mono-guided depth sampling is set to 0.5 and  $\beta$  is set to 3. Additional details regarding network architecture can be found in the supplementary material.
4.3 Comparisons
We compared PanoGRF with S-NeRF (spherical variant of NeRF [21]), IBRNet [32], NeuRay [20], and OmniSyn [17]. We trained S-NeRF from scratch, as it is a per-scene optimization method. Other methods are pre-trained on Matterport3D and tested on unseen testing scenes. We fed the original cube maps of panoramas (in Matterport3D and Replica) to the perspective methods (IBRNet and NeuRay). As there are only panoramas in ERP (equirectangular projection) format for Residential,