SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MVS only
Figure 12: Qualitative results of ablation studies for depth estimation on Matterport3D. Mono only and MVS only respectively refer to the results obtained when using only  $360^{\circ}$  monocular depth and  $360^{\circ}$  multi-view stereo.
Cross Attention Renderer
Ground Truth
Figure 13: Qualitative comparisons with Cross Attention Renderer on Replica
Ground Truth
Figure 14: Qualitative comparisons with Cross Attention Renderer on Matterport3D
Table 12: The quantitative results of fine-tuning of the general renderers. The best results are in bold.
baseline	1.0m	0.2m
method	setting	PSNR↑	WS-PSNR↑	SSIM↑	LPIPS↓	PSNR↑	WS-PSNR↑	SSIM↑	LPIPS↓
NeuRay	gen	27.751	27.253	0.8470	0.2565	34.318	33.376	0.9409	0.1158
PanoGRF	gen	28.818	28.487	0.8778	0.1996	35.817	35.056	0.9554	0.0959
NeuRay(+Consist)	ft	26.755	26.250	0.8181	0.2820	33.354	32.359	0.9240	0.1304
PanoGRF	ft	24.341	23.941	0.8175	0.2834	35.828	35.138	0.9578	0.0912
PanoGRF (+Depth)	ft	27.399	27.202	0.8556	0.2446	33.691	33.066	0.9333	0.1342
The reliability of  $360^{\circ}$  monocular depth estimation is not perfect. Therefore, our paper employs uniform sampling to compensate for the remaining depth candidates. In the ablation experiments, we consistently maintain  $N_{mono} + N_{uni} = 64$ , where  $N_{uni}$  represents the sample number of uniform distribution. We discovered that the configuration yields the best results with regard to the metrics of WS-L1, WS-L2, and WS-RMSE.
G Comparisons with Cross Attention Renderer [7]
The Cross Attention Renderer (CAR) is a method that operates on a wide-baseline perspective pair. We divided the two panoramas into cube maps and utilized the corresponding sides of the cube maps as inputs for CAR. Specifically, we rendered the corresponding side of the cube maps at the middle viewpoint. For example, we input the left side of the cube maps and generate the left side of the cube maps at the intermediate viewpoint as the output. We repeated this process for all six corresponding pairs of cube maps. In contrast, our method directly takes two panoramas as input and performs ray casting based on perspective projection. We then render the results for each side of the cube maps at the intermediate viewpoint, preserving the panoramic nature of the input.
We conducted quantitative comparative experiments on Matterport3D and Replica. The qualitative comparisons with CAR on Replica and Matterport3D can be seen in Fig. 13 and Fig. 14. The results clearly demonstrate that PanoGRF significantly outperforms CAR in terms of rendering quality. CAR suffers from limitations associated with its input field-of-view (FoV), particularly in edge regions that are only visible from a single perspective view. CAR relies on a pure stereo-matching method for geometric estimation, which leads to suboptimal rendering performance, as evidenced by the results presented in Table. 9. In contrast, PanoGRF is specifically designed to handle full FoV inputs, and it mitigates the issue of view inconsistency by incorporating the  $360^{\circ}$  monocular depth network.
H Fine-tuning of PanoGRF
We conducted per-scene fine-tuning for NeuRay [20] and PanoGRF on the first test scene of Matterport3D with baselines of 1.0 and 0.2 meters, respectively. The general renderers were fine-tuned for 10k iterations, and the quantitative results are presented in Table.12. Under the baseline of 1.0 meters, the general renderer of PanoGRF, denoted as PanoGRF-gen, achieved the best performance. NeuRay-ft, the fine-tuned renderer of NeuRay, underwent fine-tuning using RGB loss and depth consistency loss, following the methodology described in their original paper [20]. PanoGRF-ft was fine-tuned with only RGB loss. However, the results of NeuRay-ft and PanoGRF-ft were inferior to NeuRay-gen and PanoGRF-gen, respectively. This suggests that fine-tuning under a wide-baseline setting does not improve the performance of general renderers. It is likely that fine-tuning with a wide baseline leads to overfitting. Even with the introduction of a depth uncertainty loss [24] by supervising the renderer depth of PanoGRF with predicted spherical depths during the fine-tuning process, the results of PanoGRF-ft remained inferior to those of the general renderer of PanoGRF. On the other hand, the performance of PanoGRF-ft under the baseline of 0.2 meters was slightly better than that of PanoGRF-gen. However, adding the depth loss [24] was still unable to improve PanoGRF-gen when fine-tuning. Inaccurate predicted depths in certain regions may be the cause, misleading the estimation of NeRF's geometry and thereby reducing the rendering performance.
$^{3}$ Cross Attention Renderer: https://github.com/yilundu/cross_interestrenderer
Additionally, NeuRay-ft was still inferior to NeuRay-gen under the narrow baseline. This may be attributed to the limited field-of-view, which can result in the aggregation of incorrect features. When projecting 3D sample points onto other source perspective views (cube-maps), these points may fall outside the image borders of the source perspective view or be located behind the source perspective camera.
I Quantitative Comparisons with Baseline Methods for Narrow-baseline Panoramas
Table 13: Quantitative comparisons with baseline methods on Matterport3D under the baseline of 0.2 and 0.5 meters. The best results are in bold.
baseline	0.2m	0.5m
method	PSNR↑	WS-PSNR↑	SSIM↑	LPIPS↓	PSNR↑	WS-PSNR↑	SSIM↑	LPIPS↓
S-NeRF	20.79	19.52	0.6967	0.3756	17.95	16.81	0.6278	0.4856
OmniSyn	28.95	28.26	0.9132	0.1804	26.59	26.07	0.8897	0.2005
IBRNet	30.53	29.63	0.9271	0.1363	28.22	27.26	0.8844	0.1987
NeuRay	33.54	32.33	0.9485	0.1074	30.88	29.81	0.9196	0.1536
PanoGRF	34.29	33.27	0.9515	0.0977	31.41	30.46	0.9238	0.1318
We compared PanoGRF with baseline methods under the baseline of 0.2 and 0.5 meters on Matterport3D. As shown in Table. 13, the quantitative results demonstrate that PanoGRF consistently outperforms all the baseline methods under the baseline of 0.2 and 0.5 meters. These findings indicate that our method is applicable to both wide-baseline and narrow-baseline panoramicas. In comparison to generalizable methods designed for perspective views, our method is particularly well-suited for synthesizing panoramic views by leveraging the aggregated features based on spherical projection.
J More Details of PanoGRF
J.1 Renderer
J.1.1 Training
PanoGRF employs the Adam optimizer [16] with an initial learning rate of 4.0e-4. The pre-training process of PanoGRF was conducted on an A100 GPU for 100k iterations, which required approximately two days. The learning rate is halved every 20k iterations, and a batch size of 512 was used during training.
To streamline the experiments, we train PanoGRF and all the baselines under a  $1.0\mathrm{m}$  baseline for comparisons and test them under various baselines. In the ablation studies, the spherical depth estimators (Mono, MVS, and fusion) are pre-trained under a fixed  $1.0\mathrm{m}$  baseline. The general renderers in the ablation studies are pre-trained under corresponding baselines when tested under various baselines. We discovered that if we pre-trained the general renderers under a  $1.0\mathrm{m}$  baseline and tested them under larger baselines, there was no noticeable difference in performance between them.
J.1.2 Architecture
We adopted a coarse-to-fine sampling approach, similar to NeRF [21], and sampled 64 points in both phases. We followed a similar architecture as NeuRay [20] to build our renderers. The coarse and fine renderers share the same image encoder, geometry feature extractor, and visibility encoder. But they have different distribution decoders and aggregation networks  $\mathcal{F}$ , similar to NeuRay. During training, the weights of the  $360^{\circ}$  spherical depth estimator are fixed due to GPU memory limitations. Our image encoder, visibility encoder, distribution decoder, and aggregation network are implemented similarly to NeuRay, except for the padding mode. In the convolution layers of the image encoder and visibility encoder, we employ circular padding horizontally and zero padding vertically instead of direct zero padding. This is done to adapt to equirectangular image inputs and simulate Circular
Figure 15: Architecture of geometry feature extractor.
CNNs [27]. The circular padding helps account for the continuity of pixels at the leftmost and rightmost edges of equirectangular images, which are neighbors, while the top and bottom edges are not. The distribution network consists of 5 sub-networks, each with 3 fully connected layers. The appearance feature extractor is a ResNet [9] which contains 13 residual blocks and outputs the appearance feature map with 32 channels. The architecture details of the geometry feature extractor are shown in Fig. 15. The image encoder in the geometry feature extractor contains 14 residual blocks. The spherical depth input for the geometry feature extractor is first downsampled to  $1/4$  of its original resolution and then fed into the extractor to reduce GPU occupancy.
J.2 Spherical Depth Estimator
J.2.1 Training
We initially train the  $360^{\circ}$  monocular depth network and then freeze its weights when training the remaining components of  $360^{\circ}$  MVSNet. The Adam optimizer is employed with a fixed learning rate of 0.0001. Both the  $360^{\circ}$  monocular depth network and  $360^{\circ}$  MVSNet are trained for 100k iterations, which required approximately one day on a single V100 GPU. The batch size is set to 2.
J.2.2 Architecture
The  $360^{\circ}$  monocular depth network [13] and  $360^{\circ}$  MVSNet utilize ResNet-18 as the feature extractor. The 3D CNN regularization network is composed of 3 downsampling and 3 upsampling blocks, similar to [17]. The depth decoder consists of 2 convolution blocks. The feature map obtained from the middle layer in the  $360^{\circ}$  monocular depth network is concatenated with the regularized spherical cost volume and then decoded into  $360^{\circ}$  depth by the depth decoder. In the multi-view matching process, we compute the similarity by subtracting the feature vectors.
K Datasets
For Matterport3D, we split the training and testing set following SynSin [33]. The first 10 scenes of the test set are used for evaluation. In the case of the Replica dataset, we render a total of 18 scenes for evaluation. Additionally, we utilize the Residential dataset provided in [8], which comprises three scenes. For this dataset, we select the first and last panorama as the input views.
L Training Details of Baseline Methods
We trained NeuRay [20] and IBRNet [32] for 400k and 250k iterations, respectively. Spherical NeRF (S-NeRF) [21] underwent training for 2000 epochs, equivalent to approximately 256k iterations with the batch size of 4096. For OmniSyn [17], the in-painting network was trained for 50k iterations, which took approximately 3 days on a TitanRTX GPU. The depth estimator of OmniSyn was trained for 100k iterations. Due to limitations in GPU memory, OmniSyn was trained at a resolution of  $256 \times 256$ , and its output was resized to  $512 \times 1024$  for evaluation purposes. CAR [7] was trained for 300k iterations.