SC-OmniGS: Self-Calibrating Omnidirectional Gaussian Splatting
Authors: Huajian Huang
Yingshu Chen
Longwei Li
Hui Cheng
Tristan Braud
Yajie Zhao
Sai-Kit Yeung
Venue: ICLR 2025 Poster (2025)
Keywords: Self Calibration
Gaussian Splatting
Radiance Field
Omnidirectional Vision
Bundle Adjustment
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: joint calibrating omnidirectional camera intrinsics and extrinsics to recover fine-grained 3D Gaussians

Abstract: 360-degree cameras streamline data collection for radiance field 3D reconstruction by capturing comprehensive scene data. However, traditional radiance field methods do not address the specific challenges inherent to 360-degree images. We present SC-OmniGS, a novel self-calibrating omnidirectional Gaussian splatting system for fast and accurate omnidirectional radiance field reconstruction using 360-degree images. Rather than converting 360-degree images to cube maps and performing perspective image calibration, we treat 360-degree images as a whole sphere and derive a mathematical framework that enables direct omnidirectional camera pose calibration accompanied by 3D Gaussians optimization. Furthermore, we introduce a differentiable omnidirectional camera model in order to rectify the distortion of real-world data for performance enhancement. Overall, the omnidirectional camera intrinsic model, extrinsic poses, and 3D Gaussians are jointly optimized by minimizing weighted spherical photometric loss. Extensive experiments have demonstrated that our proposed SC-OmniGS is able to recover a high-quality radiance field from noisy camera poses or even no pose prior in challenging scenarios characterized by wide baselines and non-object-centric configurations. The noticeable performance gain in the real-world dataset captured by consumer-grade omnidirectional cameras verifies the effectiveness of our general omnidirectional camera model in reducing the distortion of 360-degree images.

Published as a conference paper at ICLR 2025
SC-OMNIGS: SELF-CALIBRATING OMNIDIRECTIONAL GAUSSIAN SPLATTING
Huajian Huang'+ Yingshu Chen
Longwei Li
Hui Cheng²
Tristan Braud'
Yajie Zhao Sai-Kit Yeung'
The Hong Kong University of Science and Technology
2 Sun Yat-sen University
3 ICT, University of Southern California
ABSTRACT
360-degree cameras streamline data collection for radiance field 3D reconstruc- tion by capturing comprehensive scene data. However, traditional radiance field methods do not address the specific challenges inherent 360-degree images. We present SC-OmniGS, a novel self-calibrating omnidirectional Gaussian splatting system for fast and accurate omnidirectional radiance field reconstruction using 360-degree images. Rather than converting 360-degree images to cube maps and performing perspective image calibration, we treat 360-degree images as whole sphere and derive mathematical framework that enables direct omnidirectional camera pose calibration accompanied by 3D) Gaussians optimization. Further- more, introduce differentiable omnidirectional camera model in order rectify the distortion of real-world data for performance enhancement. Overall, the omnidirectional camera intrinsic model, extrinsic poses, and 3D Gaussians are jointly optimized by minimizing weighted spherical photometric loss. Extensive experiments have demonstrated that our proposed SC-OmniGS is able recover a high-quality radiance field from noisy camera poses or even no pose prior in chal- lenging scenarios characterized by wide baselines and non-object-centric config- urations. The noticeable performance gain in the real-world dataset captured by consumer-grade omnidirectional cameras verifies the effectiveness of our general omnidirectional camera model in reducing the distortion of 360-degree images.
INTRODUCTION
The radiance field techniques pioncered by NeRF (Mildenhall al. 2020) have become an essential paradigm to facilitate scene reconstruction and novel view synthesis. NeRF-based approaches (Bar- ron et al.) 202T] Zhang et al. 2020; [Barron et al.] 2022, Fridovich-
al.) 2022; CCen et al. 2022 Müller et al. (022)) implicitly representing the structure and appearance of captured objects generally necessitate dense set of calibrated images for training. However, NeRF requires compre- hensive data capture to reconstruct scene accurately. 360-degree images can greatly facilitate such data capture. Previous works, such as Huang et al. (2022) and Chen et al. (2023b), have demon- strated the feasibility and efficiency of reconstructing omnidirectional radiance fields in large scenes using sparse and wide-based baseline 360-degree image inputs.
Although 360-degree images have shown potential in reconstructing omnidirectional radiance fields, the quality of the reconstructed models is highly dependent on the accuracy of camera intrinsic and extrinsic parameters. Existing methods for recovering 3D information from 360-degree images, including structure-from- on an idcalized spherical camera model describe the mathematical relationship between 2D 360- degree images and 3D world projection. The 360-degree images are typically obtained by stitching multiple wide angle images, inheriting the distortion from each lens and resulting in complex distortion pattern. The adverse impact of such distortion is neglected in the idealized spherical camera model. Consequently, the inaccurate camera projection modeling leads poor SfM pose estimation, ultimately compromising the quality of 3D radiance field reconstruction when using real-world data. To enhance system performance under camera perturbation and reduce reliance
(SfM) systems (Moulon et al. 2013 Huang & Yeung 2022), rely
"Equal contribution. Corresponding author: huajian @ ust

---

Published as a conference paper at ICLR 2025
Input
SC-OmniGS
Side view
Center view with training camera visualization
Other novel views
w/o calibration
Figure I: SC-OmniGS jointly optimizes the omnidirectional camera model, poses, and 3D Gaus- sians using a differentiable omnidirectional rasterizer. It can achieve rapid radiance field reconstruc- tion with pose prior and render high-fidelity novel views.
on SfM, some approaches (Lin et al. 2021 Jeong et al. 2021 Chen et al. 2023a Park et al. 2023) have explored radiance field self self-calibration, where camera intrinsic and extrinsic parameters jointly optimized with the radiance field representation. However, these solutions focus on traditional images, using well-established camera models for perspective cameras. A naive approach to self-calibrating the omnidirectional radiance field would consist of projecting the 360-degree images onto cube maps with perspective images. However, this approach undermines the integrity of 360-degree images, leading to increasing optimization complexity and instability (Huang al., 2024b). Given the lack of camera models accounting for the distortion of 360-degree images and the limitations of existing self-calibration approaches, there is an urgent need for a framework that calibrates the omnidirectional camera model and poses directly.
In this paper, we propose SC-OmniGS, novel system that self-calibrates the omnidirectional cam- era model and poses along with omnidirectional radiance field reconstruction. We leverage 3D Gaussian splatting (3D-GS) techniques (Kerbl et al., 2023) represent radiance fields by set of 3D Gaussians with explicit positions, covariances, and spherical harmonic coefficients, accelerat- ing the optimization process. In order to realize self-calibrating omnidirectional Gaussian splatting, we adopt a differentiable rasterizer that renders omnidirectional images by splatting 3D Gaussians onto a unit sphere (Li et al. 2024). Crucially, we derive omnidirectional camera pose gradients within the rendering procedure, enabling the optimization of noisy camera poses and even learn- ing from scratch. An example is illustrated in Figure | To rectify distortion patterns in the input image, we propose differentiable omnidirectional camera model comprising learnable 3D spher- ical grid to regress the camera distortion. We thus obtain undistorted omnidirectional images by re-sampling input images based on the learned omnidirectional camera model. We jointly optimize 3D Gaussians, camera poses, and camera models by minimizing photometric loss between rendered and undistorted omnidirectional images. The overview of SC-OmniGS framework is demonstrated in Figure 2 Moreover, considering omnidirectional images in the equirectangular projection have an unbalanced spatial resolution, we introduce weighted spherical photometric loss ensure the spatially equivalent optimization. Furthermore, we apply anisotropy regularizer to constrain 3D) Gaussian scales preventing the generation of filamentous kernels, particularly near the polar areas. conducted extensive experiments using synthetic dataset
To verify the efficacy of SC-OmniGS, OmniBlender (Choi et al. 2023) and real-world 360Roam dataset (Huang et al. 2022). The results showed that our proposed system effectively calibrate the intrinsic model and extrinsic poses of the omnidirectional camera, achieving state-of tf-the-rrr performance on the omnidirectional radiance field reconstruction.
To summarize, the main contributions of this work include:
We proposed the first system for self-calibrating omnidirectional radiance fields, which jointly optimizes 3D Gaussians, omnidirectional camera poses, and camera models.
We provided the derivation of omnidirectional camera pose gradients within the omnidi- rectional Gaussian splatting procedure, enabling the optimization of noisy camera poses
2

---

Published as a conference paper at ICLR 2025
and even learning from scratch. It can also facilitate other applications such GS-based ommidirectional SLAM.
We introduced novel differentiable omnidirectional camera model that effectively tackles the complex distortion pattern contained in omnidirectional cameras.
2 RELATED WORK
Omnidirectional Radiance Field. Neural radiance field (NeRF) (Mildenhall et al. 2020) has emerged as powerful neural scene representation for novel view synthesis. NeRF represents seene as a neural network with radiance and opacity outputs for each 3D point. Although most ex- isting radiance field approaches (Chen et al. 2022: Barron et al.] 2023; Sun et al.) 2022 Xu al.) 2022) can synthesize photorealistic novel views by learning from dense perspective image captures, they tend suffer from inaccurate geometry reconstruction due to the limited field-of-view cover- age and sparse view inputs. To achieve an immersive scene touring with six degrees of freedom (6-DoF), Huang et al. (2022) proposes omnidirectional radiance field learning from sparse 360- degree images with addaetvy-adaptive blocks. while some previous works incorporate 360-degree
3D priors for better geometry feature learning (Chen al.,, 2023b; Kulkarni et al. 2023 Wang al.] 2024]. EgoNeRF (Choi et al.] (202)) employs quasi-uniform angular grids to enhance performance in egocentric scenes captured within a small circular area. The recent 3D Gaussian splatting (3D-GS)
techniques parameterize radiance fields explicit 3D Gaussians, significantly accelerating render- ing and optimization (Kerbl et al. 2023). With the efficient 3D-GS representation. concurrent Om- niGS (Li et al. 2024) optimizes 3D Gaussian splats via sparse panorama inputs while 360-GS (Bai et al. 2024) further exploits indoor layout priors for robust structure reconstruction.
While panoramas offer continuous and wide field of view for omnidirectional optimization, all dis- cussed works focus on radiance field reconstruction merely from known camera parameters, which vulnerable to inaccurate camera modeling.
Self-Calibrating Radiance Field. To simplify the training process of radiance fields and alleviate
the reliance on pre-computed camera parameters, some works optimize camera poses or learn poses from scratch during scene reconstruction (Wang et al. [202]] Jeong et al., |021 Lin et al.) 2021). Wang et al. (2021) shows that camera pose and intrinsic parameters can be jointly optimized during NeRF learning for forward-facing scenes. SC-NeRF (Jeong et al. (021) additionally learns non- linear distortion parameters and introduces camera self-calibration algorithm for generic cameras during NeRF learning. BARF (Lin et al.) 2021) proposes a coarse-to-fine camera registration process from imperfect camera poses for bundle-adjusting NeRFs by gradually activating higher frequency bands of positional encoding. I.2G-NeRF (Chen et al. 2023a) introduces an effective local-to-globy camera registration strategy with an initially flexible pixel-wise aligmment and a frame-wise global alignment. NoPe-NeRF (Bian et al.
2023) employs monocular depth priors for camera estimation with no pose initialization, but it is limited to depth prediction accuracy. For better joint estimation of the scene and camera, CamP (Park et al. (202) introduces the camera preconditioning technique, which applies a preconditioning matrix to camera parameters before passing them to the NeRF model. Recently, SL.AM systems (Huang et al.] 2024a] Yan et al. 2024 Matsuki al. 2024 Keetha et al. 2024) started adopting 3D-GS radiance field for efficient simultancous localization and photorealistie mapping while the camera intrinsic model is calibrated. Fu et al. (2024) relies on monocular depth estimation for jointly optimizing camera poses and 3D Gaussians.
Existing self-calibrating methods are devised to optimize the radiance field from perspective images. SC-OmniGS is the first work dealing with cel--caiibatiion of omnidirectional radiance fields.
Camera Model. A camera model a camera projection function that establishes mathematical relationship between 2D images and 3D observation. Typically. camera models can be classified into two groups, including parametric camera models, e.g. (Kannala & Brandi 2006 Usenko al, 2018) and generic camera models, e.g. (Swaninathan et al.) | |Schops et al.) (020). Parametric camera models assume 3D vision that lens distortion is symmetrical radially and use high-order polynomials to approximate models of real lenses. Conversely, generic camera models exploit mass of parameters to associate each pixel with 3D ray and calibrate distortion. Recent neural lens modeling (Xian et al. 2023) employs invertible neural network (Ardizzone et al.] 2018-2022 to model lens distortion while its optimization is memory conory-consuming. In this paper, we propose a generic camera model tailored for the 360-degree camera.
3

---

Published as a conference paper at ICLR 2025
Inkialiation with coarse points
Differentiable Ormidirectioaal
Differnniable
Ommiddrection Image Input
Poses
Model
&
Backward Flow
Figure 2: A schematic overview of SC-OmniGS optimization flow.
3 PRELIMINARY: 3D GAUSSIAN SPLATTING
3D Gaussian splatting (3D-GS) (eebbl et al. 2023) represents the scene with set of 3D Gaussians, of which 1th Gaussian is parameterized by 3D position P,, covariance E, opacity di and color represented by spherical harmonics (SH) coefficients. The 3D Gaussian reconstruction kernel is formulated as
T3D PP)
G3D -
P,)
(P- -P
(P
(1)
where P E PER Y,Y. denotes the sampling position in the world space. To render image, 3D Gaussians are transformed from the world space to the camera space {x := (x, y, 2) |x E R² by viewing transformation matrix T RR and x RP 3D Gaussians then projected onto the image plane {u == (u.v) T E R²}. The projection function for perspective image is described as
f/x/2+c f/0/2
u oxx)
where far, focal lengths and Cary Cy the principle points of the pinhole camera model. Since this projection process is not affine, the 3D Gaussian reconstruction kernel rap (P) cannot be directly mapped to 2D. To address this problem, Zwicker et al. (002) introduced the local affine approximation of the projection function:
u; + J; (x xxxx)) (RP, +t) +t) J (x-Rxx -)).
(3)
The Jacobian J, is defined by the partial derivatives of projection function & at point x,:
ao 0x
Ou:
Ou:
0u,
J,
(x,)
By
dv.
(4)
8x
By
82
8vs fz Bug 0,
Ou:
fa 2g
0e:
fg
8v
fuye
(5)
0y
02
According to Eq. 1
and 3 the 2D Gaussian reconstruction kernel thus calculated by
(u) Gud u:) cxpf (,,,,,,,"",,RR""")))))"
u,))
The final rendering color C(u) of pixel u in the image can be computed by volumetric rendering:
C(u) Σ ca
II (1-a), aj = 12D(u),
iEN
J=1
M
Q == Σ (dir:), dir, =normoalzze P, (-RR(-)
(8)
where N denotes the set of ordered 3D Gaussians affecting the pixel after splatting onto 2D image, while M the degree of SH coefficients. SH (-) denotes spherical harmonics functions of the normalized viewing orientation dir,.
METHODOLOGY: SC-OMNIGS
SC-OmniGS is self-callbbeee self-calibrating framework for omnidirectional radiance field reconstruction. It takes multiple 360-degree images without pose information or with noisy pose estimations as input recover fine-grained omnidirectional radiance field. We adopt 3D-GS (Kerbl et al. 2023) the
4

---

Published as a conference paper at ICLR 2025
radiance field representation to achieve fast reconstruction and real-time novel view rendering. Sim- ilar to 3D-GS, we initialize the 3D Gaussians from coarse points input obtained from SfM estimation omnidirectional depth map. We then jointly optimize 3D Gaussians, the omnidirectional cam- era model, and poses. The overview of our framework is demonstrated in Figure 2
In this section, first revisit omnidirectional Gaussian splatting and introduce differentiable rasterizer that can render omnidirectional images in the equirectangular projection. In addition, we conduct mathematical analysis of omnidirectional camera pose derivatives within the rasterizer. Furthermore, we propose novel onnidirectional camera model to rectify input training images. Finally, the joint optimization is performed to minimize weighted spherical photometric loss and anisotropy loss.
4.1 OMNIDIRECTIONAL GAUSSIAN SPLATTING
To develop a universal rasterizer, we adopt an idcalized spherical camera model to describe the projection relationship of an omnidirectional camera (Li al. (024). Rather than splatting 3D Gaussians onto an image plane, we project them onto unit sphere and subsequently expand the unit sphere to 2D image in the equirectangular projection. The projection function for an omnidi- rectional image, denoted as Φ", is defined as:
S2 orclan2 (s., Jy arcsin(y/
W H
arctam2 (x.2) ((x,)
arcsin (y/dn/ggd))))) + H
(9)
where arctan2 is the 2-argument arctangent function and V=² -2 H and W de- note image height and width respectively. According to Eq. function d at point x, is JR and
the partial derivatives of projection
f
f2 x
Oo J
+
(x:)
+
(10)
Ox
a.M. &e
& &V
We substitute J, in Eq. Gaussian splitting:
6 and obtain the 2D Gaussian reconstruction kernel for omnidirectional
(u) G2 (u- u;) expf- ( "'RER"""""""""""" u,)
(11)
Eventually, the rendering color (u) of pixel in the omnidirectional image be computed by:
C²(u) = Σ Ga II (1 az), as a - (1).
(12)
IE
4.2 GRADIENTS OF OMNIDIRECTIONAL CAMERA POSE
In addition to backpropagating gradients with respect 3D Gaussians, our differentiable omnidi- rectional rasterizer also propagates gradients with respect to world-to-camera transformation met- rics T [R|t] for camera pose optimization. To ensure numerical stability and avoid singular- ities during optimization, we represent and optimize the transformation matrix in a compact and singularity-frree form, which is a 7-dimensional vector comprising a rotation quaternion and trans- =qqit] [gw f- 4y Q= f. ty fz]. By applying the chain rule to the rendering function #C de
lation: T
in Eq. 12 the gradients of camera pose can be decomposed into two primary branches: de ØT
and
Baf
Orea. Since ac and 0c Bey
Dol have been previously derivated for 3D Gaussian
Day
Drgp
"T""
Bc
Urip
optimization (Kerbl et al. 2023 Li al. 2024). we further elaborate unsolved parts subsequently.
Part 1: ".. the gradient of color w.r.t. pose [qlt]. The view-dependent color of a 3D Gaussian is obtained from spherical harmonics coefficients as depicted in Eq g related to its normalized
dc
viewing orientation. Hence, de is equal to
dc dc
Ødir
dc
Odir
OR
Adir
(13)
TT
Adir
8T
Adir
0R
0q
Øt
5

---

Published as a conference paper at ICLR 2025
Trainable
f
Unit sphere
Focal Length Coefficient
Angle Distortion Coefficients
Ornnidirectionll Camera Model
Output Image
Distortion Pattern
Input Image
Figure 3: Differentiable omnidirectional camera model.
Part 2: Sr2p.
" the gradient of 2D Gaussian w.r.t. pose [q/t]. Camera pose gets involved in the
splatting of Gaussian onto 2D omnidirectional images. According Eq. I1
0rip 8T
8r2p
du;
8r
aJg
0r òR
Ōu:
8T
0J:
8T
ØR.
aTV
(14)
Ou,
0rap
Txx;
ÖR
0x;
+
ou;
0x;
0J
0x;
aR
òR Ōq
4.3 OMNIDIRECTIONAL CAMERA MODEL
Omnidirectional cameras, which typically consist of at least two fisheye lenses, capture 360-degree images through image stitching. However, factory calibration prioritizes seamless stitching over rectifying distortion. As such, stitched omnidirectional images inherently retain distortion from the original camera lenses and deviate from ideal spherical camera models. Unfortunately, there is a lack of well-cstablished camera models capable of accurately representing ommidirectional cam- era distortion, which inevitably compromises 3D reconstruction quality. To address this limitation, propose the first generic omnidirectional camera model that learns complex distorting patterns through differentiable optimization. Our omnidirectional camera model comprises a frozen unit sphere and trainable focal length coefficient ft and angle distortion coefficients, Figure 3 For model initialization, we create a spherical grid S E IR H W x3 and set the correspond- ing angle distortion coefficients 'D with the same dimension then estimated by the Hadamard product of the spherical grid and learned angle distortion coeffi- cients. This approach is more stable than directly learning camera ray distortion. Consequently, the omnidirectional camera model θ is defined as:
illustrated in
zeros. The camera ray distortion is
(15) Our differentiable camera model is decoupled from the rasterization pipeline. ensuring that it does compromise the efficiency of the rendering process. By leveraging the learned camera model parameters O we can extract distortion-fret omnidirectional image 1 from the input image using bicubic grid sampling. Please refer to Algorithm [ for details. The extracted images 7 are then utilized to compute the total loss against the rendered images l".
: f
4.4 JOINT OPTIMIZATION
The optimization in terms 3D Gaussian, camera pose T', and camera model is performed by minimizing the photometric loss, comprising the mean absolute error (MAE) and structural similar- ity index measure (SSIM) loss. However, the equirectangular image projection is not conformal, as the region deformation increases along parallels towards poles. In other words, similar 3D spatial information would occupy more pixels when projected to the top and bottom areas of the 2D image. To ensure spatially equivalent optimization, introduce weighted spherical photometric loss, which is defined as:
C 1",1")= = E(
10), +(1 1(1-SSIN -SSM(((,,
(16)
= wT, w(u) = cos (v - ey + 0.5)/
(17)

---

Published as a conference paper at ICLR 2025
where À is a hyperparameter, I represents of image pixels, and w(+) is the spherical weights (Sun et al. 2017) used to ensure a spherically uniform sample. In addition, we leverage an anisotropy regulanzer to constrain the ratio between the major and minor axis lengths of 3D Gaussians, thereby preventing them from degenerating into filamentous kernels. The anisotropy regularizer is formulated as:
Laniso
max max(s.) min(s;) 7) 7
(18)
iEN
where s; is the scaling of 3D Gaussians (erbl et al. 2023) and is the ratio threshold. Overall, the joint optimization objective is:
C = Lopp + Liaiis--
(19)
EXPERIMENTS
5.1 EXPERIMENT SETUP
Implementation Detail. Our SC-OmniGS implementation is built on Pytorch and CUDA. We uti- lize Adam optimizer to update trainable parameters. The hyperparameters for 3D Gaussians opti- mization are set according to the default settings of 3D-GS (Kerbi et al.] 2023). with X = 0.2 and total of 30,000 optimization iterations. We set the ratio threshold to TO. The omnidirectional camera model is shared across all views on individual scene. Moreover, we set the learning rate of the camera model θ to le-4 and activate the angle distortion coefficients D using the Tanh function. For simplicity, we fix fa to 1. The initial learning rates for each camera quaternion and translation t are 0.01, with exponential decay to 1.6e-4 and 6e-3, respectively, in 100 steps per camera. When calibrating from scratch, we increase the initial learning rate of to 0.1.
Baselines. For comparison, and CamP (Park et al.] (023) as SOTA radiance field calibration baselines trained with training cameras initialized with preset perturbations or from scratch with no pose prior. For reference, we also run 3D-GS (Kerbl et al., 2023) and OmniGS (Li et al. (024)) as non-calibration SOTA baselines. However, apart from OmniGS, other baselines devised for perspective images are not compatible with omnidirectional images as input. To accommodate baselines for fair comparisons, we adopted two practices: 1) We converted each omnidirectional image into a cube map consisting of six perspective images, and then we took the cube maps as input to run the open-source systems with default configurations. 2) Following 360R oam (Huang et al., 2022), we replaced the ray sampling functions of NeRF-based methods (BARF, L2G-NeRF, CamP) with omnidirectional ray sampling to support omnidirectional image training and rendering. Additionally, since point cloud initialization is demanded by 3D-GS based methods, we conducted experiments using different initialization strategies to further verify our system's 's robustness and flexibility.
select BARF (Lin al. 2021). I.2G-NeRF (Chen et al., 2023a
Datasets. We evaluated SG-OmniGS against several SOTA models on datasets of 360-degree im- ages, including eight rorldworld multi-room scenes from 360Roam dataset (Huang et al.) [0022) each with on average 110 training views and 37 test views, and three synthetic single-room scenes from OmniBlender dataset (Choi et al., (003)) each with 25 training views and 25 test views. 360Roam dataset utilizes camera poses estimated by SIM as ground truth and also provides SiM sparse point cloud. OmniBlender dataset provides noise-free camera poses and depth maps.
All methods were run on desktop computer with an RTX 3090 GPU. We use metrics PSNR, SSIM, and LPIPS for evaluating novel view synthesis. Please refer to Appendix C for details on camera perturbations and experimental setup.
5.2 EVALUATION ON SINGLE-ROOM SYNTHETIC DATASET
We conducted experiments on three synthetic scenes from OmniBlender (Choi et al.) 2023), namely Barbershop, Classroom, and Flat. As depicted in Table 1 we configured four settings of radiance field calibration,
Camera poses with perturbation and 3D Gaussians initialized from single rendering depth map.
No camera poses prior but 3D Gaussians initialized from single rendering depth map.

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparisons on synthetic dataset OmniBlender. Checked "Perturb" indicates perturbed training camera poses for training, f indicates training from scratch, 3D-GS based meth- ods are marked with different point cloud initializations: random sampling (random), projection from an estimated mono-depth (est. depth), or from rendered mono-depth (render depth). Meth- ods marked with superscript are modified via omnidirectional sampling. We mark the best two
results in each experiment group with first and second
Ouniiilender
Penutt
Berbershup
Clasarson
Flat
Barbershop SSIMI
Chussroo
Flut
PENR
SSIM
LPIPS)
PSNRT SSIM1 LPIPS
PSNKT
SSIMT
LPEPS) PSNRT
LPIPS)
PSNKT SSIMI
LPEPS)
PSNR
SSIMI
LPIPS)
3D-G5 (render deyth (onder depth
0.922
0.003
0.912
0.101
0.161
OmniUS (reddr depth)
24, 155
01830
0.216
20.17
175
11,699
01354
22,944
0,995
0.446
01:61
0,458
18.788
0.700
0.372
BARF
28.796
0.851
0.242
25.854
0.741
0.309
28.072
0.823
0.252
23.477
0.752
0.250
25.705
0.739
0.309
22.235
0.759
0.294
HARP"
$0.005
01869
0.191
19.204
11208
01261
$1.001
0.80%
0.145
76.799
0.996
0.191
78.865
0.765
0.263
R0417
0.69
0.144
1.2G-NeRE NeRF
29:023 0.858
0.222
25.585
0.729
0.325
27,970
0.825
0.243
28.749
0.856
0.224
18.064
0.597
0.48
18.037
0.713
0.353
12G NeRF'
50.083 0.870
0.188
29.140
0.765
0.267
31,000
0.865
0.145
29,705
0.867
0.189
25.523
0,762
0.268
30.576
01863
0.146
CamP
29:916 01888
0.183
774
0815
01181
29,40
0.905
0.499
16.258
0,699
0:360
Cums
1,905
0.162
0.108
29.90
0.645
0.391
1.658
(nndhe)))
16:255
0.096
33,066
31.256
0.055
0.901
0.064
crnder depth)
37.612
0.97%
0,828
33105
35,240
0.972
31,151
0.85
34.129
0JK6
OmmiSS (rasier depll
18,507 1.689
0.542
17.160
B.622
0.555
18.754
0.747
0.395
15.431
0.675
0.542
17.120
B6il
0.556
18.72%
0.744
0.393
BARF
27.871 0.823
0.296
24.752
0.700
0.360
27.52.
0.814
0.259
15:299
0.631
0.410
16.794
0.564
0.455
20.65
0.735
0.329
HARI
27,595 11307
0.310
25.899
11700
0:300)
28.410
0.200
01211
21
21.518
01905
0.303
75.70
0.205
0.394
28.140
0:251
12G.NNFF
25.300 0.840
0.255
25.623
0.334
27.911
0.679
0.317
19:461
0.714
0.359
CunP
0.834
0.273
25.738
0.605
0.711
0.306
CunF"
0,839
0,341
26.710
0.750
0.433
14,064
0856
0.124
Oun (madiee)))
35. 195
08955
0.075
31.82
0835
0.308
32,814
0.963
0.111
31.
31.423
0.946
0.084
28,99
0,806
0.214
91.673
0885
0.114
Ours (est depth)
25 03.3431000
11952
0,032
32.294
11851
0.106
32.934
0.955
01088
23. 40)
0.940
0057
29.55 305
0.961
08.95
31.276
1897
0194
Oun Graddr depili)
35,601
0.961
0.800
30,815
0.846
0.173
33,064
0.910
0.110
34,365
0.086
0.063
30.212
0.377
0.176
32:424
0.906
0.112
No camera poses prior but 3D Gaussians initialized from a single estimated depth map.
No camera poses prior and random 3D Gaussians initialization.
In the first setting, we perturbed the training camera poses using the same preset noises, indicated by "V" under the "Perturb" column in Table [] OmniGS is the SOTA method in non-calibration omnidirectional radiance field reconstruction. When the input camera poses contain noticeable per- turbance, OmniGS suffers significant performance degradation and struggles to synthesize clear novel views. BARF and L2G-NeRF exhibit acceptable performance with perturbed training cam- eras. After modifying ray sampling functions, we can effectively improve NeRF- based methods' performance, proving the necessity of properly treating omnidirectional images as a whole. How- ever, we cannot apply similar modification to 3D-GS based methods. It is non-trivial to achieve omnidirectional radiance field bundle adjustment, while our SC-OmniGS achieves dominant perfor- mance, on par with OmniGS trained with ground-truth cameras.
Additionally, we initialized all training cameras at the origin, enabling training the models from scratch without pose priors. This is denoted by a "f" under the "Perturb" column in Table 1 In comparison to all baselines, our SC-OmniGS demonstrates stable and excellent performance. To verify SC- Om-iGS flexibility and robustness, we utilized an onnidirectional monocular depth timation method, e.g. EGformer (Yun et al. 2023), to estimate a depth map of the first image for 3D Gaussians initialization without the necessity of a known camera pose. Despite a slight decrease in rendering quality, the results demonstrate that our method still exhibits significant performance improvements compared baseline methods. Finally, rather than using the rendered or estimated geometry as the starting point, we randomly sampled 300k points with random colors and positions as the initial 3D Gaussians to run our method. Our method is able to effectively optimize the scene representation, displaying a low sensitivity to initial values.
Figures 4a and 4b display visual comparisons among calibration methods trained from scratch. Based on the conventional pinhole camera model, inaccurate camera optimization for individual perspective views leads to disconnected faces of a cube map, such as red insets of BARF and I.2G- NeRF In contrast, our omnidirectional camera model assists in optimizing cameras with concern about the holistic field of view, achieving a continuous synthesis.
5.3 EVALUATION ON MULTI-ROOM REAL- WORLD DATASET
In real-world scenarios, we studied three situations of SC-OmniGS and reported the average metric scores across scenes in Table
8

---

Published as a conference paper at ICLR 2025
Ground mrith
1.2G.NoRF
(a) Barbershop'
(b) Classroom'
(c) Canteen
(d) Innovation
Figure Qualitative comparisons of 360-degree novel views among calibration methods. Our results outperform in both rendering quality and camera accuracy. f indicates training from scratch.
Table Quantitative comparisons on real-world dataset 360Roam. "Point Init" indicates the way of point cloud initialization for 3D-GS based methods, checked "Perturb" indicates perturbed camera poses as inputs, "train" and "test" indicate training and test views, respectively. Methods marked with superscript are modified via omnidirectional sampling. We mark the best results in each experiment group with first and second
Inir
PSNKT
SSIM
LPIPS
PSNKT
SSIM?
LPIPS.
3D-GS
Kerbl et
2023
SIM
23.943
0.744
0.223
20.791
0.684
0.261
OmniGS
Creral
SIM
28.517
0.861
0.137
24.212
0.768
0.176
SC-Omni (085)
0.141
25.297
0.180
Omniky Li
2014
0.705
15.619
0.489
BARF Q A121
N/A
21.699
0.594
0.465
20.200
0.572
0,481
BARK
Linel 2111
N/A
22.136
0.575
0.492
20.484
0.546
0.510
L2G-NeKT
2023gg
NIA
21.797
0.598
0.460
20.507
0.576
0.473
L2G-NoRF
CamP (Park drate 20143
N/A
24.592
0.735
0.264
14.253
0.438
0.573
Cam TTark et 2023
N/A
26.
0.786
0.239
13.659
0.437
0.622
SC OC-OOmils
Random
38.562
0.852
0.175
24.343
0.770
0.224
OmniGS
SfM camera poses without perturbation and 3D Gaussians initialized from SfM point clouds.
SfM camera poses with perturbation and 3D Gaussians initialized from SfM point clouds.
SfM camera poses with perturbation and random 3D Gaussians initialization.
Real-world omnidirectional images captured by 360-degree cameras inherit the distortion from each lens and result in complex distortion pattern. However, most methods leverage an ideal spherical camera model to describe omnidirectional projection while overlooking the impact 360-degree camera distortion. With proposed calibration approach, SC-OmniGS can further optimize cam- era parameters in particular the camera intrinsic model, eventually outperforming the non-calibration method OmniGS trained with SfM cameras, demonstrated in the first block of Table 2 Under the situation camera perturbation, SC-OmniGS demonstrates consistent performance across both training and test views, matter how 3D Gaussians are initialized.
As visualized in Figure 4 our SC-OmniGS also dominates qualitative performance in omnidirec- tional scenarios. BARF and L.2G-NeRF tend synthesize low-quality and blurry images, while

---

Published as a conference paper at ICLR 2025
Agg
train
Translation
Rotation noise scale
Translation noise scale
Rotation noise scale
(a) Synthetic seene Barbershop.
(b) Real-world scene Lab.
Figure 5: Performance with different camera perturbations (PSNR1). Zoom in for details.
Table 3:
Ablation study on scene Center of 360Roam, in terms of the optimization of camera pose, camera model, or both. "Perturb" indicates perturbed camera poses, "train" and "test" indicate training and test views, respectively. We mark the best results with first and second
Perturb
Perturh
Calibration
train
Test
train
test
PSNR
SSIM
LPIPS | PSNR 1 SSIM
LPIPS
PSNR 1 SSIM
LPIPS
PSNR
SSIM 1 LPIPS L
none
28.728
24.264
0.763
0.213
22.740
0.717
0.372
15.597
0.553
-aamera model
30.230
0.877
0,153
25.123
0.795
0.195
22.743
0.730
0.408
15,702
0,543
0.568
pose
28.334
0.837
0.191
24.906
0,781
0.224
28.130
0.834
0.198
24.739
0.777
0.233
camera model pose
30,035
0.872
0.169
25.802
0.813
0.203
29.706
0.867
0.177
25.304
0.799
0.220
CamP generates floating fuzzy artifacts, albeit with some fighffrequency details. Please refer to Appendix D for more quantitative and qualitative comparison results.
5.4 ROBUSTNESS AND ANALYSIS OF SC-OMNIGS
Robustness. To further assess the robustness of our method against varying levels of camera pertur- bation, we conducted experiments using the same learning rate with increasing scales of translation and rotation noise applied to the training cameras. In Figure
visualize the performance trend depicting the impact of increasing noise scales on the synthetic scene Barbershop and the real- world scene Lab. In the left charts of Figures 5a and [5b] fixed the default rotation noise scale and varied translation noise scales, while the right charts represent variable rotation noise scale and fixed translation noise scale. Our camera calibration demonstrates greater robustness translation errors with only minor degradation compared to rotation errors. Furthermore, when compared to other calibration baselines (see Barbershop Table D. SC-OmniGS consistently outperforms them with most increased rotation noise scales.
Ablation study. As novel self-calibrating omnidirectional radiance fields method, SC-OmniGS proposed two main components, i.e. generic omnidirectional camera model and camera pose opti- mization. validate the effectiveness our camera calibration, we conducted ablation studies on real scene Center, with and without perturbation to training cameras. The results are presented in Table When the input camera poses are estimated by SiM without perturbation, we can slightly increase the quality of radiance field reconstruction by camera pose refinement, although its perfor- mance gain not higher than adding an omnidirectional camera model. When trained with pose perturbation, our full model, incorporating both camera model and pose optimization, consistently achieves improvement in both training and test view synthesis.
CONCLUSION
This paper introduces SC-OmniGS, the first self-calibrating omnidirectional Gaussian splatting sys- tem that enables swift and accurate reconstruction of omnidirectional radiance fields. With the differentiable omnidirectional camera model and Gaussian splatting procedure, our approach jointly optimizes 3D Gaussians, omnidirectional camera poses and camera model, leading robust camera optimization and enhanced reconstruction quality. Extensive experiments validate the effectiveness of SC-OmniGS in recovering high-quality omnidirectional radiance fields, either with noisy poses or without pose prior. Our work offers an efficient and precise omnidirectional radiance field recon- struction for potential applications in virtual reality, roboties, and autonomous navigation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research project is partially supported by the Innovation and Technology Support Programme of the Innovation and Technology Fund (Ref: ITS/319/22FP).
REFERENCES
Lynton Ardizzone, Till Bungert, Felix Draxler, Ullrich Köthe, Jakob Kruse, Robert Schmier, and Peter Sorrenson. Framework for Easily Invertible Architectures (FrEIA), 2018-2022. URL
https://gitth: http://ttt.hu........lssslllleee htt.h//...mmmm..l..ee//.........
Jiayang Bai, Letian Huang, Jie Guo, Wen Gong, Yuanqi Li, and Yanwen Guo. 360-gs: Layout- guided panoramic gaussian splatting for indoor roaming. arXiv preprint arXiv:2402.0076. 2024.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Burtin-Brualla, and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855-5864, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV, 2023.
Wenjing Bian, Zirui Wang, Kejie Li. Jia- Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4160-4169, 2023.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pp. 333-350. Springer. 2022.
Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, and Fei Wang. Local-to- global registration for adjust-ag neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8264-8273, 2023a.
Zheng Chen, Yan-Pei Cao. Yuan-Chen Guo, Chen Wang. Ying Shan, and Song-Hai Zhang. Panogrf: Generalizable spherical radiance fields for wide-baseline panoramas. Advances in Neural Infor- mation Processing Systems, 36, 2023b.
Changwoon Choi, Sang Min Kim, and Young Min Kim. Balanced spherical grid for egocentric view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16590-16599, 2023.
Sara Fridovich-Kei,, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5501-5510, 2022.
Yang Fu, Sifci Liu, Amey Kulkarni, Jan Kautz, Alexci A Efros, and Xiaolong Wang. Colmap-free 3d gaussian splatting. In Proceedings of the IEEE conference computer vision and pattern recognition, 2024.
Huajian Huang and Sii-Kit Yeung. 360vo: Visual odometry using single 360 camera. In Interna- tional Conference on Robotics and Automation (ICRA). IEEE, 2022.
Huajian Huang, Yingshu Chen, Tianjia Zhang, and Sii-Kit Yeung. 360roam: Real-time indoor roaming using geometry-aware 360 radiance fields. arXiv preprint 22Xiv:2208.02705, 2022.
Huajian Huang, Longwei Li, Cheng Hui, and Sai-Kit Yeung. Photo-slam: Real-time simultaneous localization and photorealistic mapping for monocular, stereo, and rgb-d cameras. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2024a.
11

---

Published as a conference paper at ICLR 2025
Huajian Huang, Changkun Liu, Yipeng Zhu, Hui Cheng, Tristan Braud, and Sii-Kit Yeung. 360loc: A dataset and benchmark for omnidirectional visual localization with cross-devies queries, 2024b.
Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5846-5854, 2021.
Juho Kannala and Sami S Brandt. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses, IEEE transactions on pattern analysis and machine intelligence, 28(8):1335-134) 2006.
Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):1-14, 2023.
Shreyas Kulkarni, Peng Yin, and Sebastian Scherer. 360fusionnerf: Panoramic neural radiance fields with joint guidance. 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7202-7209. IEEE, 2023.
Longwei Li, Huajian Huang, Sai-Kit Yeung, and Hui Cheng. Omnigs: Omnidirectional gaus- sian splatting for fast radiance field reconstruction using omnidirectional images. arXiv preprint arXiv:2404.03200 03202, 2024.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5741-5751, 2021.
Hidenobu Matsuki, Riku Murai, Paul H. J. Kelly, and Andrew J. Davison. Gaussian Splatting SLAM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European conference on computer vision, pp. 405-421. Springer, 2020.
Pierre Moulon, Pascal Monasse, and Renaud Marlet. Global fusion of relative motions for robust, accurate and scalable structure from motion. In Proceedings the on Computer Vision, pp. 3248-3255 2013.
the IEEE International Conference
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Trans. Graph., pp. 102:1-102:15, 2022. doi: 10.1145/5222333333112
Keunhong Park, Philipp Henzler, Ben Mildenhall, Jonathan T Barron, and Ricardo Martin-Brualla Camp: Camera preconditioning for neural radiance fields. ACM Transactions on Graphics (TOG), 12(6))1-11, 2023.
Thomas Schops, Viktor Larsson, Mare Pollefeys, and Torsten Sattler. Why having 10,000 parame- ters in your camera model is better than twelve. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2544-2544, 2020.
Cheng Sun, Min Sun, and Hwann- -xoan Chen. Direct voxel grid optimization: Super-fast con- vergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5459-5469, 2022.
Yule Sun, Ang Lu, and Lu Yu. Wiighte-to--phhcicalllymmmiiiii quality evaluation for omnidirec- tional video. IEEE signal processing letters.
2017. 2017.
Rahul Swaninathan, Michael D Grossberg, and Shree K Nayar. A perspective distortions. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings.. volume 2, pp. II-594. IEEE, 2003.
12

---

Published as a conference paper at ICLR 2025
Vladyslav Usenko, Nikolaus Demmel, and Daniel Cremers. The double sphere camera model. In 2018 International Conference on 3D Vision (3DV), pp. 552-560. IEEE, 2018.
Guangcong Wang, Peng Wang, Zhaoxi Chen, Wenping Wang, Chen Change Loy, and Ziwei Liu. Perf: Panoramic neural radiance field from single panorama. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07066 000000 2021.
Wenqi Xian, Aljaž Božič, Noah Snavely, and Christoph Lassner. Neural lens modeling. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1435-84455 2023.
Qiangeng Xu, Zexiang Xu, Julien Philip. Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and UIrich Neu- mann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5438-5448, 2022.
Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang. Dong Wang, and Xuelong Li. Gs-slam: Dense visual slam with 3d gaussian splatting. CVPR, 2024.
1lwi Yun, Chanyong Shin, Hyunku Lee, Hyuk-Jae Lee, and Chae Eun Rhee. Egformer: Equirect- angular geometry- biased transformer for 360 depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6101-6112, 2023.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields, 2020.
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa splatting. JEEE Transactions Visualization and Computer Graphics, 8(3):223-238, 2002.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A SOCIETAL IMPACTS
This research explored the efficient and robust self-calibrating omnidirectional radiance field for large omnidirectional scenarios, experimenting with real-world data captured with the consumer- grade 360-degree camera and synthetic data. It has broad potential impacts and applications in the real world. For example, supports real-time photorealistic rendering for virtual environments, which enhances virtual immersiveness and enables mixed-reality production. In addition, it can be incorporated into SLAM techniques to upgrade localization robustness.
B LIMITATION.
When confronted with challenging omnidirectional scenes, i.e. multi-room-level scenes with sparse discrete views, training from scratch is a challenging task without the assistance of typical SfM pipeline. We conducted an additional training from scratch experiment using the 360Roam dataset. All self-calibration methods fail to learn radiance fields without any pose priors while our SC- OmniGS is no exception. To address these issues, integrating SC-OmniGS into an omnidirectional SLAM framework is promising direction, which can be future work.
C EXPERIMENT DETAILS
C.I PSEUDO-CODE OF DITTERNNTABLE OMNIDIRECTIONAL CAMERA MODEL
Algorithm iillustrates the backpropagation process and the usage of the proposed generic camera model.
Algorithm 1: Differentiable Omnidirectional Camera Model
Input: input image I
Initialization
H, W, C + image dimension of I;
u <- image pixel coordinates;
S d' d'(u); // project UV back camera space
ft + /:// focal length coefficient.
D initialize zeros dimension (HI W,3);
D + enable gradients; // learnable angle distortion coefficients
* Image Undistortion */
D Tanh((D) /// apply activation function
- S ft S D; // Eq. 15
- oS): undist UV ccordinates
Output undistorted image 1" + grid sarpdesmmleLL,, ú);// bicubic grid sample
D backpropagate and update via total loss L:
C.2 DATASETS
360Roam. 360Roam (Huang et al., 2022) provides 360-degree captured images by consumer- grade 360-degree camera for indoor scenes with multiple rooms, and corresponding initial sparse point clouds from StM. We selected eight scenes with relatively large scales for evaluation, including Bar, Basc, Cafe, Canteen Center, Innovation, Lab, and Library. All data are under CC BY-NC-SA 4.0 license.
OmniBlender: OmniBlender (Choi et al. 2023) contains multi-view 360-degree images rendered from Blender synthetic single indoor scenes under MIT License. It provides ground-truth camera
14

---

Published as a conference paper at ICLR 2025
parameters, and additionally rendered ground-truth depth map of each scene to initialize sparse point cloud for 3D-GS based methods.
The synthetic Blender scene Classroom is under CCO license, Barbershop and Flat under CC-BY 4.0 license. All original models can be downloaded in holtttts:///eie blender. org/ down load/ demo- demo-files/
C.3 PERTURBATION DETAILS
In comparison experiments in Sec. and 5.2 we add translation noise to SfM or ground-truth camera translation, and multiply rotation by rotation noise. Specifically, we set translation perturba- tion noise Troise al scate tru_,, where is random samples from uniform distribution over [-1,1), default Tacale = 0.5, and invur is the inverse of maximum radius of camera positions for scale normalization. We set rotation perturbation noise Raoise 3 Recale: where B is normalized rotation direction with dimensional values randomly sampled from a normal distribution over the angle range ,-1", and default Recale = 0.5. Finally, we get preset perturbed translation I and rotation Rt:
== noise R Rnoise.
In Sec. 5.4 for robustness measurement, we fixed rotation noise scale Recate = 0.5 and changed translation noise scale with T seaae E [0.5, 0.6, 0.7, 0.8, 0.9, 1.1, 1.2, 1.3). and also fixed translation with noise scale sealee = 0.5 and changed rotation noise scale with Rscale E [0.5, 1, 2, 3, 4, 5, 6, 7].
C.4 BASELINES
We trained experimental models by all baselines, i.c., 3D-GS (erbll et al., (023). OmniGS (Li al. 2024). BARF (Lin et al. 2021]. L2G-NeRF (Chen et al., 2023a). CamP (Park et al. 2023), using their official published source codes and default training configurations. The baseline authors hold all the ownership rights on their software.
By default, we convert each 360-degree image Appendix C.2 into cube map with six non- overlapped 480 x 480 perspective images and re-computed six camera parameters. BARF, L2G- NeRF and CamP trained scenes using converted perspective training and test images. In particular, increase training iterations of 3D-GS to six fold, i..., 180,000 iterations for each scene for fair comparison.
In addition, modified the calibration baselines, i.., BARF, L2G-NeRF and CamP, by replacing original perspective ray sampling with omnidirectional ray sampling for training and rendering. These modified baselines, OmniGS and our SC-OmniGS trained scenes using resolution 760 1520 for 360Roam dataset and 1000 x 2000 for OmniBlender dataset.
C.5 RUNTIME
Table reports the quantitative comparisons of training time and inference speed among different methods. On average, for scene with GeForce RTX 3090 GPU, BARF trains for over 2 days, L2G-NeRF and CamP for half a day, 3D-GS (six-fold iterations), OmniGS and our SC-OmniGS within 30 minutes. It is noted that SC-OmniGS does not increase much training time with camera self-calibration compared to OmniGS without camera calibration, meanwhile SC-OmniGS supports real-time rendering.
D MORE EXPERIMENT RESULTS
D.1 ADDITIONAL ABLATION STUDY
Considering the characteristic of the omnidirectional image, we introduce weighted spherical pho- tometric loss Swep as defined in Eq. 16 for spatially equivalent optimization. Furthermore, we observe that noisy camera poses lead to the generation of numerous incorrect 3D Gaussians at the beginning of optimization, making it challenging to filter them out during optimization. To address this, re-initialize the 3D Gaussian with the input coarse points twice, at the 2000 and
15

---

Published as a conference paper at ICLR 2025
Method
Training time
Rendering speed for one panorama (FPS)
BARF
2 days
< 0.05
L2G-NeRF
> 12 hours
0.05
CamP
> 12 hours
< 0.2
3D-GS
30 mins
60
OmniGS
30 mins
> 60
SC-OmniGS
mins
> 60
Table 4: Runtime comparison for methods running one GeForce RTX 3090 GPU.
Table 5: Ablation study, "Re-init" indicates re-initialization of 3D Gaussians; w/o Cusp means we disable the spherical weight and calculate classical photometric loss for optimization; "Perturb" indicates perturbation; i indicates training from scratch without pose priors. We mark the best two results with first and second
Classroom
Perturb
PSNR f
SSIM
LPIPS
w/o Re-init
29.183
0.823
0.193
w/o Dwep
28.225
0.811
0.192
Ours
30.212
0.837
0.176
4 iterations. To further verify the effect of the weighted spherical photometric loss and cal- ibration strategy, we conducted additional experiments on Classroom as an ablation study. The test view results are reported in Table Slnd
and Figure 6
D.2 MORE QUANTITATIVE AND QUALITATIVE COMPARISONS
We report the complete image quantitative evaluation results on 360Roam dataset in Table and the additional camera pose optimization comparisons in individual scenes in Table 7 Under different scenes and different point cloud initializations, SC-OmniGS outperforms other calibration baselines achieving robust camera calibration capability.
Figures supplement some qualitative rendering and depth comparisons among adapted calibra- tion baselines with omnidirectional sampling in the scenes same Figure in the main manuscript. We should intuitively notice that baselines with omnidirectional sampling render continuous 360- degree views, while our SC-OmniGS still gains the best rendering fidelity and most accurately cal- ibrated cameras. Furthermore, Figures 12 12 visualize more comparison results of novel 360-degree and perspective views among calibration baselines.
(a) Ground truth
(b) w/o Lumap
(c) Ours
Figure 6: Ablation study of weighted spherical photometric loss Lusp- Without using Cw the estimated poses of some cameras suffer obvious errors leading to performance degradation in novel view synthesis.
16

---

Published as a conference paper at ICLR 2025
Table 6: The complete image quantitative evaluation results on real-world dataset 360Roam. Checked "Perturb" indicates perturbed camera poses as inputs, "Point Init" indicates the way of point cloud initialization for 3D-GS based methods, "train" and "test" indicate training and test views, respectively, Methods marked with superscript are modified via omnidirectional sampling.
3D-GS
OmniGS
SC-OmniGS
BARF
L2G-NeRF
CamP
On 360Roam
train
test
train
test
train
test
train
test
train
test
train
test
Perturb
X
X SIM
X
Point Init
SfM
SIM
N/A
N/A
N/A
PSNRT
20.983
18.764
24.511
21.567
25.653
22.556
19.047
18.020
19.089
18.333
22.181
13,534
Bar
SSIM
0.734
0.673
0.849
0.760
0.862
0.783
0.543
0.523
0.547
0.533
0.736
0.388
LPIPS
0.235
0.268
0.155
0.191
0.158
0.200
0.528
0.538
0.518
0.527
0.283
0.556
PSNRT
23.677
20.764
28.914
24.254
30.070
25.504
20.409
19.638
20,582
19.991
23.874
13.402
Base
SSIM
0.733
0.681
0.876
0.768
0.897
0.816
0,506
0.499
0.511
0.505
0.674
0.372
LPIPS
0.206
0.233
0.101
0.135
0.098
0.133
0.555
0.562
0.544
0.549
0.319
0.632
PSNR
24.715
19.428
28.846
24.315
29.283
24.838
22.020
19.440
22.198
20.506
25.086
14.251
Cafe
SSIM
0.788
0.712
0.902
0.803
0.905
0.813
0.627
0,590
0.637
0.608
0.780
0.448
LPIPS
0.171
0.214
0.087
0.128
0.108
0.161
0,452
0.474
0,440
0.452
0.229
0.579
P
23.211
19.077
27.318
21.632
27.335
22.159
21.103
18,558
21,116
18.476
24.360
12.861
Canteen
SSIM
0.733
0.631
0.849
0.712
0.838
0.734
0,591
0.546
0.592
0.540
0.761
0.426
LPIPS
0,253
0.330
0.168
0.236
0,204
0,263
0.483
0,507
0.480
0,505
0.225
0.595
PSNRT
24.677
21.801
28.728
24.264
30.035
25,802
21.641
18,870
21,953
19.468
25.098
14.574
Center
SSIM
0.754
0.696
0.848
0.763
0.872
0.813
0.598
0.559
0.609
0.564
0.737
0.486
LPIPS
0,239
0.282
0.170
0,213
0.169
0,203
0,489
0,524
0.475
0,507
0.288
0.607
PSNRT
24.258
22.062
28.980
25,201
30.554
26.390
21.964
21.357
22.021
21.525
24.318
14.389
Innovation
SSIM
0.712
0.677
0.858
0.771
0.898
0.819
0.573
0.568
0.574
0.570
0.687
0.424
LPIPS
0,250
0.269
0.137
0.164
0.120
0.148
0.440
0.445
0.438
0.440
0.308
0.558
PSNRT
24.924
22.003
31.651
27.325
32,890
28.875
23.614
22.889
23.624
22.873
25.840
I3,365
Lab
SSIM
0.824
0.785
0.926
0.869
0.939
0.898
0.725
0.715
0.725
0.716
0.812
0.544
LPIPS
0.145
0.167
0.069
0.093
0.066
0.087
0.351
0.361
0.360
0.371
0.198
0.468
PSNR
25.103
22.427
29.192
25.137
30.137
26.250
23.796
22,830
23.794
22,883
25.779
15,446
Library
S
0.671
0.620
0.782
0.699
0.806
0.746
0.589
0.574
0.589
0.574
0.692
0.417
LPIPS
0.286
0.324
0.209
0.249
0.206
0.243
0.423
0.435
0.423
0.435
0.260
0.585
OmniGS
SC-OmniGS
SC- OmniGS
BARF
L2R-
CamP"
On 360Roam
train
test
train
test
train
test
train
test
train
test
train
test
Perturh
Point Init
SFM
Random
SIM
N/A
N/A
N/A
PSNR
18.915
14.718
24.876
22.090
25.410
22.556
19.457
18.499
19,803
18.794
22.946
12.600
Bar
SSIM
0.640
0.431
0.840
0.763
0.854
0.785
0.519
0.498
0.533
0.510
0.765
0.380
LPIPS
0.404
0.504
0.192
0.235
0.166
0.205
0,567
0,580
0,542
0.557
0,273
0.636
PSNR
21.449
14.559
28.322
24.780
29.226
24.308
20.986
20.024
21.382
20.122
25.179
13.25T
Base
SSIM
0.674
0.351
0.842
0.777
0.880
0.777
0.488
0.472
0.501
0.481
0.728
0.381
LPIPS
0.328
0.498
0.172
0.198
0.114
0.157
0.590
0.601
0.557
0,572
0.282
0.653
PSNR1
22.313
15.680
28.156
23.917
29.278
25.171
22.169
19.895
22.518
20.146
26.908
13.689
Cafe
SSIM
0.734
0.441
0.894
0.789
0.904
0.827
0.607
0.563
0.617
0.571
0.829
0.429
LPIPS
0,294
0.462
0.123
0.178
0.108
0.145
0.478
0.497
0.454
0.479
0.196
0.620
PSNR1
22.814
14.273
27.494
21.251
27.259
22.139
21.395
18.887
21.761
17.027
26.388
12.69T
Canteen
SSIM
0.732
0.458
0.844
0.692
0.837
0.732
0.564
0.521
0.575
0.476
0.817
0.445
LPIPS
0.331
0,536
0.198
0.289
0.206
0.265
0.526
0.558
0.503
0,571
0.196
0.627
PSNR1
22.740
15.597
28.972
24,482
39.706
25.304
22.275
19.689
22.859
16.855
26.616
14.47I
Center
SSIM
0.717
0.510
0.847
0.779
0.867
0.799
0.584
0.524
0.604
0.478
0,780
0.487
LPIPS
0,372
0.553
0.205
0.265
0.1
0.220
0.505
0.540
0,474
0.559
0.264
0.608
PSNRT
21,880
16.047
28.916
25.943
30.079
24.788
22,291
21.242
22,761
21.450
25,890
13.36T
Innovation
SSIM
0.697
0.440
0.828
0.785
0.887
0.762
0.545
0.535
0.558
0.545
0.738
0.421
L.IPS.
0.325
0.447
0.199
0.219
0.1
0.7
0.475
0.482
0.449
0.460
0.287
0.616
PSNR1
22.049
16.642
32.175
27.568
32.801
28.812
23.997
22.838
24.622
22.951
27.002
14.315
Lab
SSIM
0.762
0.563
0.930
0.874
0.938
0.895
0.709
0.692
0.729
0.707
0.837
0.530
LPIPS
0.299
0.421
0.089
0.122
0.068
0.090
0.361
0.372
0.314
0.330
0,209
0.594
PSNR
24.725
17.437
59,588
24.710
30.095
26.202
24.514
22.796
24,944
22,838
28.141
14.891
Library
SSIM
0.684
0.445
0.791
0.703
0.80S
0.743
0.584
0.558
0.600
0.568
0.798
0.422
LPIPS
0,323
0.495
0.225
0.289
0.207
0.244
0.430
0.451
0.402
0.430
0.205
0.623
17

---

Published as a conference paper at ICLR 2025
Table 7: The training camera pose quantitative evaluation among calibration methods. Checked "Perturb" indicates perturbed camera poses inputs, f indicates training from scratch, "Point Init" indicates the way of point cloud initialization for 3D-GS based methods, "p" and "R" indicate Root Mean Squared Error (RMSE) of camera position (in world units) and rotation (in degrees), respectively. Methods marked with superscript are modified via omnidirectional sampling. SC- OmniGS performs robust camera calibration capability in different scenarios and point initialization.
On 360Roam
BARF
BARF
L2G-NeRF NeRF
CamP
CamPo
SC-OmniGS SC-OmniGS
Perturb
V
v
Point Init
N/A
N/A
N/A
N/A
N/A
N/A
random
SIM
p 4
0.31873
0.11240
0.23656
0.05947
0.16559
0. 16692
0.03811
0.03401
Bar
R
0.12260
0.03499
0.08151
0.06093
0.02700
0.02568
0.01880
0.01402
p
1
0,38139
0.03944
0.22836
0.18336
0.19603
0.19792
0.08044
0.02074
Base
R
0.10911
0.00561
0.05018
0.02207
0.02758
0.02575
0.02459
0.00318
p
L
0,34125 0.32115
0. 14891
0.18808
0.14154
0.14064
0,00651
0.00627
Cafe
R
0.12002 0.08143
0.03887
0.07296
0.02560
0.02694
0.00236
0.00212
0.47954 0.24846
0.55104
0.58446
0.16421
0.16661
0.04292
0.03002
Canteen
R
0.18377 0.09021
0,23060
0,18187
0.02624
0.02444
0.00592
0.00253
p 4
0.72546 0.53148
0.72888
0.81537
0.19709
0.19951
0.17692
0.10194
Center
R
0.26783 19900
0.22620
0.38847
0.02768
0.02532
0.06964
0.00746
p 4
0.23938 0.20665
0.11435
0.30508
0.20174
0.20299
0.00565
0.02205
Innovation
R
0.08755 0.06569
0.03044
0.06525
0.02823
0.025232
0.00190
0.00598
p 1
0.07353
0.02230
0.03886
0.01235
0,23800
0.23774
0.01353
0.01432
Lab
R 4
0.02864
0.00385
0.01433
0.00301
0.03342
0.02524
0.00248
0.00191
1
0.27276
0.02723
0.26827
0.02759
0,21650
0.21446
0.11948
0.00632
Library
R
0.07719 0.00248
0.07728
0.00283
0.02787
0.02771
0.01251
0.00162
On OmmiBlender
BARF
BARF L2G-NeRF L2G-NeRF
CamP
CamP"
SC-OmniGS SC-Omni(e SC-OmniGS
Perturb Point Init
N/A
N/A
N/A
N/A
N/A
N/A
Random
Est. depth
Render depth
0.14411
0.00053
0.00560
0.00048
0.18435
0.17873
0.11106
0,00032
0.00025
Barbershop
R L
0.09418
0.00040
0.00529
0.00047
0.08132
0.07486
0.04919
0.00034
0.00024
P 1
0.00882 0.00059
0.36072
0,00062
0.21609
0.21072
0.00015
0.00023
0.00014
Classroom
R
0.00995 0.00094
0.28451
0.00095
0.18112
0.16902
0.00028
0.00040
0.00021
1
0,21386
0.00053
0.40058
0.00048
0,25824
0,25266
0.00051
0.00108
0.00032
Flat
R
0.15046 0.00109
0.19573
0.00133 13
0.07878 0.06339
0.00077
0.00351
0.00035
On OmniBlender
BARF
BARF
L2G-NeRF LeR-NeRF
CamP
CamP" f
SC-OmmiGS SC-OmniGS SC-OmniGS
Perturb Point Init
f
N/A
N/A
N/A
N/A
N/A
N/A
Random
Est. depth
Render depth
|
0.34757
0.00065
0.37682
0.00050
0.41992
0.11743
0.001 26
0.00061
0.00033
Barbershop
R L
0,30309
0.00058
0.24394
0.00047
0.07589
0.25327
0.00202
0.00065
0.00059
p 1
0.45917
0.00041
0.41830
0.00055
0.49153
0.33876
0.00071
0.00064
0.00018
Classroom
R
0.34051
0.00061
0.30008
0.00096
0,25800 0.51458
0.00093
0.00111
0.00018
0.31282
0.00050
0,39268
0.00034
0,27143
0.00096
0.00308
0.00093
0,00060
0,21171 R
0.00045
0.23691
0.00044
0.15632 0.01593
0.00883
0.00171
0.00088
18

---

Published as a conference paper at ICLR 2025
Ground
BARF
L.G--NORE
Camp
Barbershop'
(b) Classroom'
(c) Canteen
(d) Innovation
Figure 7: Qualitative comparisons of 360-degree novel views among calibration methods equipped with omnidirectional sampling. Our results outperform in both rendering quality and camera accu- racy. i indicates training from scratch, indicates baselines modified via omnidirectional sampling.
BARF
L-GG-NRR-
Campo
Ours
Barbershop'
(b) Classroom
(c) Canteen
(d) Innovation
Figure 8: Depth visualization of 360-degree views rendered by calibration methods equipped with omnidirectional sampling. Our results outperform in geometry accuracy and details. t indicates training from scratch, indicates baselines modified via omnidirectional sampling.
19

---

Published as a conference paper at ICLR 2025
Ground-truit -trut
BARE
L2G-NeRE
Canal
Cars
Figure 9: Novel views on synthetic scene Flat among baselines trained from scratch.
Ground-rruth
BARE
L2G-NeRF
Cami
Ours
Figure 10: Novel views on real scene Cafe among baselines trained with camera perturbation.
20

---

Published as a conference paper at ICLR 2025
Ground-truut soudd-truth
BARE
L2G-NeRE
Camal
Chrs
Figure 11: Novel views on real scene Bar among baselines trained with camera perturbation.
Ground-rruth
BARE
L2G-NORF
Cani
Ours
Figure 12: Novel views on real scene Base among baselines trained with camera perturbation.
21


Reviews:
[{"review_id": "jK5TwuXHWl", "reviewer": "Reviewer_CA85", "summary": "This paper proposes an extension of 3D Gaussian splatting (GS) for omnidirectional images and enabling self-calibration. GS for omnidirectional images is previously studied (OmniGS), but it assumes the pre-computed camera positions. The proposed method refines the camera poses during gradient-based optimization. Experiments show that the proposed method improves the vanilla OmniGS. A main technical contribution of the paper is to derive the backward gradient for pose refinement of spherical images.", "questions": "I would appreciate it if the authors emphasized the technical (or scientific) novelty of the proposed method again.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "### Self calibration + omnidirectional images\nThe proposed method would be the first attempt to combine the self-calibration and omnidirectional GS.\n\n### Backward gradient for spherical images\nFor pose refinement of spherical images, the paper derives the gradients in Eqs. (13--14). This would be a technical novelty in this paper.", "weaknesses": "### Limited technical improvement\nSelf-calibration of GS has been well-studied so far. Also, omnidirectional GS is existing. The proposed system would be practical, but the scientific motivation for combining those two is not quite large, i.e., the technical novelty is limited.\n\n### Gradient derivation\nThe key technical part of the paper, the derivation of gradients on camera poses of spherical images (Eqs. (13--14)), is rather straightforward. This would be naturally extended from perspective cases. \n\nWhile the paper describes \"converting 360-degree images to cube maps...\" this is just about the camera models.\nAlthough in different contexts, for example, Metashape and OpenMVS support spherical camera models for the SfM problem, which involves bundle adjustment (i.e., non-linear optimization using first-order derivative), so they should compute gradients in somewhat similar ways to Eqs. (13--14).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RiI2BkvLDr", "reviewer": "Reviewer_83HV", "summary": "This paper proposes a system for self-calibrating omnidirectional radiance fields, aiming to optimize 3D Gaussians, omnidirectional camera poses, and camera models in tandem. While the authors describe this as the first system of its kind, the contribution can largely be seen as an engineering effort to integrate multiple optimized parameters within a single framework. The primary novelty in the work appears to lie in the introduction of a differentiable omnidirectional camera model that enables ray-wise distortion handling and in the derivation of gradients for pose optimization.", "questions": "- The paper reports PSNR results to demonstrate performance, but since it also optimizes camera poses, it would be beneficial to include a comparison of the optimized extrinsic parameters against ground truth values.\n\n- Given the emphasis on self-calibration, could the paper also show improvements in the intrinsic parameters after optimization?\n\n- Are the optimizations of focal length and distortion parameters shared across all views, or are they optimized per view? Clarifying this would be help.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-structured and easy to follow, making the methodology and findings accessible to readers.\n- The introduction of a differentiable omnidirectional camera model that enables ray-wise distortion and the derivation of gradients for pose optimization are valuable innovations that expand the applicability of the system.\n- The use of spherical weights in the photometric loss ensures spatially balanced optimization, which enhances the robustness and accuracy of the optimization process.\n- The experiments and results demonstrate superior performance compared to previous methods, highlighting the effectiveness of the proposed approach.", "weaknesses": "Misleading Terminology in Title: While the title suggests \"self-calibrating omnidirectional Gaussian splatting,\" the approach relies on initialization from a structure-from-motion (SfM) pipeline rather than directly calibrating intrinsic parameters from the images alone. This approach is more accurately an optimization process rather than an auto-calibration technique in the classical sense (e.g., auto-calibration from absolute dual quadrics in multiple-view geometry).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtaPCkejte", "reviewer": "Reviewer_dnTE", "summary": "The paper proposes a self-calibrating Gaussian splatting method for reconstructing omnidirectional radiance fields from 360-degree images without poses or with noisy poses. In this framework, scene representation, camera poses, and camera models are jointly optimized by minimizing a weighted spherical photometric loss. Additionally, a differentiable omnidirectional camera model is introduced to learn camera distortion. Experimental results show that the proposed method effectively recovers high-quality radiance fields from 360-degree image inputs.", "questions": "(1) It would be better to provide the experimental results mentioned in the weaknesses.\n\n(2) In Table 1, the paper does not provide results with random initialization and estimated depth in the comparison of perturbed poses.\n\n(3) The paper states that it can address scenes with wide baselines. However, the method cannot be trained from scratch on real-world multi-room scenes, even though real-world datasets have more views than synthetic datasets. I recommend including a detailed analysis and results of the failure cases to better understand the reasons behind this limitation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(1) The paper proposes a novel self-calibrating method that extends the omnidirectional Gaussian splatting to handle unposed or noisy 360-degree images.\n\n(2) The paper introduces a differentiable omnidirectional camera model, which uses trainable focal length and angle distortion coefficients to represent camera distortion. \n\n(3) The proposed method achieves state-of-the-art performance in novel view synthesis.", "weaknesses": "(1) The method estimates camera poses but lacks comparisons of pose accuracy. I think that comparing only rendering quality, especially with NeRF-based calibration methods, is insufficient to determine whether the superior performance of this paper is due to pose optimization, the camera model, or the scene representation using 3D GS. Therefore, I suggest adding quantitative and qualitative comparisons of camera poses on two datasets. \n\n(2) The experiment only compared with NeRF-based calibration methods and lacks comparisons with 3D GS-based baselines, such as COLMAP-free 3D GS. Besides, these NeRF-based calibration methods are originally designed to address noisy poses, rather than unposed images. I think it would be fairer to compare with poes-prior-free methods, such as NoPE-NeRF or LocalRF.\n\nReferences:\n\nA1. Fu, Y., Liu, S., Kulkarni, A., et al. COLMAP-Free 3D Gaussian Splatting, CVPR, 2024.\n\nA2. Bian, W., Wang, Z., Li, K. et al. Nope-NeRF: Optimising neural radiance field with no pose prior, CVPR, 2023.\n\nA3. Meuleman A, Liu Y L, Gao C, et al. Progressively optimized local radiance fields for robust view synthesis, CVPR, 2023.\n\n(3) The paper does not include an ablation study of the anisotropy regularizer loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FuVopjuJxa", "reviewer": "Reviewer_wcnp", "summary": "The authors introduce the first system capable of self-calibrating omnidirectional radiance fields, by simultaneously optimizing 3D Gaussians, omnidirectional camera poses, and camera models. Unlike previous works that project  360-degree images onto cube maps, this study preserves the integrity of 360-degree images by directly modeling the 360-degree camera. Additionally, this approach does not rely on precise camera calibration, allowing it to flexibly adapt to various downstream tasks, such as omnidirectional SLAM, by optimizing both intrinsic and extrinsic 360-degree camera parameters. This method achieves the best results among omnidirectional and self-calibration approaches based on NeRF and Gaussians.", "questions": "I have some confusion regarding the input. I notice that in datasets like the 360Roam dataset, there are 110 training views and 37 test views. Are these views entirely omnidirectional data, or are there some additional perspective camera images used as auxiliary data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the paper is sound. Omnidirectional images have varying information densities across different pixels. Using cube maps-based projection combined with traditional perspective analysis is bound to introduce distortion (or aliasing). By directly modeling the 360-degree camera projection, the method leverages the continuity of 3D space, which can mitigate distortion and provide higher fidelity.\n2. I like the chain rule-based derivation of the pose gradient in Equations 13 and 14, as it makes the camera optimization more reasonable.\n3. The experiments are very thorough. The authors compare two datasets and cutting-edge methods (OmniGS), while also discussing the experimental results under different camera and point cloud initializations (Tab. 1), as well as the results after adding various perturbations (Fig. 5). This supports it to be a well-rounded work.", "weaknesses": "1. Some figure captions are too brief (Fig.2, Fig.3), requiring readers to refer back to the main text for clarification on several unclear points, which disrupts the reading flow.\n2. In the visualizations, it seems that only the rendered results are compared, lacking some geometric information, such as depth visualizations and point cloud reconstructions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)