ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian Splattings
Authors: Suyoung Lee
Jaeyoung Chung
Jaeyoo Huh
Kyoung Mu Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D scene reconstruction
3D Gaussian splatting
Omnidirectional images
Primary area: machine_vision
TL;DR: We develop a new method for 3D scene reconstruction from omnidirectional images via 3D Gaussian Splattings.

Abstract: Omnidirectional (or 360-degree) images are increasingly being used for 3D applications since they allow the rendering of an entire scene with a single image. Existing works based on neural radiance fields demonstrate successful 3D reconstruction quality on egocentric videos, yet they suffer from long training and rendering times. Recently, 3D Gaussian splatting has gained attention for its fast optimization and real-time rendering. However, directly using a perspective rasterizer to omnidirectional images results in severe distortion due to the different optical properties between the two image domains. In this work, we present ODGS, a novel rasterization pipeline for omnidirectional images with geometric interpretation. For each Gaussian, we define a tangent plane that touches the unit sphere and is perpendicular to the ray headed toward the Gaussian center. We then leverage a perspective camera rasterizer to project the Gaussian onto the corresponding tangent plane. The projected Gaussians are transformed and combined into the omnidirectional image, finalizing the omnidirectional rasterization process. This interpretation reveals the implicit assumptions within the proposed pipeline, which we verify through mathematical proofs. The entire rasterization process is parallelized using CUDA, achieving optimization and rendering speeds 100 times faster than NeRF-based methods. Our comprehensive experiments highlight the superiority of ODGS by delivering the best reconstruction and perceptual quality across various datasets. Additionally, results on roaming datasets demonstrate that ODGS effectively restores fine details, even when reconstructing large 3D scenes. The source code is available on our project page (https://github.com/esw0116/ODGS).

ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian Splatting
Suyoung Lee
Kyoung Mu Lee 1,2 Dept. of ECE & ASRI, PIPAI, Seoul National University, Seoul, Korea
Jaeyoung Chung"!
Jaeyoo Huh
(esw0116, robot0321)@smu ac. ac.kr
jaeyoo9008gaii1..
com com
kyoungmudsnu. ac kr
Abstract
Omnidirectional (or 360-degree) images are increasingly being used for 3D applica- tions since they allow the rendering of an entire scene with single image. Existing works based on neural radiance fields demonstrate successful 3D reconstruction quality egocentric videos, yet they suffer from long training and rendering times. Recently, 3D Gaussian splatting has gained attention for its fast optimiza- tion and real-time rendering. However, directly using a perspective rasterizer omnidirectional images results in severe distortion due to the different optical prop- erties between the two image domains. In this work, present ODGS, novel rasterization pipeline for omnidirectional images with geometric interpretation. For each Gaussian, we define a tangent plane that touches the unit sphere and is perpendicular to the ray headed toward the Gaussian center. We then leverage perspective camera rasterizer project the Gaussian onto the corresponding tangent plane. The projected Gaussians are transformed and combined into the omnidirectional image, finalizing the omnidirectional rasterization process. This interpretation reveals the implicit assumptions within the proposed pipeline, which we verify through mathematical proofs. The entire rasterization process is par- allelized using CUDA, achieving optimization and rendering speeds 100 times faster than NeRF-based methods. Our comprehensive experiments highlight the superiority of ODGS by delivering the best reconstruction and perceptual quality across various datasets. Additionally, results on roaming datasets demonstrate that ODGS effectively restores fine details, even when reconstructing large 3D scenes. The source code is available on our project page
Introduction
With the development of VR/MR devices and robotics technologies and the increasing demands of such applications, 3D scene reconstruction has become one of the crucial tasks in computer vision. Traditional works have employed structure-from-mmots and scene geometry from multiview 2D images by finding the correspondences between images. As target 3D scenes become broader and more complex, accurate reconstruction demands larger volume of images and increases the computational burden required for identifying correspondences. Recently, some approaches have tried to alleviate these challenges by utilizing wide-angle cameras to capture wide field-of-view images. Omnidirectional images, which provide a 360-degree field of view, gaining increased interest because they encompass whole scenes within single image, thereby reducing the cost of inter-ine imee-image feature matching. The growing popularity 360-degree cameras for personal video recording and the concurrent release of related datasets further facilitate the research on 3D content reconstruction from omnidirectional images.
algorithm that estimates camera motion
indicates equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Such 3D reconstruction techniques 37 51 began to be mainly studied in SLAM systems to obtain accurate camera poses with matched 3D points from monocular omnidirectional video obtained from
robots. However, these models focus on restoring structural information rather than contents, and they
often bypass the fine details and texture of 3D scenes. After neural radiance field (NeRF) [36] has
shown outstanding 3D reconstruction performance, several works such as [T0 19 21 30 attempted
to reconstruct 3D implicit representation from omnidirectional images. Despite showing prominent reconstruction quality, those methods commonly suffer from slow rendering and lengthy training. 3D Gaussian splatting [27] 3DGS in short, overcomes the challenges of NeRF by representing 3D contents with numerous Gaussian splats. The 3D Gaussians are initialized from the sparse point cloud obtained from the structure prructurefrrom-ooo motion and optimized through the differentiable image rasterization pipeline. Since the CUDA- iiDD---mlementedd rasterization for 3DGS is much faster than volume rendering used in the NeRF family, 3DGS has dramatically improved rendering speed while maintaining or improving performance. Although many follow-up works have been proposed after 3DGS's success, only a few address 3DGS in the omnidirectional image domain.
In this work, propose ODGS that aims to reconstruct high-quality 3D scenes represented by Gaussian splatting from multiple omnidirectional images. The gist of our method is designing CUDA rasterizer that is appropriate for omnidirectional images. Specifically, we create unit sphere from the camera origin, considering it an omnidirectional camera surface, and assume that each Gaussian is projected onto the tangent plane of the point where the vector from the camera origin to the center of the Gaussian and the unit sphere meets. Since each Gaussian is projected onto a different plane, calculate a rotation matrix for the coordinate transformation ensure that each Gaussian is properly projected onto its corresponding tangent plane. Then, the projected Gaussians subsequently mapped onto the omnidirectional image plane. Our proposed rasterizer is also easily parallelizable like the original 3DGS rasterizer, demonstrating fast optimization and rendering speed. Finally, we carefully apply the densification rule split or prune the Gaussians for omnidirectional projection. We apply a dynamic gradient threshold value for each Gaussian based on its elevation, as the azimuthal width of the projected Gaussian is stretched when transformed into an equirectangular space. We conduct comprehensive experiments comparing the reconstruction quality in various 360-degree video datasets with various environments, including egocentric and roaming, real and synthetic. The results show that ODGS achieves much faster optimization speed than existing NeRF-based methods and reconstructs the scenes with higher accuracy. Additionally, the perceptual metrics and qualitative results demonstrate that our method restores textural details more sharply.
To summarize, our contributions are three-fold:
We introduce ODGS, a 3D reconstruction framework for omnidirectional images based on 3D Gaussian splatting, achieving 100 times faster optimization and rendering speed than NeRF-based methods.
We present a detailed geometric interpretation of the rasterization for omnidirectional images, along with mathematical verification, and propose CUDA rasterizer based on the interpretation.
We comprehensively validate ODGS on various egocentric and roaming datasets, showing both more accurate reconstructed results and better perceptual quality.
2 Related works
In computer vision, ongoing research has been on creating 3D representations of the surrounding environment using multi-view images. Among them, omnidirectional images capture the surrounding space in single image due to their wide field of view, making them increasingly popular for 3D reconstruction and mapping. Traditional structre-frommmmotion (SIM) algorithms [38] 42 43
simultaneously estimate camera poses and 3D geometry structure by extracting and matching feature points across multiple images. This field has developed over many years, resulting in the release of user-frient frienfie open libraries such as COLMAP 43 or OpenMVG [37]. Recent advancements continue to improve feature matching for spherical images [16]. In indoor environments, additional information such as room layout 2 40 41 and planar surfaces [14] 45 are used to promote the reconstruction quality. The geometry structures estimated from omnidirectional images are also utilized for localization
22 28 or for simultancous localization and mapping (SLAM) research [5]

---

44
The wide field of view provided by omnidirectional cameras enables the simultaneous capture of extensive spatial information. making them highly beneficial in robotic applications for environmental perception and understanding. Beyond sparse geometry structure in SfM, Multi-Vies Stereo (MVS) [5] supports dense reconstruction based on epipolar geometry to achieve better results. Recently, multi-view stereo techniques leveraging deep neural networks have been actively researched. 9
49
Another approach to representing 3D is by stacking multiple layers of multi- sphere images. Inspired by multi-planar images, this method facilitates the egocentric representation of scenes I& These methods show the possibilities of 3D reconstructions using onnidirectional images, but often lack textural details for photo- reattsre 3D reconstruction or limit the representation to confined spaces.
32 35
In recent 3D reconstruction research, Neural Radiance Field (NeRF) 36 has demonstrated the capability for photo-realistic novel-view synthesis, leading to studies on NeRF-based 360 image 3I) reconstruction. This approach is widely used for directly reconstructing scenes in 3D
10 17 21 33
or indirectly representing 3D by estimating depth 6 [ 30 In particular, EgoNeRF 10 a recently
published NeRF-based reconstruction method, pointing out that typical Cartesian coordinate is not appropriate for representing a large scene with omnidirectional images. It introduces a new spherically balanced feature grid and hierarchical density adaptation during ray casting, achieving prominent reconstruction quality. However, although NeRF- methods have shown more realistic reconstruction than traditional techniques, they have the inherent limitation of requiring extensive time for reconstruction and rendering.
3D Gaussian splatting (3DGS) (27 is novel 31) representation that demonstrates photo-realistic novel view synthesis while supporting fast optimization and real-time rendering. 3DGS explicitly expresses a space using a set of Gaussian primitives and quickly creates novel views through rasterization pipeline without the time-consuming ray-casting process in NeRF. Due to its high applicability, extensive research is rapidly advancing, covering not only typical reconstruction but also sparse reconstruction 12 (5)) dynamic scene reconstruction 24 50 52), SLAM
26
34 and
even generation [I]] 46 54] However, 3D scene reconstruction based on omnidirectional images has been barely studied. This is partly because developing a suitable rasterizer for omnidirectional images that allows real-time rendering is challenging, and such implementations are not publicly available. 360-GS [2] is the first method that proposes omnidirectional reconstruction with 3DGS, employing two-step strategy. However, it relies on layout-guided error correction, which limits its applicability to indoor scenes. In this paper, we present a carefully implemented CUDA rasterizer that rotates the projection plane unit sphere, which can efficiently optimize 3D Gaussians without any constraints or assumptions on scenes, Further, we propose a dynamic densification rule designed for a 360-degree camera from our analysis, enabling us to reconstruct high-quality scenes rapidly. We note that few concurrent works, such as Gaussian splatting with optimal projection strategy 23 or OmniGS 311, partly share the contributions with ours.
3 Methods
3.1 Preliminary: Rasterization Process in Typical 3D Gaussian Splatting
3D Gaussian splatting (3DGS) (27 is recently proposed 3D representation that models scenes using set of 3D anisotropic Gaussians derived from multi-view images. It initializes the 3D Gaussians using a traditional strucuure-frommmotion library and optimizes their properties-such as position, color, scale, rotation, and opacity-through photometric loss. In this section, we explain the rasterization pipeline for 3D Gaussians in a perspective camera as proposed by 3DGS, followed by a discussion of the differences in the rasterization process for an omnidirectional camera.
A 3D Gaussian is represented by its mean and covariance, where the covariance matrix Σ is expressed as the product of a rotation matrix R and a scale matrix S ( = RSS R to facilitate optimization through gradient descent. When the 3D Gaussian is projected onto the image plane of perspective camera, the resulting distribution becomes complex since the perspective projection is not linear transformation. Following the approach in EWA splatting (56). 3DGS approximates the projected distribution on the image plane as 2D Gaussian. While introducing some errors, the local afline approximation simplifies the modeling of the projected 3D Gaussian, ultimately reducing compu- tational complexity and increasing rendering speed. Based on the perspective camera projection function क7 (()) K [ms/pe, Po/pe, 1]" with intrinsic matrix K, the first-order approximation of the
3

---

projection π is given
Og (H)
fo
0
fapa
J 0p
14
R2x3
(1)
0
where fax fg are focal lengths the camera and μ = [pzz Pay Mag is the mean vector of 3D Gaussian expressed in the camera coordinate system. As a result, the 2D Gaussian distribution on the image plane is represented with mean (H4) E R2 and covariance ΣnD JWEW"J" 2R222, where W denotes the transformation matrix from world space to camera space. The 2D Gaussian represents the intensity on the image plane and is normalized as follows to ensure the maximum value the center becomes 1.
For a E TR? Gan(2:) = exp (()))=-(())) Σn x(H))
(2)
This bell-shaped intensity is multiplied by the Gaussian's 's opacity to determine the pixel-wise opacity a. After frustum culling and sorting by depth, the color of each pixel C' is determined as,
C= >= Tj II k=1
o&):
JEN
where Cj is the color of each Gaussian. This accumulation process is performed in the order of depth sorting. Each pixel is processed independently by a single GPU thread, enabling rapid rendering of 3D Gaussians into images through this rasterization process. While 3DGS describes the rasterization pipeline of 3D Gaussians for the perspective camera, a rasterization pipeline for an omnidirectional camera requires distinct approach that regards its different optical characteristies. The following section explains our carefully designed rasterizer for the omnidirectional images.
3.2 Designing Rasterizer for Omnidirectional Images
A 360-degree camera captures all rays from the surrounding 3D environment to the camera origin and represents them on unit sphere S², We employ spherical projection or equirectangular projection (ERP) to map the projected image on the sphere S to the equirectangular space R2,, then transform to pixel space R², We describe series of steps on how 3D Gaussian is approximated as 2D Gaussian in the pixel space and the feasibility of such an approximation. The gist of our rasterizer design for the omnidirectional camera leveraging locally approximated perspective projection on S while minimizing errors.
As demonstrated in Figure we follow a camera coordinate convention with the z-axis forward, the x-axis to the right, and the y-axis down. We define spherical coordinates by setting the azimuth from the forward z-axis within the range [-x, T and the elevation @ from the z-x plane within the range [-m/2, x /2]. Projecting the mean l of the 3D Gaussian onto the unit sphere results in fu = ///|//] which corresponds to the azimuth Φ,, = arctan (Mz/Pzz and elevation 0, = arctan (-1t,/ V + 7.)) This spherical coordinate representation (ø) 0) is converted into pixel
space by multiplying scalar and adding center shift,
W
W
H
H
M) 2
T
(H)))) =
+
02
2T
2
π
where W, H are the width and height of the omnidirectional image, respectively.
While finding the corresponding point the center of 3D Gaussian on the pixel space is straightfor- ward, calculating the covariance requires more careful consideration. We model the distribution of 3D Gaussian projected onto pixel space as 2D Gaussian for computational efficiency and stability, following similar approach to 3DGS. We leverage the perspective camera and local affine approx- imation to intuitively describe the non-linear transformation introduced by the spherical camera characteristies and the equirectangular projection. Then, we mathematically prove the correctness of the proposed method.
Let us assume a perspective camera with unit focal length, where the image plane is tangent to the unit sphere. We rotate the perspective camera's forward direction to align with the position of the Gaussian center, μ. shown in Figure I (a). The image plane is tangent to the unit sphere at p., the

---

Asimuth
Elevation
T 0
T
Tungrat plans
Legend
(o) Coordiante transformation
Equirediangalar space
Gl
Oy
[c] Horizontal stretching
Goussian in 3D
S
Pixel space
(h) 'aangntt plane projection
(d) Rescaling and Combination
Figure 1: Illustration on rasterization process of ODGS. We describe the process of projecting a 3D Gaussian to the omnidirectional pixel space. (a) The coordinate is transformed from the original camera pose (black) to the target Gaussian (green), making the z-axis of the coordinate head towards the center of the Gaussian. (b) The Gaussian is projected onto the corresponding tangent plane. (c) The projected Gaussian horizontally stretched when transformed into equirectangular space. (d) The Gaussian in equirectangular space is linearly transformed to the pixel space, followed by combination with the other projected Gaussian.
point where line from the sphere's center to the center of 3D Gaussian intersects the sphere. We define the rotation matrix of the perspective camera T Tu. which is accomplished in two rotations in azimuth and elevation.
Coos d O
sin O O cos Oi
T Tap Top
cos O sin fz
sin 6, cos 6y
x
0
sin Ø
(5)
Cos Un
sin Φr
sin 0y sin 6i
cos Op sin Oμ
sin O2 cos ΦN
coos 02 sin 6y
O2 cos d)
The rotation of the coordinate system helps minimize the error between the unit sphere and the image plane, while also simplifying the covariance calculation. In the rotated camera coordinate, the position of the Gaussian is represented as po (0, O, ||])). Thus, the Jacobian matrix from Eq. is simplified as,
1/ |All
1/IA|l 0.
J.
(6)
because the focal length of the perspective camera is assumed to be one (fr = 1). Thus. the covariance of the 3D Gaussian projected onto this tangent plane is modeled as J. WEWT JJ as shown in Figure (b). We assume that the covariance of this 2D Gaussian is small enough to disregard the difference between the tangent plane and the sphere surface, allowing us to transfer directly onto the sphere surface. Although this assumption does not generally hold, we ensure its validity through the split rule in 3DGS, which keeps the size of the Gaussian small. Next, we map the 2D covariance from the spherical surface & to the equirectangular space (ø, f) R2. as described in Figure
[ (c). The equirectangular projection transforms the spherical surface onto cylindrical map, scaling ring at latitude 0 with initial radius of cos 0 on the sphere to radius of 1. This projection introduces horizontal scaling factor of see 0, leading to increased distortion as 0 approaches the poles. We incorporate the distortion through Q.. and then rescale the covariance to the pixel
5

---

space by applying the appropriate scaling factors So.
Q = sec 02
0
W/2n
1
S.-
=
#
As result, the final Jacobian matrix is given as,
W | sec p cos Uμ
2 2 see pp sin 0x
W
Jmmni = SoQOJOT
(8)
ا|اا|| sin Bg sin Un ا|لا|| cos 8p
" ||al|| sin 8r cos On
where the final 2D covariance is presented as Σ②D omnn WEWTJ omni We verify the correct- ness of the derived method by directly differentiating the equirectangulat alr projection function To in Eq. yielding the same result J omnss (m_((() as detailed in Appendix A.2
Uj
As a result of the series of steps, the final 2D covariance is used for rendering the image, as described crustum-shaped culling as
in Eq. 2 and Eq. []] One key difference is that, instead of performing in perspective cameras, we perform culling in spherical shell. The rasterization pipeline is fully differentiable and implemented in CUDA, which can be used as a typical 3DGS. The detailed gradient calculations through back- mack-propagation are provided in Appendix A.3
3.3 Densification Policy for Omnidirectional Images
Due to the characteristic of equirectangular projection, 3D Gaussian can be rendered in different shapes depending on its relative elevation to the camera; Gaussians near the poles are drawn larger. Therefore, we propose a dynamic densification strategy specifically designed for omnidirectional images. While the original method uses a pre-defined gradient threshold for densifying Gaussians, we apply varying gradient threshold Tμ according the elevation angle 0, as,
Ty Tmin (1 cos 0,,) (Tmax Tmin),
(9)
which mitigates excessive densification of Gaussians near the poles.
Experiments
4.1 Experiment Details
Datasets We evaluate our method on three egocentric datasets (OmniBlender, Ricoh360, OmniPho- tos) and three roaming datasets (360Roam, OmniScenes, 360VO) to show its superiority regardless of domain. First, EgoNeRF 100 released OmniBlender and Ricoh360, which have different char- acteristics. OmniBlender contains 11 synthetic scenes generated with an omnidirectional rendering engine in Blender [13), with four indoor and seven outdoor scenes. The images were captured by rotating in cireular motion while ascending, each with a resolution of 2000 1000. Each scene in OmniBlender consists of 25 training and test images. Ricoh360 contains 12 real-world omnidirectional outdoor scenes captured by rotating in place in a cross-shaped pattern. Each scene consists of 50 training images and 50 testing images with resolution of 1920 x 960. OmniPhotos (3] has released 10 real-world omnidirectional scenes captured by rotating in circular motion with commercial 360-degree camera on a selfie stick. Each scene has 71 to 91 images with a size of 3840 x 1920. In our experiment, we resize them to half resolution 1920 960, and we use 20% of images for the test.
For the roaming scenarios, we utilize several multi-view omnidirectional datasets, which were not originally released for 3D reconstruction tasks. 360Roam [21] dataset consists of 10 real-world indoor scenes captured by Insta360camera, Each scene has 71 215 omnidirectional images with size 6080 x 3040, and we resize them to 2048 x 1024. OmniScenes 28 is originally made for assessing the quality of visual localization of omnidirectional images in harsh conditions. Since is proposed to measure the robustness of visual localization algorithms. it contains significant scene changes, motion blur, or some visual artifacts such as jpeg compression. We use the released version 1.1, which includes real-world indoor captured scenes in resolution 1920 x 960. 360VO 20 is a simulation dataset for evaluating the localization and mapping algorithms the roboties field. contains 10 virtual outdoor road scenes, where each scene has 2000 images with size 1920 x 960.

---

Table Quantitative comparison of 3D reconstruction methods on various datasets. The best metric for each dataset is written in bold. Our method shows the best performance on almost all settings regardless of optimization time, with the fastest rendering speed.
10 min SSIM,
100 min SSIM,
Time (sec.)
Dataset
Methods
PSNR+
LPIPS,
PSNR+
LPIPS
NeRF(P)
19.20
0.6124
0.5359
20.04
0.6092
0.4949
62.71
3DGS(P)
29.36
0.8770
0.1400
21.19
0.7528
0.3021
0.112
OmniBlender
TensoRF
25.36
0.7249
0.3855
26.08
0.7416
0.3170
10.77
EgoNeRF
28.29
0.8309
0.2194
30.89
0.8934
0.1260
23.78
ODGS
32.76
0.9234
0.0469
33.05
0.9229
0.0343
0.028
NeRF(P)
14.33
0.5616
0.5794
16.16
0.5617
0,5716
62.46
3DGS(P)
25.12
0.7932
0,2397
22.07
0.7228
0.3218
0.132
Ricoh360
TensoRF
23.35
0.6812
0,5200
23.97
0.6936
0.4653
10.30
EgoNeRF
24.74
0.7467
0,3243
25,49
0.7737
0.2825
23.89
ODGS
24.94
0.8135
0.1489
26.27
0.8462
0.1051
0.026
NeRF(P)
18.14
0.6158
0,5514
20.80
0.6388
0.4772
62.08
3DGS(P)
25.61
0.8310)
0,2100
23.30
0.7859
0.26670
0.110
OmniPhotos
TensoRF
22.78
0.6841
0,5089
23.73
0.7038
0.4467
9.707
EgoNeRF
25.20
0.7722
0.2662
26.90
0.8349
0.1766
23.88
ODGS
26.24
0.8704
0.1108
27.04
0.8878
0.0875
0.028
NeRF(P)
15.07
0.6848
0.4839
15,26
0.6813
0.5025
62.98
3DGS(P)
20.17
0.7001
0.3536
19.34
0.6576
0.3837
0.104
360Roam
TensoRF
18.00
0.5988
0.7488
18.12
0.5895
0.7133
9.052
EgoNeRF
20.45
0.6358
0.5334
21.18
0.6718
0.4444
24.03
ODGS
21.08
0.7066
0.3003
20.85
0.7111
0.2254
0.029
NeRF(P)
15.69
0.7218
0.4546
15.98
0.6890
0.4914
62.90
3DGS(P)
23.61
0.8444
0.2835
17.14
0.7119
0.3906
0.194
OmniScenes
TensoRF
23.58
0.8118
0.3534
24.21
0.8208
0.3091
8.100
EgoNeRF
22.78
0.7997
0.3463
24.76
0,8313
0.2623
23.66
ODGS
24.42
0.8526
0.1391
24.51
0.8505
0.1282
0.032
NeRF(P)
15.71
0.6186
0,4949
17.78
0.6373
0.5064
61.97
3DGS(P)
22.87
0.7861
0,2970
22.73
0.7822
0.3061
0.091
360VO
TensoRF
19.74
0.6543
0.5876
20.31
0.6721
0.5640
7.815
EgoNeRF
22.47
0.7325
0.4342
23.78
0.7677
0,3680
23.96
ODGS
24.63
0.8245
0.2175
26.68
0.8694
0.1264
0.026
We uniformly select 200 images for each sequence for training and testing. Since these datasets do not split the train and test images, we conducted our experiment by dividing them by 4:1 for train and test, respectively. We note that all datasets have CC-BY-4.0 licenses. Although some datasets provide camera poses and dense point clouds, we run the structure- costueefroommoiiion specifically OpenMVG [7], on all the datasets and use obtained poses and point clouds for experiment.
Implementation details Our framework is basically built with PyTorch [39 but we manually implement the omnidirectional rasterizer using the CUDA kernel. All experiments, including optimization and inference time measurements, are conducted using a single NVIDIA RTX A6000 GPU. We deseribe the optimization arguments in the Appendix A.1
4.2 Experiment Results
Baselines With no available code for 3DGS on omnidirectional images at the time of our ex- periments, we compare our method with NeRF-based methods, specifically TensoRF 7 and EgoNeRF 100 We also convert omnidirectional images into perspective images to compare the typical 3D reconstruction methods, NeRF (36) and 3DGS (27]. Specifically, we transform the omni- directional images into six perspective images using cubemap decomposition, popularly used in many

---

(P)
3DGS(P)
T
EgaNeRF
ODGS
OmniiBlender hattiiddddrr/ttto.eeee /beettoeeeettrreee
0.6
PSNN22
251
0.8
LPIPS
0.4
0.7
20-
0.2
15
0.5
0.0
60
90
100
20
40
60
100
Time (siin)
'1 Time (min)
Time (min)
OmniPPhotce / Ballintow
122
0.5
20-
0.4
PSNN2
26-
SSIM
1.PPSS
23-
20-
0.1
100
200
300
100
200
300
0
100
200
300
Time (mina)
(min)
Time (min)
Figure 2: Changes of PSNR, SSIM, and LPIPS according to the optimization time for each method. ODGS shows the best result as well as the highest convergence speed in both scenes.
The six decomposed images compose a cube-shaped surface and calculate the corresponding camera pose of each surface. For inference, the six views for each face in the cube are rendered and then combined into an omnidirectional image.
studies involving 360-degree cameras 25 47
Quantitative comparison To ensure the experiment's fairness and highlight the efficiency of our method, all methods were evaluated after optimizing the model with the same amount of time. We evaluate the performance of all methods at 10 and 100 minutes of training time, measured in wall-clock time. For evaluation metric, we use PSNR (dB). SSIM (48). and LPIPS [53) for comparing reconstruction quality, where AlexNet [29 backbone is used for measuring LPIPS. Table T shows the quantitative performance comparison and rendering time (seconds) for all datasets. The (P) mark in the method column indicates those methods are trained with converted perspective images. Our results show dominant results all metries, including inference time. NeRF and TensoRF, which use a grid based on a Cartesian coordinate system, encounter difficulties representing large scenes, resulting in poor quantitative metrics. EgoNeRF, which introduces a spherical balanced grid to mitigate the challenge, shows better quality than TensoRF but still needs better perceptual metrics. Also, these methods require more than a second to render a single omnidirectional image for an arbitrary viewpoint, which is impractical for real scenarios. Meanwhile, 3DGS with perspective images shows the best results except ours when optimized for 10 minutes, but severely suffers from overfitting and gets worse results after 100 minutes of optimization. In terms of rendering time, despite reporting faster speed than NeRF-based models, original 3DGS takes longer than typical perspective image rendering because it involves non-linear warping of each image when stitching six images to create one omnidirectional image. ODGS, in contrast, outperforms the other methods in image reconstruction quality and rendering speed. The outstanding results for SSIM and LPIPS imply that our method generates images with accurate structure and prominent perceptual quality.
Figure 2 shows the change of PSNR, SSIM, and LPIPS depending on the optimization time for two example scenes. We note that we stopped training NeRF and TensoRF at 100 and 200 minutes, respectively, since their performances converged. Our method shows the fastest optimization speed in both scenes while maintaining the highest score regardless of optimization time. Typical NeRF and TensoRF recorded significantly lower results than ours, verifying that the Cartesian coordinate is inappropriate for radially extending rays. EgoNeRF shows comparable PSNR with ours in Ballintoy, but needs a long optimization time. We attribute the fast optimization of ODGS to two aspects. First, while NeRF-based methods use an implicit representation that embeds the scene into a neural network, 3DGS employs explicit representation and directly moves or morphs the elements to optimize the model. Also, 3DGS exploits the position of SfM point clouds, which can serve as a good initialization point for optimizing Gaussian splats. 3DGS (P), on the other hand, shows high vulnerability to
8

---

(a) Ground truth
(b) 3DGS(P) 20
(c) EgoNeRF 1T
(d) ODGS
Figure 3: Qualitative comparisons in the egocentric scenes (10 min.). Each scene is brought from Ricoh360, OmniBlender, and OmniPhotos, respectively. Best viewed when zoomed in.
858
(a) Ground truth
(b) 3DGS() 27
(c) EgoNeRF 10
(d) ODGS
Figure 4: Qualitative comparisons in the roaming scenes (10 min.). Each scene is brought from 360Roam, OmniScenes, and 360VO, respectively. Best viewed when zoomed in.
overfitting. believe the phenomenon happens because of the weak correlation among the six faces of the cubemap after decomposition. Since there is no overlap between the six faces, 3DGS is optimized six times independently faces facing the same direction. Therefore, even with the same input, the amount of information used is significantly reduced, causing overfitting occur quickly.

---

Qualitative comparison We also visually compare our method with the other methods in various scenes. Souree]show the samples reconstructed imgges from egocentric datasets. We note that the images in the figure are rendered at 10 minutes of training. The images from EgoNeRF are blurry and contain some artifacts, such as stripe lines or checkerboard patterns, which appear prominent near the edges. The model is not sufficiently optimized to render the sharp image details. 3DGS trained with the cubemap perspective images sometimes show sharp reconstruction contents, such as a cubic pattern of frame in the middle row (yellow boundary) but often include unintended projected Gaussian splats that cause image distortion. We attribute the phenomenon to the rapid overfitting properties of perspective 3DGS. In contrast, model successfully reconstructs sharp details in the images. The superiority of ODGS becomes more noticeable in the roaming dataset, as shown in Figure Although EgoNeRF proposes balanced grid for egocentric video, it cannot maintain a uniform ray density for every grid the camera wanders inside large environment. As a result, the scene pattern is often completely lost, creating completely different results, and the overall reconstruction quality deteriorates. While perspective 3DGS shows better quality than EgoNeRF, often misses some objects structures where the adjacent faces of the cube meet. For instance, in the top row, there is an inverted Y-shaped artifact instead of a chair in a purple patch. This happens because the chair located where the three sides of the cube meet, and the object any of the sides. ODGS overcomes the challenge by optimizing the Gaussian using the whole image and showing prominent performance on both egocentric and roaming datasets.
made from
(a) Ground truth(full)
(b) Ground truth
(c) Static threshold
(d) Dynamic threshold
Figure 5: Qualitative comparison of rendered images according to the Gaussian densification policy
during optimization.
Dynamic Densification Strategy for Onnidirectional Images We qualitatively compare and display the results in Figure when when applying the proposed dynamic densification rule proposed in Section 3.3 As shown in the figure, the lanes appear split, with artifact-like patterns emerging on the road due to static densification, as employed the original 3DGS work (27]. Conversely, our densification strategy significantly enhances the model's 's representation power, leading to markedly more accurate rasterization results.
Ablation Study:
Conclusion
In this work, propose new method called ODGS, specifically designed reconstruct 3D scenes from omnidirectional images using 3D Gaussian splatting. To optimize 3D Gaussian splatting in the omnidirectional image domain, equirectangular projection from the 3D space to the image. Specifically, we define a tangent plane for each Gaussian and project the Gaussian into the plane, followed by horizontal stretching and rescaling to the pixel space. Compared to the state-of-the-a NeRF-based methods, ODGS shows about 100 times faster optimization and rendering speed, which allows the user synthesize the novel view in real-time. Furthermore, ODGS shows the best reconstruction performance for various input images, including egocentric and roaming scenes, indoors and outdoors.
introduce new rasterizer that appropriately models the
Limitations and future work ODGS still relies on local affine approximation when projecting a Gaussian splat to the camera surface. Equirectangular projection is not linear transformation, and straight lines in the 3D space should be expressed as curves in the omnidirectional image. However, a 3D Gaussian is approximated as 2D Gaussian, leading errors that produce artifacts in the rendered image. Adopting a more accurate distribution for spherically projected Gaussians can reduce errors and enhance the efficiency of the framework.
10

---

Acknowledgements
This work was supported part by the ITTP grants [No.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University), No. 2021-0-02068, and No. No.2023-0-00156) the NRF grant [No. 2021M3/9E40000822 funded by the Korea government (MSIT), and the SNU-LG AI Research Center.
References
Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin. Matryodshka: Real-time 6dof video view synthesis using multi-sphere images. ECCV, 2020.
[2] Jiayang Bai, Letian Huang, Jie Guo, Wen Gong, Yuanqi Li, and Yanwen Guo. 360-gs: Layout-guide panoramic gaussian splatting for indoor roaming. arXiv preprint arXiv:2402.0076. 2024.
[3] Tobias Bertel, Mingze Yuan, Reuben Lindroos, and Christian Richardt. Omniphotos: casual 360 vr photography. ACM TOG, 2020.
[4] Gary Bradski. The opencv library. Dr. Dobb's 's Journal: Software Tools for the Professional Programmer, 25(11):120- 123, 2000.
[5] David Caruso, Jakob Engel, and Daniel Cremers. Large-scale direct slam for omnidirectional cameras. In IROS, 2015.
[6] Wenjie Chang, Yueyi Zhang, and Zhiwei Xiong. Depth estimation from indoor panoramas with neural scene representation. In CVPR, 2023.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
Rongsen Chen, Fang-Lue Zhang, Simon Finnie, Andrew Chalmers, and Tachyun Rhee. Casual 6-dof: free-viewpoint panorama using handheld 360 camera. IEEE TVCG, 2022.
[9] Ching-Ya Chiu, Yu-Tin Yu-Ting Wu, Shen, Yung-Yo Chuang. et al. 360mvsnet: Deep multi-view view stereo network with 360deg images for indoor scene reconstruction. In WACV, 2023.
[10] Changwoon Choi, Sang Min Kim, and Young Min Kim. Balanced spherical grid for egocentric view synthesis. In CVPR, 2023.
[11] Jaeyoung Chung. Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain- free generation of 3d gaussian splatting scenes. arXiv preprint urXiv:2311.1338, 2023.
[12] Jaeyoung Chung, Jeongtaek Oh, and Kyoung Mu Lee. Depth-regularized optimization for 3d gaussian splatting in few-shot images. In CVPR Workshop. 2024.
[13] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018
[14] Mare Eder, Pierre Moulon, and Li Guan. Pano popups: Indoor 3d reconstruction with plane- network. 3DV, 2019.
[15] Yasutaka Furukawa, Carlos Hernández, Multi-view stereo: tutorial. Foundations and Trends Computer Graphics and Vision, 2015.
[16] Christiano Gava, Vishal Mukunda, Tewodros Habtegebrial, Federico Raue, Sehastian Palacio, and Andreas Dengel. Sphereglue: Learning keypoint matching on high resolution spherical images. In CVPR, 2023.
[17] Kai Gu, Thomas Maugey, Scbastian Knorr, and Christine Guillemot. Omni-nerf: neural radiance field from 360 image captures. In JCME, 2022.
[18] Tewodros Hahtegebrial, Christiano Gava, Marcel Rogge, Didier Stricker, and Varun Jampani. Somsi: Spherical novel view synthesis with soft occlusion multi-sphere images. In CVPR, 2022.
[19] Ching Yu Hsu, Cheng Sun, and Hwann-Tzong Chen. Moving in 360 world: Synthesizing panoramic parallaxes from single panorama, arXiv preprint arXiv:2106.1088: 2021.
[20] Huajian Huang and Sai-Kit Yeung. 360vo: Visual odometry using single 360 camera. In ICRA, 2022.

---

[21] Huajian Huang, Yingshu Chen, Tianjia Zhang, and Sai-Kit Yeung. Real-time onnidirectional roaming in large scale indoor scenes. In SIGGRAPH Asia 2022 Technical Communications, 2022.
[22] Huajian Huang, Changkun Liu, Yipeng Zhu, Hui Cheng, Tristan Braud, and Sai-Kit Yeung. 360lec: A dataset and benchmark for omnidirectional visual localization with eross-device queries. arXiv preprint arXiv:2311.
2023.
[23] Letian Huang, Jiayang Bai, Jie Guo, Yuanqi Li, and Yanwen Guo. On the error analysis of 3d gaussian splatting and an optimal projection strategy. In ECCV, 2024.
[24] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang. Xiaoyang Lyu, Yan-Pei Cao,, and Xiaojuan Qi. Se-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:23/2. 14937, 2023.
[25] Hyeonjoong Jang, Andreas Meuleman, Dahyun Kang, Donggun Kim, Christian Richardt, and Min Kim. Egocentric seene reconstruction from an onnidirectional video. ACM ToG, 2022.
[26] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint arXiv:3312.02133 2023.
[27] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühller and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023.
[28] Junho Kim, Changwoon Choi, Hojun Jang, and Young Min Kim. Piccolo: Point cloud-centric omnidirec- tional localization. In CVPR, 2021.
[29] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012.
[30] Shreyas Kulkarni, Peng Yin, and Sebastian Scherer. 360fusionnerf: Panoramic neural radiance fields with joint guidance. In IROS, 2023.
[31] Longwei Li. Huajian Huang, Sai-Kit Yeung. and Hui Cheng. Omnigs: Omnidirectional gaussian splatting for fast radiance field reconstruction using omnidircctional images. arXiv preprint arXiv:2404.03202, 2024.
[32] Ming Li. Xueqian Jin, Xuejiao Hu, Jingzhao Dai. Sidan Du and Yang Li. Mode: Multi-view omnidirec- tional depth estimation with 360 cameras. ECCV, 2022.
[33] Qiaoge Li, Itsuki Ueda, Chun Xie, Hidehiko Shishido, and Itaru Kitahara. Omnivoxel: A fast and precise reconstruction method of omnidirectional neural radiance field. GCCE, 2022.
[34] Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew J Davison. Gaussian splatting slam, arXiv preprint arXiv:2312.067744 2023.
[35] Andreas Meuleman, Hyeonjoong Jang, Daniel Jeon, and Min H Kim. Real-time sphere sweeping stereo from multiview fisheye images. CVPR, 2021.
[36] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[37] Pierre Moulon. Pascal Monasse, Romuald Perrot, and Renaud Marlet. OpenMVG: Open multiple view geometry. In RRPR, 2016.
[38] Alain Pagani and Didier Stricker. Structure from motion using full spherical panoramic cameras. In ICCV Workshop, 2011.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style. nigh-performance deep learning library. In NewrIPS, 2019.
[40] Giovanni Pintore, Eva Almansa, Marco Agus, and Enrico Gobbetti. Deep3dlayout: 3d reconstruction of an indoor layout from a spherical panoramic image. ACM TOG, 2021.
[41] Shivansh Rao, Vikas Kumar, Daniel Kifer, C Lee Giles, and Ankur Mali. Omnilayout: Room layout reconstruction from indoor spherical panoramas. In ICCV, 2021.
[42] Davide Scaramuzza, Agostino Martinell, and Roland Siegwart. A flexible technique for accurate omnidi- rectional camera calibration and structure from motion. In ICVS, 2006.
12

---

[43] Johannes L Schonberger and Jan-Michael Frahm. Structureffoom-moklee motion revisited. In CVPR, 2016.
[44] Shinya Sumikura, Mikiya Shibuya, and Ken Sakurada. Openvslam: A versatile visual slam framework. In ACMMM, 2019.
[45] Cheng Sun, Chi-Wei Hsiao, Ning-Hsu Wang, Min Sun, and Hwann-Tzong Chen. Indoor panorama planar 3d reconstruction via divide and conquer. In ICCV, 2021.
[46] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024.
[47] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai. Bifuse: Monocular 360 depth estimation via bi-projection fusion. In CVPR, 2020.
[48] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Ecro P Simoncelli, ct al. Image quality assessment: From error visibility to structural similarity. IEEE 7IP, 2004.
[49] Changhee Won, Hochang Scok, Zhaopeng Cui, Marc Pollefeys, and Jongwoo Lim, Omnislam: Omnidirec- tional localization and dense mapping for wide-baseline multi-camera systems. In ICRA, 2020.
[50] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024.
[51] Qi Wu, Xiangyu Xu, Xicyuanli Chen, Ling Pei, Chao Long, Junyuan Deng, Guoqing Liu, Sheng Yang, Shilei Wen, and Wenxian Yu. 360-vio: A robust visual-inertial odometry using 360 camera. IEEE TIE, 2023.
[52] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. CVPR, 2023.
[53] Richard Zhang, Phillip Isola, Alexci A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[54] Shijie Zhou, Zhiwen Fan, Dejia Xu. Haoran Chang. Pradyumna Chari, Tejas Bharadwaj. Suya You, Zhangyang Wang, and Achuta Kadambi. Dreamscene360: Unconstrained text-to-3d scene generation with panoramic gaussian splatting. ECCV, 2024.
[55] Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. Fsgs; Real-time few-shot view synthesis using gaussian splatting. ECCV, 2024.
[56] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa splatting. IEEE TVCG, 2002.
13

---

A Appendix supplemental material
A.1 More Implementation Details
We follow the hyper-parameters of original 3DGS [27] excluding some hyperparameters. Firstly, we set iterations as 200k, densify _nt___llll _iter as 100k. However, we stopped the optimization after 100 minutes, regardless of the current iteration. For densification set percent dense 1e-3, densify _rrd threshold min (Tmin) as 2e-5, and densify_ threshold max (Tmax) as le-4.
A.2 Proof of Mathematical Equivalence of the Derived Method
Here, we present the direct derivation of Eq. 8 function To from Eq. 4
by differentiating the omnidirectional projection
H:
0
W
On, (p)) 0μ
2 p +
2
||ll
/23 +
Pafy
H
12
+
1
F
My
W
w
uell
V +He
Vu?+u*
[all
Vug+ue H|ll
-My
-My ue
(10)
*|اا|
||a||
vu² +u?
*|ل|||
a||l|
|All
V
W
sec 0₂ cos 6,
02 sin d
2 lell
2x [ill
H |||l|
sin O2 sin 6
ल |||l||| cos 0μ
sin 0₂ cos 6
x [ill
oman
This proof demonstrates the mathematical correctness of our description outlined through Eq. 5 Eq. [6. and [ The description in the main paper reveals the underlying assumptions (local affine approximation, tangent plane to sphere surface) and confirms their mathematical validity.
A.3 Back-Propagation Rasterization in omnidirectional Image Domain
The gradient computation from Gaussian covariance related to Eq. 10 value matrix for the projected 2D covariance matrix(E) as 2L Ø The size of
We denote the gradient
2L is the same as
Ø5
the original covariance matrix. Note that the values ax (1.2)
at. and 0T. are same since both
25 (2.1)
and 25
0L are symmetric matrices.
We define T as J W. The gradient value matrix of T is computed as below:
14

---

OC
=2 Va T1.2) + T(1,3) ((.3)) * dL
(1.))
(1.1
dc
+ +Ta., + (2.2) + (2.3) (1.33) / (1.2)
ac
ÔT.
(Ta.)) + (1.2) 22.22 (1,3) ((2,)) (1.1)
OT (1.2)
aL
+ +Ten + (22)) (222) + (2.3) (2.3)
0Σ (1.2)
ac
ØL
2 (Ta..) + (1.22 Va.2) (1133) V(.3)
(13)
(1.11
aL
+ (TanNVaa. + (222) (3.2) + T (2,3) (3,3))
0 (1,2)
(11)
oc
0C
2 (T + (2.2) (1.22 + ((233) (1.3))
8T (2.1)
0 (2.2)
dc
+ (To.)) + 11.2) + T (1,3) V(1,3))
3Σ (1.2)*
0C
dC
2 (Ta.)) + (2.2) (2.2) T (2,3) ((2.3) *
8T (2.2)
0Σ (2.2)
OC
+ (Ta,
+ 1 (1,2 (2.22) + T (1.3) (223))
0Σ (1,2)*
dC
OC
2 (2..1))(3.11 + T(2 (3.22 (2.3) V(3.3))
ØT (2.3) (23)
2 (2,2)
OC
+ (TonnVan + (1.2) + (1.33) (3.33)
0Σ (1,2)
0c is calculated multiplication of WT and 8C Then, the gradient for Jacobian matrix, 37 gradient of J is calculated, the gradient for each position is computed as follows:
After the
8T
0C
W
tt
OC
W
2 - 12 OL
Öt
(12 +2) 0J (1.1)
2
(t2
+2)
OJ (1,3)
H ty 2 +()
H 1 (12 212) OL
(
0J (2.1)
tV + 2
aJ (2.2)
H 1. (2 (2) +
ac
m
(12 12)
aJ (2.3)
JC
H t (12-222)
OC
2H ty
2 2 3C
+
Oty
4VC
0J (2.1)
OJ (2.2)
(12)
to (2-
OC
4/2 +
0J (2.3)
3c
W
-2
OC
W
tx
OC
.0
2T
(12 12)
0J (1,1)
π
(12
12) 2
DJ (1.3)
H
t.tgts (2 (12 12)
12)
0C
H to (t2 21.)
0C
(f2 + 12) 3/2
0J (2.1)
1/
8
0J (2.2)
ty (E 21 (12 22)
0c
+ (f2 + 3/2
DJ (2.3)
15

---

A.4
Comparison with Original 3DGS with more input images.
Table A: Quantitative comparison of 3DGS (P) in 6-views and 18- views.
10 min
100 min
Dataset
Method
PSNR,
S
LPIPS|
PSNR,
SSIM,
LPIPS|
3DGS (P6)
29.36
0,8770
0.1400
21.19
0.7528
0.3021
OmniBlender
3DGS (P18)
27.85
0.8387
0.1737
24.56
0.7907
0.2478
ODGS
32.76
0.9234
0.0469
33.05
0.9229
0.0343
3DGS (P6)
25.12
0.7932
0.2397
22.07
0.7228
0.3218
Ricoh360
3DGS (P18)
24.76
0.7726
0.2565
23.14
0.7277
0.3109
ODGS
24.94
0.8135
0.1489
26.27
0.8462
0.1051
3DGS (P6)
25.61
0.8310
0.2100
23.30
0.7859
0.2670
OmniPhotos
3DGS (P18)
24.93
0.8007
0.2412
23.21
0.7541
0,2996
ODGS
26.24
0.8704
0.1108
27.04
0.8878
0.0875
3DGS (P6)
20.17
0.7001
0.3536
19.34
0.6576
0.3837
360Roam
3DGS (P18)
20.88
0.6992
0.3571
21.05
0.6994
0.3405
ODGS
21.08
0.7066
0.3003
20.85
0.7111
0.2254
3DGS (P6)
23.61
0.8444
0.2835
17.14
0.7119
0.3906
OmniScenes
3DGS (P18)
24.00
0,8400
0.1993
21.43
0.7864
0.2828
ODGS
24.42
0.8526
0.1391
24.51
0.8505
0.1282
3DGS (P6)
22.87
0.7861
0.2970
22.73
0.7822
0.3061
360VO
3DGS (P18)
23.22
0.7875
0.2939
23.57
0.7938
0.2825
ODGS 24.63
0.8245
0.2175
26.68
0.8694
0.1264
Although we report the results the original 3DGS method using cubemap decomposed perspective images in Table we also compare the results when 3DGS is optimized with more perspective images. Whereas the original decomposition generates six perspective images for one omnidirectional image, the new decomposition produces 18 images by adding 12 perspective cameras where each camera faces an edge of the cube. Table shows the performance the number of perspective images for optimization. When using the 18 views (P18), the performance comparable to the 6 views (P6) at the 10-minute mark but surpasses the view results after the 100-minute optimization. At first, the increased number of images for training prevents the model from sufficiently learning from all views in the early stages (10 minutes), resulting in slightly lower performance. However, after sufficient optimization time (100 minutes) passes, the additional views allow for further optimization, leading to improved results. Still, ODGS shows the highest performance in most metrics, even considering 3DGS using views, demonstrating the superiority of our rasterizer.
optimized results according to
16

---

A.5 Detailed Quantitative Results
We provide detailed quantitative results that compose Table I below. We report PSNR, SSIM, and LPIPS of each method for all scenes in Table Table G The tables show that our method outperforms all the baselines in almost all scenes. Please note that the room! scene in the OmniScenes dataset was omitted because the OpenM VG did not function properly due to the image sequence we report the averages of the scenes, excluding room].
jumping in the middle. In Table T
Table B: Quantitative comparison of 3D reconstruction results on Omniblender dataset (10min/100min). The best result for each metrie is written in bold. Our method shows the best performance on almost all settings.
opt
scene
Nerl(P
3DGS/P)
TensoRF
EgoNoRF
ODGS
archiviz-ttat
20.42/0.7411
F10402
31,20/ 11220/99188/// o
28.48/ 10.2751
29,06 0.8501 /0.2477
0.0246
butbershop
19,50 116011 /1..544
/1.9422 0.0849
27.5%
10:325))
31.13/
/0.1793
0.9778 0.02U3
histro hike
17.490..5145 /0.615
30.56 (0.9798 0890
30,01 0009005 1.30055
0.9712/ 8.0195
Nstro square
16.27 70:090000.000000000
24.95 70.5387 /0.1273
19.52/ 0 0.56620 00:9974
23.84 0.88258 00.1605
28.59/
0,9500 0.0269
clessrpom
18.16/1..6125 71.6144
26,39 /1.8:19 0 (0.2291
25 (22 0 0.7225 / 10,5100
26. 26.33 / 0.70022/ / 00.3740
31 31.54/ (.8674) 0.1161
10min
lone moak
fisher but
23.43/0:77 /0.7171 70.4084
16.88/0.5560 00,5561 /0.5029
20 /0.8153/0000 18009
30.01 /0.9204/ 00.000600.00 /0.0029
23. 18 0.6933 / /0.3530
28. / 7549 / /0:4224
29.75 775/0..7700 / 70.508
-0.8777 / /0.1433
33.49/ 0.9638 0.0201
32,39 / 008851 0.0610
LDU
17.79 (2-211 1/ /0.47725
30.68 /0.9309 /O. 10.1050
22 60 O 0.8551 / /0.21744
31,89 00.5083 /0.1171
35.17/
009673 0.6314
pavilion unddaa chair
19,99 7027 /1.55288
29.09 /0.9127/0.000 001/22/0..105
26, 33 / 00..55000.300 /0.3237
29.01 /0.88277 / /0.1354
32.13
0.9499) / 0.0324
pavilion midlay pond
17,86 1.86/0..5900 5496 /0.5600
24.53 22..0.9994/.1111
22. 11:0.66113 /0.6613/0.5000
23.8610. (..4411 / /0.1963
25.23/0.5125/
0.5115 0.0954
restroom
25.37 /0,5421 /0.6432
31,20 21.20/0..3440/000 0.7840/0..256
28.60 0.0664 00.509
24,07 0,09300 0.4460
35.37/
0.9020 0.0684
average
19,20 12.00/1.6114/110 /0.5359
29.36
00.8770/0.1400
25.36 0.7249 10.3855
28.29
0,8309 0.2194
32.76/
0.9469
archiviz-flu
21.28 /1.7326 710.3953
21.61
/0..005/0000 0.2274
05555 10.2185
0099755 00,007
0.9454/ 0.0243
barbershop
D..55/ /0.5326
/0.7708//111 10.3194
00.2606
0.0551 /0.0835
0.0204
bistro
00.507 /0.5828
19.39/
202444
22.58: 0.6223 10:4310
00.9516/0.00 0.0047
0.9752/ 0.0144
bistro aquare
17.06 0..501.. /1.5611
21.27 /1.0000001000 24244
19:86 0.57144 10:45119
25,49
0 (0.0540
0.953/ 8.0228
chaerrom
18.87 1. /00.367
18:52 70.6499 /0.4318
26 25.95/
O 7481 10:4354
29.55 0. 0383306/1
31.54/ 0.8524/ 0.0567
100min
lone mork
lisher but
24.87 / /1.3964
12.27 (D. 54.59 /0.4838
25.92 /0.7822 0.2200
19.12 /0.7479 /O (Q.1114
23.73/ 71661 / /0.2550
28.74; 0.7514 1033500
31.32/ 1 0.80018 / 00.2338
31..90/ D. 0.9381 / /0.0000
32,88/ IL966IN 0.01
32.75 / 0.8604 0.0525
LOU
19.95 (D.6619 F11.4108
21.59 /0.8199/ (025515
28. 60 (00000000 8430 / /D.1756
33.53 D. (1.9400///// 00,0903
35.43/
(L95733 0.0277
puvilion midday chuir
21.58 ( D.6954 /0.4329
19.60 /0.7522 / /0.3560
27.11 00:015 /
31.08 0.933.5 / 000668
32,93/ 0.9562 0.0266
pavilion midday pond
17,50 12.3346 /1..5442
20,89 /..2677 / 002259
22.59/ 0.6755 /0.25122
25.71 7 (..81445 / (0.1240
25.40 0.S18Y / 0.0783
restroom
24.307 0., 0.5407 /0.6079
22.91 F0.6301 /0.4204
20:49) 7261 / 0.3377
32.86 7 0.8555 00.2167
35.14/ / (18957 / 0.6354
avergg
70.0470.600. 70,4949
21.19 /1.752 (0.322
38.08/0..4116703310
30.89 70.89/00.8070 0.800//0..10000
33,05 / (0.922 / 8.6343
Table C: Quantitative comparison of 3D reconstruction results on Ricoh360 dataset (10min/ I 100min). The best result for each metric is written in bold. Our method shows the best performance on almost all settings.
upt. time
svene
NeRF(P)
3DGS(P)
TensoRF
EgoNeRF
ODGS
bricks
12.40 0.4489 0.6400
/0.875
/0.2275
/0.4918
23.09/70.7233 /022064
23.90/ 0.8185 0296
bridge
14,96 00566/..550 0.5553/ 05553300.586 0.5866
23.78/0.7783/
/0.2145
21.93 /.6455/ 00.66/0..53
23.34/0.7199 0.7119 0999/0..00 0.3009
23.88 /0,7967/ 0.1286
bridge under
19.71 0.4987/ 0.6769
24.30/ 00.292/
21.99
...6333 /0.579]
24.11 0.7500 / 00.3202
25.12 0.8347 0.1339
tower
/.5182/ 0.550/
21.33 00.5//0754322222 /0.2548
/0.600 /0.6002
13.80/ 0.6861
/0.771/ /0.1435
center
14.76 D.6691 00.221
0.8364 /0.2687
27.97 0.8450
28.10 0.8710/ 0.1206
furm
14,45 / 0.4970/ 0.6262
21.66:
0.6897 /0.3248
20.80 / 000556333/000 /0.514
21.80 0.6403/0633////33 00.336
0.6888 20.74/0.6881
20.74
/0.2270
10min
flower
12.03 / 0.11.2// 0.6912
21.71 0.5942/ 00.324
20.07
00.5414 06611//000 /..669
21.51 19 0.1121
22.19 /0,7273 0..195
galkery chair
15:40/ 0.6650 0.1999
27.76
0.8772 10,196
26 00 0. 7907 /0.5233
27.01 / 0.8336 / 0..3409
27.29 0.8777 00,1353
gallery park
0:5176
30
/U.2384
/0.7304/ /0.5120
3045
/0,1341
gullery pollar
14.50/ D.6445 00,9902
27.79/ 0.56/3//........
/
27.31
/00682
garden
13.97 /3.66/0 0.5430
27.53
0.7919/0.21188
25.37 /0.6/19/00566111
26. 26.18/ 7..715 /0.3577
23.20
0.75843////// 7 00.2299
poster
14,99 /..95/0.655//000 0.5679
26.14/0.8599
01.5/00859//000000 /0,2098
23.20/
05..39/08221///00
26,90 0.8762 00.1249
average
14.33/0.5616/ 00,7944
25.12/ 0..292/0..2997
23.35
0..6812/....333 0.5200
74/ O. 0.7467 /0.3243
24.94 0.8135 0..4489
bricks
15.01
/0.2245
22.60
00.7410/00225555 /0.2855
21.66 0.6353 /0.4375
23.93 / 7616616//0 / 0.24455
24.62
0.8479 0.1021
bridge
17.32/0.5558/ /0.5620
21.94
0.71577 /0.3133
/0.658 /0.4305
/ 0 16
0.8154/ 0.1063
hridge inder
16.42/ /.502/ 00.6447
19.03 0.63777 /0.3663
/0.657 4836
7924 /024492
25.93/ 0.8538/ 0.1026
cat tower
15.45 / 0.5323/ 0.5824
21.24/
00.851 /0.3565
23.02
(0.6993/ 0477477
24.52/ / 7163 /0.477
25.35
/0,0088/ 0.1109
center
17.09 / 0.6566 / 0.4855
20.04/ 0.69974 /0.4237
27 27.50 0.8152 ( 0.3840
29, 12 0 0.3625/ 70.2119
29.39/
0.59900 0.0808
furm
15.93 0.1830 0.6173
21.49: /
21.09/ / 0.55765/ 0.4662
22.25 0.6745 0..065
16.34 0.6039//
0.1109
100min
flwwr
13.57
0.6815
20.45/
/0.3531
5506
gallery chair
17.50 D.6873 00.2240
0.5509 /0.2161
/ 0,4766
27.62 00.8831/ 0.1135
gallery park
/..24/0.5847) 0.5400
23:22/ 0.7637/ /0.3027
24.64 7.7757 0.4724
25.64 0.7888 00.000
26.19 0.8401 0.1076
gallery pollar
16.57 0.6-105 0. 0.19-16
21.93/ / 0.7429 /0.3205
26. 19 / 0,7960 0.3336
27.97 0.8116 00.2101
26.74 0.8970 0.0693
gamden
17.81 0.55640 0.5101
25 9T/ 0.7792 /0,2528
25. 91.99/ 0.6338 0.520)
27.16 7 7441 /0.3112
27.09 0.833/ O. 0.1006
poster
/.6116
0,5794
25.45/ 0.7 0.3406
24.32 0.7750 0.4161
26.50/ 0.8497 0 0,2613
26.92/ 0.8508/ 0.1113
average
16.16/0.3617/0.577
0.7200002222/20 /0.322
/0.6936/ 0.6653
/ 70.2825
25.44 0.8260 0.1366
17

---

Table D: Quantitative comparison of 3D reconstruction results on Omniphotos dataset (10min/100min) The best result for each metric is written in bold. Our method shows the best performance on almost all settings.
opt time
Nerf(P)
3DGS(P)
TensokF
EgoNoRF
ODGS
Ballintey
13.90/0.792//00 /0.477
/0:8875/
8000008 00:410
49 /0.71 2022
0.9088 0016
BaihuiPark
16.76/0.59446 (.5587)
23.39 /B.812 70.2600
22. 22166/066855 /0.5516
24.35/ 0.55 00,2682
25.34 0,5600 /0.1409
Cathedral
15.38/0.4756,
(1.6851 (1.655
23.01 D 7885 D.225
19.35 00.563W /0.5511
23.11 00.7267 /022808
23.74/ 0.2394/
Coust
20.38 0../152////2// 000.6122/..00 00.5253
27. 0...378// / 00.2011
24.69 0.. 7026 00.1110
0..04/ /0.2320
28.75/ 0.5837 00,9966
Field
23.44/0.700 0.7/// /0.4293
29.51 (0.8392/100 70.1600
27.25 / 0.7427 / /0.4602
53 0 7843 /0.2618
29.71 0.8702/ 00.8992
10min
Nunohiki2
18.79
00.5295
D /0.2324
22.93 /0.6719 /0.530X
7052/0/3// /0.3378
(.8289 00.49%
SecraGardenn
/0.66000/////// 00,5040
25. D.88579 70.6
22.19 0,7065 /0.5205
24.57 00.7885 70.2450
00.0769
Shrinesl
05.67/0 0.7258
22. 0.744 /0.2/00226
19.52 0.5117 /0.6441
28 / 0.6374/0.3666
23.45
0,8108 /0.1560
Temple3
15:27 /0.592/ 10.611
25. 17 /D..549//D 18.183
20.83/ 7 6800 / /0.5691
24.3 31 / 7916 /0.2321
26.14/
0
/0,0068
Walongting
17.89/0,7083. 00.4451
25.88 25.80 / 0.8845 / /0,1497
22.89 / 0.769770.008 0.3998
89 / 0.84405 /0.1991
26.95/ 0.9203
/0.0626
avange
18,14/0.6158//0 /0.5544
25 61 70.8310/0.2200
22.78 / 0.6941 / 00.5089
25 70 0. 7722 /0.2660
26.34/ (.87744 00.
Balliniay
23.32/0, 03.22/0.7533// /0.379
26. D
00.278
26.55 /0.819 /0.3738
83 808 /0.1822
30,06/
/0.9220 (.0778)
BaihaiPark
18.48 00.///0..//00000 /0.449900.4999
20:81 /.27581/ /0.2808
23.13 / 7097 /0:4783
26.19 0.8390
00.1778
19.94
(1.7955 2352352
Cathedral
18.01 0.5032 00.6033
20. D.66779// 10.3682
20.23 0. 5435 00.48488
88:8// 0.5056 10.1878
23.39 0.8769 10.1081
Ccast
23.12/ 06.12/0.666//// (0.4263
27. 29 D D.84334/ D..2091
26. 0 7377 /0. 3.33668
29.28 / 0:8553 / 0.16627
29 23 0.5904/ 0.0770
Field
25.81/ 0.7274 00.3980
29. / 0.84420 / 10.1665
27.72 0.. 0.7488 00.1209
30,04/ / 0 /011776
29.15 0 0.0651
100min
Nunobiki2
21.02/ 0.63366 00.4705
20.30 0.7716 00,3622
23.63 0 6.66666/ /0. 1666
25. O 7879
/0.2115
25.75/ 0.5659 0.0996
Serretaaneen1
/0.6/30./////56 00,46-1
23. 13.43// D.
23.39
/
1011313
0.8963 0.0725
Strines
18.13/
(0.6417
21.32/ 027266/ 0.2211
20.01 0.5285 00,5717
22.83 0..3333 /0.2495
0.5270/ 0.1319
Temple3
18.69/ 0 6232/222/// 0.5500
22.22 0.78889 00.2523
22.06 0. 7011 0 0-18-10
26.
0.85188 00.1411
24. 58 0 0.0822
Waloogging
20.98 0.7328 0.3736
21 / 0.7945 / 0.2964
24.02/ 0 0.7875 / 0.3454
27.84 0.8914
0.1240
27.66/ 0.9255 0.0531
overage
20,80/ 20.80/0.6388
0.6338 00,4772
23.30 / 0.7859 / 0.2670
23.73 0,7088 / 0.4467
26.90 / 00.8349 /..14///.////001766
/0.1766
26.33/ / 0.8786 0.1023
Table E: Quantitative comparison of 3D reconstruction results on 360Roam dataset (10min/ 1 100min). The best result for each metric is written in bold. Our method shows the best performance almost all settings.
opt time
soome
Nerf(P)
3DGS(P)
TensoRF
EgoNeRF
ODGS
her
13.49/00.621 0.6218////// 0.5524
18.75/ 0..982 00.3347
15.98/0.5212
0.7717
18.24/0.57130044
19.25/ / D.6892 / 0.,078
hase
14.18/ 0166/006530 00530/ / 0.5849
20.55/ 0.69965 00.00
17.34/0. /1.35/0.5358// 008221
19.44/0.5808 / 00.855
20.93 /..677 00.3044
cafe
14.60 / 0.6645 0.4988
20.35 / 0.7503 /0.268 0.2685
17.15 / 00.5/22////// /0.7526
19.01 0..6555
20.23 / 00274444.2060
cantecn
14.00/ 0.6688 0..0026
/..6//0..66/..... 0.3301
/0.5841 /0.7301
17.60/ 0.5582
D.6655
center
15.96 7 7035 /0.4507
20 66 / / 0.3799
18.11 6290) 0..774
21.46 (0.6566666/// /0.5736
22.4 0.7477 0.3135
10min
center1
comiidr
15.77 / 0.7330 /0.4300
16.32/ / 0.7468 /0.4231
21 OT. 1 0.7329 (0.3289
18. (0.70/ / 0.7182 00.166
18.70 0.6668) 00.6500
18.52 / 6497 00.7354
21.56/ / 0.6990 / 5505
21.12/ / 0.67644 0.5133
21.73 0.7335 /0.2666
22.15 7:2221 /0.3260
innovation
14.47 0.6423 0.5007
21 07 / 0.6992 0.3332
18.84/ 00.5711: 00.5583
20.72 0.6279 0.4997
21.54/ / 0.6824 0.3312
15.33/1 76266 /0:4248
22. (T/ 80980 0..2668
6622//22000 0.66099
20.54 0.77110/
23.15 0.817/ 0,1566
library
16.01 00.6225 /0.5107
22.71 / 0.6480 003362
18.02/
0 05.554/// 0..784
21.39 / 0.5992 0.0000
22 45 D.6435 00.2644
office
15.64 / 0.6847 / /0.4650
16.47 / 0.5937 /0.5469
19.04/ 0.6108 0.7479
21.62/ / 0..224//2/// / 0.5658
18.94 / 0.6516 0.3787
average
15.07 00.6848////88 /0.4899
20. 17 5 0,7001 /003536
18.00 0.59988///88 00098/00.440 007448
20.45 0..6358 0.5334
21,08/ / 0.7066 00.3033
14.12 10.670 /0:5174
O
00.5 /0.335
19.35 0..66/2///////
0.7152/ 0.2250
base
14.73 0.6634 00,9300
20 / 0.7057 0.2756
17,40 0.55158/ / 0.7792
20.05 / 0.6214 0.4797
21.33/ 0.7098/ 0.1945
cafe
14.96/ / 0.6666 / 00,5169
19:03/ / 0.6672 0..3333
17.30 0.5397 0.7127
19.56 / 0.6912 0.3877
20.43/ / 0.7641 / 0.17783
canloen
14,20 22000000000 0.676) / 0..2066
17 51 O 0.6100 0.41884
17.47 0 5799 0.7118
18.36 0.6192 0.5255
19,06 / 0.6661 / 0.2733
center
/0.3646
6217/
6991 /0.4954
22.79 0.7629 0.2094
100min
centerl
15.27 / 0.7203 / 001557
19.91 / 0.7017 /0,3990
18.57 7 6629 0.7072
22.17 0.7152 0 00.0039
18. 15.36/ / D.653L / 003337
comidor
16.21 7 0.7333 /0.4537
18.47/ / 0.6396 00.1131
19,10 / 6638/ 0.6211
21.07 / 0.6912/0.4633
21.96 / 0.7337 0.22066
innovation
14.86/ / 64122/22 /0.5162
20 56 /.65522 20.34833
19:36 7 5671 /0.7170
21.71 / 0.6778 / 0033931
21.93 0..733 00.2089
15.87 0.7538 /0.4560
20.79/ / /.7280//////// 003316
19.14 / 6589 00.6508
22.14/ 76.7600/ /0.3263
23.39 0.8258 0.11339
library
16.06 0.6255 /0.5453
19:95
0.4528
17.77 5495
22.39 0.6295 /0.4988
22.022 0.6341 00.2099
office
15.93/ 0.6783/ / 0.4838
16.22/ / 0.5867 0.5304
19.29 / 6059 / 0.7021
22.23 / 0.6485 / 0.4960
18.44 / 0.6439/ 0..3199
average
15.26 0...0338113.0022
19.34/ / 0.050..5555 0.3837
/..11//.5895///// 0.77133
21.18 / 0.8718/ 0.4444
20.85 / 0.7111 / 0.2255
Table F: Quantitative comparison of 3D reconstruction results on OmniScenes dataset (10min/ 1 100min). The best result for each metric is written in bold. Our method shows the best performance almost all settings.
apl lime
soone
Nerf
3DGSLP)
TensoRF
EgoNoRF
ODGS
Pyebuck.Room
14.76/0.5940/
00.5417
20.56 0 0.73347/ 0.255
21 69/6//10 0.68000 6840/0.475)
21.13 00.705 /0.3934
22.78 0.8038 0.1449 -I=/-
room
/.7264 0.4518
22.48/ 0.8506 0.1985
19:81 /0.91590..003
21.68/ (.79008 00.3558
room?
15.09 /0.1156 0 /0.4563
22,90 0.8277 / 00.999
23.24 0.00/2082222 00.3425
2
/.7787 /0.3411
24.17/
0.8303 0.1392
10min
room
15.52 0.7466 0.4222
16.21 0.77811 1 /..7790
25.61 / 00.8816 / 1656
25.15 (18860 ( 1554
26.39/
24.97 / 00.885330.220 /0.2901
0.3708 (0.3012
24.66
22.79/ (13323 / 00.354
0.5434 /0.111
26,14/ 0,8906 0,1652
24.08/
1108/1.,000 0.8745 0.1384
16.75 /0.105 3835
24.43/ (1.915 1843
00.355/0.3118
20.553 /0.3416
24.40
weddingl
06.40/ 00.5529
24.18/ 0.8326/ 0.8301
23.02/ 0.7660/ 00.419
23:85 03.774 /0.3552
24.83/ 0.8347/ 0.1664
average
15.99/0.210/0.44
23.80/ (0.8444/
24.21
22.97 00.097/000000 /0.00///000444
24.42/ 0.8526 0.1399
pycbacc Rocen
15.90/ 01.00//053//0.000 /D35669
17.21/
/tt///02///////// 0.404
22.57 7 00.6933// //000//0.000 (0.4026
23.69/ / (176888 /0.2787
22.91 / (.MOGN 0.1322
room
14.64 //0.6604/0..0 /0.1550
11.49/ 0..5611 /0.5746
19.99/1
00.226/0..22
22.64 / /0.083 /0.2715
room 2
14.66/ 0.64644 /0.51388
11.68/ / (1.0005 /0.5027
23. 0.3003 /0..3157
23.53/ (1 (1.006 /0.2786
23.13
0.8108/ 0.1422
100min
16.37 0.7667 / /004280
/1..1/////////// D.4724
20.13 0.8056 0..3009
0.7712 0 3390
27.36 0.8793/ 0..3/3
26.06- / 8688 (1.3495
26. 013603 /0.2336
/ 0.87899 /0.2364
25.56 0.8847/ 0.1150
25 118870 0.1025
room
17.43 0.7996 (0.4195
17.95
D. 8088 0.3336
/0.00/000.0...0 /..2940..884
26.06/
/0.2652
44 D. 0.1383
Hold
16.64/ 0.6398 00.5541
21.85/ 0.7818 0 0.2422
24.04 7 7786/ 00.3818
25.00 / 00.0052//000 /0.2718
24.77/ 0.8331 0.1392
average
16.20 0L6838 / 00.425
18.08 / 0.7329/ / 0.3000
24.92/ 0.8205
/ /0.3140
25.15 / (.8352 0.22607
24.51/ 0.8505/ 0.1282
18

---

Table G: Quantitative comparison of 3D reconstruction results on 360VO dataset (10min/ 10miin/100minn 100min). The best result for each metric is written in bold. Our method shows the best performance almost all settings.
opt time
scene
N)
3DGS(P)
EgoNeRF
soyl
/ / 004639
17.70/0.7100 /0 /0.3946
19.6 0.66955 / 0.5622
21.46 0.7222 004555
20.99 / 0.7730 / 0.2993
15.58 / 0.5683 505555
24.31 0,7996 26.2639
20.59 /0 6448 0.6647
23.33 0.6677 0.4372
26.42 (1,82977 / 0.2069
sog2
18.04/ 0.5774
23 0.6680) 0.5442
77.46//0
3068
0..1995 / 0.0779
soy3
15.81 /0.515 70,5740
20.50/ 0..655/////5////// /0.4518
20.53 00.7789 0.6819
20.10/0.57660 /0.5903
21.72/ 0.6997 00,340
soy4
14,97 / D.5853 /0,5469
27.81 / 8646 /0.1666
21.73 0.66633 / 0..6116
24.51 7.773 /0.3755
28.31 00.991/000 0..16
10min
15:97 / 004242
19,30 0.7601 /0,3380
19.65 0.7225 004152
21.18. / 0.7555 /0,3971
21.69 / 0.8457 0.20034
soy6
/0.3436
/0.6470
0:22518
seg?
|5.46/ 0.6421 70.5045
25.42
0.0/03//////0// 0.2449
11.13 /.55444/ 0.7262
22. 72. 0.7224 /0.4725
26.36 0.8388 00.2210
seg8
14.67/ / 0.6730 004519
21.31 0.7900 0.3202
21. 10 /.../09 004910
23.94
0.7881 /0,36911
23.09 / 0.8316 / 00.2399
15.43/ 0.7510 0.4168
15.98 0.7552//////// 0.4194
20.69 /..997 0,5313
19.65 0.7883
/0.4803
21.04; 0 0.2972
average
15.71/0.6115
0..781 /0.2970
24.63
soyl)
17.16/ 0.66233/ / 0.44665
18.31 0 0.7185 /0,3880
19: ( (988999 / 0.5528
22.25
0.7466 /0.4081
21.60 69 0.8062 / 00.214
17.16/ 0 0,5439
24.59 0.8081 /0.2451
21.06 /.6505 0.6221
24.43 / 0.7352 / 3683685
28.40 0.8777 0.0961
sog2
005350
0.19441
seg3
18.01 0.5398/ 0.5867
20.0 67 6464 004399
21. 0.5850 0.6533
21.01 0.6600 /0.5366
22.59 0.7633 0.2271
segd
17.27 / 0.6053 0.54461
25 64 0.8336 / 00.2285
22. 20 0.6769 0.5307
26.46 / 0. /0,2737
30.20 / 0.9298 0.0564
100min
sog5
17.31 0.6816 0.4423
19.74/ 7682 013367
19. 7239) 0.49455
22. 0.78677 /0.3422
22,86 0.8778 0.1398
sog6
sog?
17.94/ 11.94/0.6677 0.6729 (15099
24.90/ 8025 2896
11.01 0.33349) D.7288
24. 43 0.75599 D 3903
28.07 (0,8333 0.1252
seg8
17.61 / 0.6948 00.4888
25. / 7761 / /0,3391
21. 0.3543// / 0.48111
25.29 0.8.126 0.3133
24.01 (1,8673 0,1800
sogy
0.7764
004417
average
17.78/0.6333/11
22.73 0...33/07722//00 0.3661
20.50 6885 0.5531
23.91/ 0.7591 0..3679
26.68 (1.8694
/ 0.1264
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction explain the motivation and the core idea of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
We wrote the limitation and the topics for future works in the subsection of conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Our work does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provided all information for reproducibility of the experiment.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release our code in htttp://ggithut moon//s00016/0055 to facilitate the related research.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http:///cc/guiie////eeeuubbbibii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
22

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We sufficiently describe the detailed experimental settings for preserving reprodueibilityy They are written in Section 4.1 and Appendix A.1 Also, we release the code to help other researchers reproduce work.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not include error bars since each optimization is computationally heavy compare with existing, slow method. Also, our methods and all baseline should be opti- mized for each scene individually, making the amount of experimentation be burdensome.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computational resource in Section
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics / hccppublic/thhiccuuiieeeeee.ee
Answer: [Yes]
Justification: Yes, have reviewd the Neruips Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impacts which can be triggered from this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
24

---

Answer: [NA]
Justification:
Our method does not include that can trigger such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We explained the license of dataset in Section 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
25

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not include any crowdsourcing or research related to human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do not include any crowdsourcing or research related to human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "HVxjdI03dQ", "reviewer": "Reviewer_cMFb", "summary": "This submission tackles the problem of extending 3D Gaussian Splatting (3DGS) to omnidirectional imagery. 3DGS and specifically its proposer rasterizer is limited to perspective camera. While omnidirectional images can be decomposed into perspective cameras, this typically introduces severe distortion artifacts that impacts the applicability and the quality of the reconstruction results.\n\nTo address this, the authors present a number of contributions:\n- A fast omnidirectional rasterizer (implementable in CUDA) that relies on a local affine approximation while preserving optimization and rendering speed. Each Gaussian is projected for rasterization on the tangent plane of a sphere centered at the camera origin, thereby decreasing the amount of distortion to a minimum.\n- The pruning and spiltting heuristics of 3DGS are adapted to account for the omnidirectional projection.\n- An array of qualitative and quantitative evaluations on egocentric (OmniBlender, Ricoh360, OmniPhotos) and roaming (360Room, OmniScenes, 360VO) datasets is shared with comparison against 3DGS and NeRF-based baselines (NeRF, TensoRF, EgoNeRF).", "questions": "- Please clarify how alpha-blending of Gaussians is implemented with per-Gaussian rasterization (see Weaknesses above). While the presented results otherwise do look good, this is a critical gap in the presentation of the approach that prevents me from proposing a higher rating.\n\n- Could the authors indicate how ODGS achieves to be significantly (more than 3x) faster than 3DGS applied on perspective images?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- With more and more image contents coming from egocentric cameras, identifying and proposing ways to address the limitations of an established approach like 3DGS to work on this new type of contents is quite relevant to the research community.\n\n- The proposed changes to cope with omnidirectional imagery are very simple: essentially projecting Gaussians to use a per-Gaussian rasterization frame that limits distortion while still allowing an efficient implementation on graphics hardware.\n\n- The quantitative and qualitative evaluations do a good job of demonstrating the benefits of introducing a custom omnidirectional rasterizer. Performance is measured against relevant baselines (3DGS of course but also NeRF-based ones, including one targeting egocentric contents: EgoNeRF) across multiple datasets. The approach of the author is shown to significantly outperforms in Table 1 in terms of quality (generally) all other approaches (using the typical PSNR, SSIM and LPIPS metrics) while actually being by far the fastest one. Figure 3 and 4 also share convincing qualitative results with significant artifacts in other approaches that ODGS does not generate.", "weaknesses": "- I believe there is a major gap in the current presentation of the approach: how to implement alpha blending of Gaussians appears to be missing. It is in particular unclear to me in which common coordinate frame Gaussians are being composited on top of another to produce a final image. As this is, in my opinion, essential to understand (and reproduce) the approach, until resolved, this gap is dragging the proposed rating of the paper down despite the strengths above.\n\n- Decomposing an omnidirectional field-of-view into six perspective cameras to allow comparison against approaches that only work on perspective imagery makes sense. Such comparison could however be improved and pushed much further: how does the comparison hold with\n  - the omnidirectional field of view being decomposed into gradually more perspective cameras,\n  - the rasterizer properly handling the different perspective coordinate frames (compositing all the relevant Gaussians overlapping in the current perspective field-of-view)? \nThe areas of introduced distortion due to the rectification could be visualized and super-imposed on the omnidirectional images of the qualitative comparisons to better understand and correlate with observed artifacts.\n\n- The abstract and introduction indicates some adjustments being applied to the densification heuristics, which are not apparently covered in the paper (not in section 3 or 4) apart from a mention of limiting the maximum size of a Gaussian.\n\n- Some reference (and possibly comparison) to (very) related work appears to be missing: [OmniGS: Omnidirectional Gaussian Splatting for Fast Radiance Field Reconstruction using Omnidirectional Images](https://arxiv.org/abs/2404.03202) (April 2024)\n\n- Some minor typos to correct:\n  - l.111 available To -> available to\n  - l.285 often misses -> it often misses\n  - l.289 ODGS overcome -> ODGS overcomes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1) How to implement alpha blending?**\n**A) We conducted our work based on the 3DGS framework, utilizing the same tile-based rasterization and alpha-blending pipeline.**\nOur work suggests how to render with an omnidirectional (omni in short) camera model instead of a perspective (persp in short) camera model, and the remaining methods are identical to those of 3DGS.\nFor clarity, the rasterization process is as follows:\n1. Projecting 3D Gaussians to 2D Gaussians in the planar pixel space (as described in Section 3.2).\n2. Tile-based culling against the spherical shell and sorting by depth.\n3. For a given pixel, accumulating color and alpha values by traversing the list front-to-back until we reach a target saturation of alpha in the pixel. (alpha-blending)\n$$\nC = \\sum_{j\\in\\textit{N}} {c_j \\alpha_j T_j}, \\quad {T_j = \\prod_{k=1}^{j-1} (1-\\alpha_k)},\n$$\n$C$ is the pixel color, $c_j$s are the colors of splats, and $\\alpha$ is computed by multiplying the learned opacity of Gaussian with the power of rasterized Gaussian in the pixel (calculated using Eq. (9)).\n\nWe omitted the details of the tile-based culling and alpha-blending process because those are not the contributions of our work.\nHowever, we acknowledge the concern that the omission may make the paper appear incomplete to some readers and reviewers.\nWe will include this information in the final version of the paper to enhance its completeness and clarity. \n\n\n### **Q2.1) What happens if we increase the number of persp cameras**\n**A) While using more views can yield slightly better results than the 6-view setup, it still falls short of the performance achieved by ODGS.**\nCubemap projection is a well-known method for converting an omni image to six persp images, where each corresponds to the face of a cube, and it is widely used in many studies involving 360 cameras [A,B,C].\nHowever, as suggested by cMFb, we also believe it would be an interesting experiment to observe the trends when using more persp cameras for optimization.\nWhile the standard approach uses six perspective cameras, we optimized the model by adding 12 perspective cameras (18 in total)  facing the edges of the cube.  \nTable R1 shows the performance of optimized results according to the number of persp viewpoints.\nWhen using the 18-views, the performance is comparable to the 6-views at the 10-minute mark but surpasses the 6-view results at the 100-minute mark.\nFirst, the increased number of images for training keeps the model from sufficiently learning from all views in the early stages (10 minutes), resulting in slightly lower performance.\nHowever, after sufficient optimization time (100 minutes), the additional views allow for further optimization, leading to improved results.\nStill, ODGS shows the highest performance in most metrics, even considering 3DGS using 18-views.\n\n### **Q2.2) Is the rasterizer properly handling the different persp coordinate frames?**\n**A) In the case of the 6-view cubemap, artifacts can occur at the edges of the cube due to the rasterizing method used by 3DGS.**\nSince 3DGS omits the 3D Gaussians at image boundaries for stable optimization, discontinuities may occur at the edges of the cube, as illustrated in Figure R1.(c).\n(The orange dashed lines in Figure R1.(b) represent the edges of the cubemap after being combined into one omni image.)\nThe image optimized with 18-view persp images has fewer artifacts, as shown in Figure R1.(d).\nHowever, artifacts still appear even at points other than the cube's edges, resulting in less favorable outcomes than ODGS.\n\n### **Q2.3) How ODGS runs more than three times faster than 3DGS applied on persp images?**\n**A) A majority of time is spent stitching the six persp images into the omni image.**\nSince the total resolution of an omni image (2000 x 1000) is similar to six persp images (512 x 512 x 6), we empirically measured that the time for generating six perspective images is similar to generating one omni image.\nHowever, producing the final omni image from the six persp images takes a large proportion of of inference time (about 80\\%) since every persp image should be non-linearly warped to compose the final omni image.\n\n\n\n### **Q3) The details of the densification heuristics**\n**A) We acknowledge that the concept might be challenging to fully understand from the paper alone, so we will include a detailed explanation for clarity.**\nAs represented by $\\mathbf{C}_o$ in Eq. (7), Gaussians of the same size in 3D space are rasterized to different sizes in an omnidirectional image depending on the angle at which they are located.\nConsequently, from a pre-defined size threshold for densification, we dynamically change the threshold according to the vertical location of the Gaussian.\nAs depicted in Figure R3, this approach effectively reduces artifacts when representing lanes of the road.\n\n\n### **Q4) Missing a related work: OmniGS**\n**A) We now recognize OmniGS as a concurrent work.**\nWe thoroughly reviewed the suggested paper, OmniGS, and noted that they employ a similar approach.\nUnfortunately, their work is only available on arXiv; no code or implementation details have been provided.\nWe would have liked to compare their method with ours, but we will have to settle for mentioning it as a concurrent work in our references.\n\n### Reference\n[A] H. Jang et al., \"Egocentric scene reconstruction from an omnidirectional video,\" SIGGRAPH 2022.  \n[B] M. Gond et al., \"LFSphereNet: Real-time spherical light field reconstruction from a single omnidirectional image,\" SIGGRAPH 2023.  \n[C] F. Wang et al., \"Bifuse: Monocular 360 depth estimation via bi-projection fusion,\" CVPR 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0UFLbPmqqK", "reviewer": "Reviewer_iEY4", "summary": "The paper introduces a novel approach for 3D scene reconstruction from 360 omnidirectional imagery that the authors make compatible with a 2D Gaussian Splatting representation [x].\n\nThe proposed method is evaluated on 3 public datasets against 4 Radiance Field variants including NeRF derivatives and 3D Gaussian Splatting [25].\n\n\n[x]: Huang, B., Yu, Z., Chen, A., Geiger, A., & Gao, S. (2024). 2d gaussian splatting for geometrically accurate radiance fields. Siggraph 2024.", "questions": "I do not have more questions or suggestions than the ones underlying the aforementioned weaknesses.", "limitations": "The authors provide one dedicated paragraph that reasonably addresses such considerations.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "### + Practical usefullness of the considered problem.\nThe paper addresses a narrow yet valuable problem in the scope of data acquisition from imaging-equipped vehicles in urban areas.\n\n### + Structure, and Organization of the Contents.\nEach dedicated section of the paper is properly balanced. The use of text real-estate is fair.\n\n### + Amount of discussions on the experimental evaluation is relatively good.\n\n### + Fair results of the proposed contribution, overall.\n\n### + Level of implementation details, in particular in the appendix is good.", "weaknesses": "### - (1). Writing, and consistency between key sections and the contents, and overall maturity of the presentation.\n\n#### (i) Writing and readability.\nIn its current state, the paper is in a relatively dense state, with grammar issues and typos, even in key sections.\nE.g., typo in the title of the paper: \n\n\"ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian **Splattings**\"\n-> \n\"ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian Splatting\"\n\nThe writing of the abstract section for example is very dense and hard to understand.\n\n#### (ii) Text vs. Contents - Inconsistencies.\nTitle and paper suggest the proposed paper builds on the 3D Gaussian Splatting framework (eg, title, l7), while it in fact uses 2D Gaussian Splatting (Eq (9)).\n\n### - (2). Comparative evaluation lacks depth.\n\nIn particular, additional comparative and standard baselines could have been envisioned to better help the reader unstand the relative positioning of the proposed conceptual contribution, perfromance-wise.\n\nE.g., COLMAP (standard SfM), Neus, NeusFacto, NeuralAngelo:\n\n- Zehao Yu, Anpei Chen, Bozidar Antic, Songyou Peng Peng, Apratim Bhattacharyya, Michael Niemeyer, Siyu Tang, Torsten Sattler, and Andreas Geiger. Sdfstudio: A unified framework for surface reconstruction, 2022\n\n- Schonberger, J. L., & Frahm, J. M. (2016). Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4104-4113).\n\n- Schönberger, J. L., Zheng, E., Frahm, J. M., & Pollefeys, M. (2016). Pixelwise view selection for unstructured multi-view stereo. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 (pp. 501-518). Springer International Publishing.\n\n- Li, Z., Müller, T., Evans, A., Taylor, R. H., Unberath, M., Liu, M. Y., & Lin, C. H. (2023). Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8456-8465).\n\n- Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., & Wang, W. (2021). NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. Advances in Neural Information Processing Systems, 34, 27171-27183.\n\nIn particular, the unified codebase makes it very easy to produce results for the aforementioned baselines. \n\n### - Other minor issues and suggestions.\n- Missing reference (2DGS):\nHuang, B., Yu, Z., Chen, A., Geiger, A., & Gao, S. (2024). 2d gaussian splatting for geometrically accurate radiance fields. Siggraph 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1) Inconsistencies between key sections and the contents**\n**A) We strongly contend that the iEY4's review is based on a substantial misapprehension of our study, thus leading to erroneous conclusions.**\nOur work is based on 3DGS [25] and is not related to 2DGS at all.\nEq. (9) describes a density function of a 3D Gaussian projected on the image plane, modeled as a 2D Gaussian distribution in the image plane through local affine approximation; the actual 3D space consists of 3D Gaussians.\n(Please refer to the paragraph in 3DGS [25] paper, Section 4, Equation 5)\nThis is completely different from the 2DGS work that originally composes a 3D space as a set of disk-shaped 2D Gaussian primitives.\n\n\n### **Q2) Comparison with other methods**\n**A) The contributions of the listed papers are outside the scope of the problem we are trying to tackle.**\nFirst, Sdfstudio, Neuralangelo, NeuS, and NeusFacto aim at neural 'surface' reconstruction, which is different from the problem we are addressing.\nThe surface normal maps generated by these models do not contain texture information and, therefore, cannot be directly compared with high-quality rendered images obtained from ODGS.\nIn addition, the datasets used in these works for training and inference do not contain omnidirectional images.\n\nMeanwhile, we excluded classic 3D reconstructions using Structure-from-Motion (SfM) and Multi-view Stereo (MVS) from the main table because they produce significantly inferior results compared to the recent reconstruction methods.\nHere, we present these results in Table R2 for better understanding.\nThe SfM results are obtained using the OpenMVG library, then the MVS results are obtained from the OpenMVS library, respectively.\n(We used OpenMVG instead of COLMAP because COLMAP currently does not support camera pose estimation from omnidirectional images.)\nBoth methods fail to create a complete point cloud, resulting in blank areas in the projected image, as observed in the samples shown in Figure R2.\nThe valid area ratio in the table indicates the proportion of pixels among images that receive 3D information and have valid pixel values.\nThe results of SfM shows a valid area ratio under 5\\%, which implies that most of the regions remain empty.\nAlthough the high accuracy of the estimated points shows high PSNRs among the valid area, the number of matched points is small, and the PSNRs of the whole region are measured to be less than 5dB.\nApplying MVS to the results of SfM increases the number of generated points, but still, at least 40\\% of pixels are not filled after projection.\nIn addition, by comparing the PSNR measured in the valid area, we can see that the accuracy of the points created by applying MVS has decreased.\nODGS, in contrast, shows consistently much higher PSNR than those methods while maintaining a 100\\% valid area ratio for all datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9aeMq9tnpr", "reviewer": "Reviewer_oCBJ", "summary": "This paper implements a rasterization module for 3D Gaussian Splatting (3DGS) for omnidirectional images. The rasterizer assumes local affine approximation and projects Gaussian primitives to the unit sphere centered by the camera position, which may be reasonable for relatively small Gaussians. Based on the CUDA implementation, experiments show faster and more accurate rendering results compared with the omnidirectional extension of NeRF (e.g., EgoNeRF).", "questions": "Are there any specific technical novelty we (readers and reviewers) should care about?\nI would rather like to support the usefulness and technical correctness of the paper, but I think others may not.", "limitations": "I did not find notable negative social impacts.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "## First attempt to omnidirectional 3DGS\nThis paper would be the first attempt to extend 3DGS to omnidirectional images in general (360-GS [2] is available but for the specific purpose of room modeling requiring layout information, and OmniGS [a] is considered to be a concurrent work uploaded just before NeurIPS deadline.)\n\n[a] Li, Longwei, et al. \"OmniGS: Omnidirectional Gaussian Splatting for Fast Radiance Field Reconstruction using Omnidirectional Images.\" arXiv preprint arXiv:2404.03202 (2024).\n\n## Technical correctness\nThe method seems to be adequate. Under affine approximation, this paper correctly formulated the projection and implemented it with CUDA. This is a good engineering effort, and many people will benefit from it. It is good news to know that the omnidirectional version of 3DGS generally achieves better rendering than omnidirectional NeRF methods.", "weaknesses": "## Novelty\nI am struggling to evaluate the technical novelty of this paper. This paper implements a useful tool, while the paper's contribution is rather in its engineering effort, and the scientific (or technical) novelty may be limited.\n\nI understand it is quite important to release those useful tools to the community, while the idea of extending 3DGS to omnidirectional images is somewhat natural. Locally affine approximation may be straightforward if someone may extend 3DGS to omnidirectional images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q) Are there any specific technical novelty we (readers and reviewers) should care about?**\n\n**A) We hope that our work will not be dismissed as merely an engineering effort.**\nThe idea of applying a local affine approximation to a sphere may seem simple at first glance. However, aside from concurrent work, this approach is being attempted for the first time. We hope that the process of developing and validating this idea will not be dismissed as merely an engineering effort.\nWe want to highlight the following unique contributions and technical innovations:\n\n* **New Omnidirectional Rasterizer**: We introduce a novel rasterization technique specifically designed for omnidirectional images, which is not a straightforward extension of existing 3DGS methods and provides a more accurate and efficient representation of 3D scenes captured from 360-degree cameras.\n\n* **Advanced Densification and Splitting Policy**: ODGS employs a sophisticated policy for managing the densification and splitting of Gaussians in the omnidirectional domain. \nOur work is not just an engineering effort, but also a significant technical consideration that enhances the quality and efficiency of the 3D reconstruction. This is a key aspect that we believe sets our work apart.\n\n* **Comprehensive Evaluation Across Diverse Datasets**: Unlike many other approaches, the paper thoroughly tests various datasets, covering both egocentric and roaming scenes. \nThis comprehensive evaluation showcases the robustness and versatility of ODGS.\n\nFortunately, all other reviewers have acknowledged the strengths of our work, and none have raised concerns about its novelty. Specifically:\n* Reviewer cMFb noted that our work is \"quite relevant to the research community, simple, and efficient.\"\n* Reviewer iEY4 highlighted as \"practically useful and valuable.\"\n* Reviewer sZSB described it as \"simple but effective, and inspiring.\"\n\nTherefore, while we can understand the concerns raised by reviewer oCBJ to some extent, we hope that our work will be evaluated positively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FHl08OnDGA", "reviewer": "Reviewer_sZSB", "summary": "This paper proposes a method of rasterizing and optimizing 3DGS for omni directional cameras. Built on top of the original 3DGS perspective camera rendering, this paper approximates the rendering of each 3D Gaussian as a perspective camera pointing towards each Gaussian. By limiting the size of individual Gaussian, this effectively approximates the omni-directional camera model applied to 3D Gaussians.", "questions": "I am very interested in one thing and hope the author can provide me with some answers: can a 3DGS model trained with Omni-directional images with this method be directly used to render using perspective camera, while preserving the rendering quality without fine-tuning?\n\nThe reason I think this is important in practice is that omni cameras are usually useful in data capturing stage, but not the inference stage. It is ideal to train with panorama images but render with perspective cameras during application. I think adding the relevant results in the paper can further strengthen the usefulness of this paper.", "limitations": "There is no societal impact involved.\n\nOne of the main technical limitations is the size of the Gaussian causing approximation error. This paper chooses to limit it but there might be a better way of handling it such as splitting the Gaussians during approximation to handle more complex scenarios efficiently.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Motivation**\n* Unlike NeRF which operates on ray-level, 3DGS by its nature is limited by the perspective projection applied to 3D Gaussians and is not easily migrated to different camera models. Extending the 3DGS method to omni directional camera model is useful in many real-world applications due to the data collection efficiency of omni-directional camera. \n\n**Method**\n* The method proposed by this paper is simple but effective. Without introducing very complicated omni-directional camera models to project the 3D Gaussians, this paper proposes to approximate the projection with individual perspective cameras pointing towards each Gaussian. This effectively approximates the rasterization without changing too much on the pipeline. \n* The method proposed by this paper is inspiring, not only in the context of omni-directional camera, but arbitrary camera models. Most cameras can be approximated with a similar manner to handle 3D Gaussians using local tangent perspective cameras.\n* The proposed method is quite general and is not limited to the type of scene, compared to the previous 360-GS.\n\n**Experiments**\n* The quantitative rendering quality is consistently higher than the previous NeRF counterparts and the wrapped perspective version of NeRF and 3DGS. The qualitative results also demonstrates considerable improvements.\n* The high rendering speed of 3DGS is well preserved to a large extent.", "weaknesses": "**Method**\n* The error of the approximation depends on the size of the Gaussian (or the size of the projected 2D Gaussian). It is mentioned in the paper that the maximum size of the Gaussian is limited to minimize the error, but I could not find the details about this limit. I think more details should be included. A hard limit in 3D might not be the best way of performing this limit.\n\n**Experiment**\n* Although the flexibility of this method is preferred over 360-GS, it is better to include the performance comparison between these two works as they are very similar.\n\n**Typo**\n* On Line 138 \"distribution of after projection\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1) The details about the maximum size of the Gaussian**\n**A) We applied different maximum size thresholds depending on the elevation angles.**\nLooking at Eq. (7), in omnidirectional projection, the Gaussian located in the polar region is rasterized to a wider area of the image than the equatorial region.\nReflecting these characteristics, we applied the split size limit of Gaussian differently depending on the upper and lower angles at which Gaussian is located.\nReflecting these characteristics, we applied the size limit of Gaussian differently depending on the angle at which Gaussian is located.\nSpecifically, the larger the absolute value of the angle, the larger the size threshold becomes, and it becomes the minimum in the equatorial region.\nWhen this size exceeds the threshold, Gaussian is split into two smaller Gaussians and optimized separately.  \nIn Figure R3, we qualitatively compare the images created with and without using the above method.\nWe understand that it could be difficult to fully grasp this concept based solely on the contents written in the paper. \nTherefore, we will provide a detailed explanation to make it easier to understand.\n\n### **Q2) Performance comparison with 360-GS**\n**A) We attempted to compare with 360-GS before submission, but were unable to do so.**\nSince 360-GS is currently only available on arXiv and the official code implementation has not been made public, we endeavored to reproduce 360-GS by ourselves.\nUsing the code we reproduced, however, Gaussian splats are hardly optimized, producing ludicrous results.\nAfter a thorough analysis of the issue, we noticed that there was an error in the equation calculating the Jacobian matrix $J$ in 360-GS.\nSpecifically, the signs of the diagonal terms of the matrix should be -, in Eq. (7) of 360-GS paper [2].\nThis could be due to a differentiation error or a mistake in the description within the paper.\nWhile our reproduction might have been incorrect, we believe it would not be appropriate to include this comparison in the main table under this situation.\nIf the official code for 360-GS becomes available, we could include the comparison in the main table. \n\n### **Q3) Is it feasible to render using a perspective camera without fine-tuning?**\n**A) ODGS delivers high-quality perspective camera rendering results without any additional fine-tuning.**\n\nOur method is fundamentally based on projecting onto the tangent plane of a sphere, which is conceptually compatible to the perspective rendering of each Gaussian.\nAs a result, the Gaussian splats trained with ODGS produce high-quality perspective images when rasterized with a pinhole camera model without fine-tuning.\nSome sample images rendered using a perspective camera model are shown in Figure R4.\n3DGS(P6) and 3DGS(P18) were optimized using perspective images and rendered with a perspective camera, while ODGS was optimized using omnidirectional images and rendered with a perspective camera.\nDespite the difference between the camera model used during optimization and the one used during inference, ODGS demonstrates significantly sharper and superior results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Statements to All Reviewers
We appreciate all reviewers for their valuable comments.
We have thoroughly examined the reviews and hope to address all questions and misunderstandings through this rebuttal.
We have responded to each reviewer's queries in a question-and-answer format.
We will also fix the typos and polish the writing to enhance the clarity of the paper in the updated manuscript.

Meta-review: The submitted paper has undergone a thorough review process by four reviewers. After careful consideration of the reviewers' feedback and ratings, I believe this paper should be accepted for publication at the conference.

The paper initially received some concerns from two reviewers about the novelty of the approach. However, the authors were able to effectively address these concerns in their rebuttal, leading the initially skeptical reviewers to increase ratings.
Additionally, the missing references of the works [1, 2, 3] in Arxiv are recognized as concurrent works. It is encouraged but not compulsory to conduct evaluation and discussion with these concurrent works. Although one reviewer remains negative, the main concerns and missing baselines are not particularly relevant to the proposed work and were therefore not weighted heavily in the final decision.

This paper provides a practical and effective solution to omnidirectional scene reconstruction using Gaussian splatting which contributes to an important and growing research area. 

[1] Huang, Letian, et al. "On the Error Analysis of 3D Gaussian Splatting and an Optimal Projection Strategy." arXiv preprint arXiv:2402.00752 (2024). ECCV 24
[2] Li, Longwei, et al. "OmniGS: Omnidirectional Gaussian Splatting for Fast Radiance Field Reconstruction using Omnidirectional Images." arXiv preprint arXiv:2404.03202 (2024).
[3] Bai, Jiayang, et al. "360-GS: Layout-guided Panoramic Gaussian Splatting For Indoor Roaming." arXiv preprint arXiv:2402.00763 (2024).
Decision: Accept (poster)