Omnidirectional 3D Scene Reconstruction from Single Image
Authors: Ren Yang
Jiahao Li
Yan Lu
Venue: NeurIPS 2025 poster ()
Keywords: 3D Reconstruction
Diffusion
Gaussian Splatting
Primary area: deep_learning

Abstract: Reconstruction of 3D scenes from a single image is a crucial step towards enabling next-generation AI-powered immersive experiences. However, existing diffusion-based methods often struggle with reconstructing omnidirectional scenes due to geometric distortions and inconsistencies across the generated novel views, hindering accurate 3D recovery. To overcome this challenge, we propose Omni3D, an approach designed to enhance the geometric fidelity of diffusion-generated views for robust omnidirectional reconstruction. Our method leverages priors from pose estimation techniques, such as MASt3R, to iteratively refine both the generated novel views and their estimated camera poses. Specifically, we minimize the 3D reprojection errors between paired views to optimize the generated images, and simultaneously, correct the pose estimation based on the refined views. This synergistic optimization process yields geometrically consistent views and accurate poses, which are then used to build an explicit 3D Gaussian Splatting representation capable of omnidirectional rendering. Experimental results validate the effectiveness of Omni3D, demonstrating significantly advanced 3D reconstruction quality in the omnidirectional space, compared to previous state-of-the-art methods. Project page: https://omni3d-neurips.github.io.

Omnidirectional 3D Scene Reconstruction from Single Image
Ren Yang Microsoft Research Asia yangrenOmi crosoft. com
Jiahao Li
Yan Lu
Microsoft Research Asia jiahao@mi crosoff. com a
Microsoft Research Asia yanlu@microsoft. com
Abstract
Reconstruction 3D scenes from a single image a crucial step towards en- abling next-generation Al-powered immersive experiences. However, existing diffusion- based methods often struggle with reconstructing omnidirectional scenes due to geometric distortions and inconsistencies across the generated novel views, hindering accurate 3D recovery. To overcome this challenge, we propose Omni3D, approach designed enhance the geometric fidelity of diffusiin--geeeattee views for robust omnidirectional reconstruction. Our method leverages priors from pose estimation techniques, such MAS13R, iteratively refine both the generated novel views and their estimated camera poses. Specifically, we minimize the 3D reprojection errors between paired views optimize the generated images, and simultancously, correct the pose estimation based on the refined views. This synergistic optimization process yields geometrically consistent views and accurate poses, which are then used to build an explicit 3D Gaussian Splatting represen- tation capable of omnidirectional rendering. Experimental results validate the effectiveness of Omni3D, demonstrating significantly advanced 3D reconstruction quality the omnidirectional space, compared to previous state-of-the methods. Project page: https: omn 3d neur ips github....
Introduction
3D scene reconstruction from 2D image plays significant role in the future development of computer vision technologies, paving the way for the next era of AI immersive media. However, this task inherently ill- ill-posed due to the significant geometric ambiguity and limited information available from single input image. While recent progress, particularly with diffusion-bars based models 2
48 33),
has shown promise in object-level 3D reconstruction [22 4 ,nd also facilitates scene-level Novel View Synthesis (NVS)
45 10 51 and scene reconstruction 20 3 with Gaussian Splatting (3DGS) [12] However, the existing diffusion-based methods often encounter significant hurdles when applied to omnidirectional 3DGS reconstruction of real-world scenes. The main obstacle is that the produced novel views tend suffer from geometric distortions and critical inconsistencies, especially when synthesizing views far from the original input perspective. Standard models may misinterpret context in omnidirectional images due to non-uniform structures and optical properties different from perspective images. These inaccuracies in geometry and appearance across generated views fundamentally hinder the ability to recover coherent and accurate 3DGS representation of the full omnidirectional scene.
To address this challenge, we introduce Omni3D, novel approach specifically designed to tackle the geometric and content consistency issues inherent in diffusion-basee view generation for omni- directional 3D scene reconstruction. Our core idea is to explicitly incorporate and refine geometric constraints throughout the view generation process. We achieve this by leveraging strong priors obtained from state-of-the-ar pose estimation techniques, such as MAS13 [7 to iteratively im-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Viewpoint in the front hemisphere
Viewpoint in the back hemisphere
Input image
Figure 1: Example of Omni3D for omnidirectional 3D scene reconstruction from single image.
prove both the generated novel views and their corresponding camera pose estimations. Specifically, Omni3D employs a synergistic Pose- View Optimization (PVO) strategy. It minimizes 3D reprojection between pairs of generated views to progressively refine the image content, ensuring better geometric alignment. Simultaneously, the estimated camera poses for these views are corrected based the feedback from the refined views. This iterative process of mutual refinement between view content and pose estimation yields set of geometrically consistent novel views with highly accurate camera parameters. These consistent views and poses serve as high-quality input for building an explicit 3D scene representation using 3D Gaussian Splatting, enabling flexible rendering with advanced quality across the entire omnidirectional space. Figure [ illusttrae an example result, demonstrating omnidirectional 3D scene reconstruction from single image by the proposed Omni3D approach.
Our primary contributions are concluded as:
We propose novel method. Omni3D, that significantly improves the geometric and content consistency of Inffusionn-eneraate novel views for single-image omnidirectional 3D scene reconstruction with Gaussian Splatting.
We propose synergistic Pose-Vis View Optimization (PVO) process that leverages pose estima- tion priors to iteratively refine both generated view content and camera poses by minimizing 3D reprojection errors.
We demonstrate state-of-the-an art performance in omnidirectional 3D scene reconstruction from a single image, showing substantial improvements on rendering quality across a wide range of view angles compared to previous methods.
2 Related Works
2.1 Traditional View Synthesis
Early methods for Novel View Synthesis (NVS) often relied on geometric primitives or layered representations. Multiplane Images (MPI) [4 36 39,
15 49 represent a scene using multiple semi-transparent planes at different depths. Methods like SinMPI (26) and AdaMPI [ extended MPIs for single-image NVS, sometimes incorporating diffusion models to hallucinate out-of-view content. However, MPI-based methods can struggle with representing complex non-planar geometry and may exhibit flatness artifacts. Another line of work [4 31 29 35 utilizes depth-basee warping,
47

---

where an estimated depth map is used to project the input view to novel viewpoint, followed by inpainting occluded regions These methods are highly sensitive to depth estimation errors and can produce artifacts near object boundaries or inconsistent content in inpainted areas. While effective for limited view changes, these traditional techniques often struggle with the large baselines and distortions inherent in omnidirectional reconstruction.
2.2
Generative Image-3D Reconstruction
Pre-trained text-to-image (T2I) diffusion models possess rich semantic and structural priors. Several works [30] 27 aim to distill these priors for 3D generation and NVS. Some approaches [38 25
18
optimize 3D representations using scores distilled from 2D diffusion model as supervision. fine-tune 2D diffusion models be conditioned camera viewpoints, enabling them to generate novel views directly. While powerful, these methods often focus on object or simple scene reconstruction and may lack generalization to complex and large-scale scenes. Furthermore, controlling camera pose accurately can be challenging, as poses are treated as high-level prompts rather than precise geometric inputs. Besides, Chung al. [5] and Yu al. [50] employ diffusion- based inpainting models to lift 2D images to 3D scenes.
42
Others
21
33
More recently, Latent Video Diffusion Models (LVDMs) [2] trained large-scale video datasets have emerged as promising source of 3D priors. implicitly learning about motion, temporal coherence, 41) leverage LVDMs for single-image-to-3D
and scene dynamics. Several works 7 14 reconstruction at the object level, and meanwhile, 45 51 focus on scene-level 3D NVS and reconstruction. However, the inherent stochasticity and iterative denoising process in diffusion models may introduce geometric distortions and inconsistencies across generated views, hindering accurate 3D reconstruction in the scenario of large angle changes. The latest method LiftImage3D [3] employs distortion-awarr Gaussian representations to mitigate view inconsistencies, but nevertheless it still reconstructs 3D in limited angles from the single input image, instead of the challenging task of omnidirectional 3D scene reconstruction.
23
24
2.3 Pose Estimation
In early years, Ummenhofer et al. [40] and Zhou et al. [53] proposed estimating depthmaps and relative camera pose given the grountruth camera intrinsic parameters. Later, the DUSt3R [43 method was proposed in 2024, which represents a sigmificant departure from traditional pipelines. It is designed to perform camera pose estimation from unconstrained image collections, requiring no prior knowledge of camera intrinsics. In the same time of pose etimation, the camera intrinsics can also be calculated. In the following, MASt3R (17) is built upon the backbone of DUSt3R with focus on local feature matching for improving image matching accuracy.
3 Method
3.1
Overall Framework
The overall framework of the proposed Omni3D is illustrated in Figure outlining a multi-stage approach to achieve omnidirectional 3D reconstruction from single image.
In Stage I, beginning with a single input image (shown as ). we employ a Multi- View Diffusion (MVD) model to synthesize an initial set of novel views These views are generated along four cardinal orbits (left, right, up, and down) to broadly cover the frontal hemisphere relative to the input. Subsequently, the proposed Pose- View Optimization (PVO) module is applied to the generated views. The PVO module collaboratively refines the estimated camera poses and corrects the generated view content, thereby mitigating geometric distortions and inconsistencies inherent in the initial MVD outputs. During the PVO process, the camera intrinsic parameters are also calculated [3]
Stage II focuses on expanding the view coverage laterally. Key views from the periphery of the frontal hemisphere generated Stage (e.g.. the leftmost and rightmost views, depicted as 2)) serve as new conditional inputs for the MVD model. This step synthesizes additional novel views (22) that extend into the left and right hemispheres. These newly generated views then undergo the PVO optimization to ensure their geometric accuracy and consistency.
3

---

Stage
Stage II
Stage III
M//
mopin
Stage IV
6099
inagl
Updat
Updat
Lpdars
Pene Pene-tew Correction (PVO)
Pore Vis Cometiin
Pose-Viee Carredion (PVO)
Omnidirectioal rendertig
Progrendie bainng
Progrestive Panne
Progeanker
Fregrandue
Paining
Poining
3D Splatting
Falrweee bacsthv PVD
Faindles berathe PVO
Pairvaiio nevotiue PVD
Palrwise Investin PVD
(e) Overall framework
M, Ro--slll
Horoggapte
W 2D vargtig
10
additiun
troodar
Docodar
Head
Poitroop
Encodar
Dococar
Heal
Updite when
Bg
MAROR frooun but alfbrestabb)
rEDrONSUce
With gradianta
vewe To
Without gradieers
computetior
Co
Mo
||20
2 lo
(b) Pairwise
Figure 2: (a) The overall framework of the proposed Omni3D approach. Omni3D generates novel views in omnidirectional space across three stages. In each stage, the generated views optimized by the Pose-View Optimization (PVO) module, which progressively applies a pairwise iterative PVO process refine both view contents and pose estimation. This mitigates geometrical distortions and inconsistencies in the generated novel views, facilitating the final stage to represent the omnidirec- tional 3D scene with 3DGS. (b) The proposed pairwise iterative Pose-View Optimization (PVO) module. For each view pair, this module first estimates the initial camera poses and intrinsics, and then overfits lightweight network for 3D and 2D optimization of generated view by minimizing 3D reprojection errors between reference and generated views. Camera poses and camera intrinsics are then updated once the optimization loss converges. This process repeats iteratively for synergistically refining both the views and their corresponding poses.
Stage III addresses the back hemisphere to achieve fully omnidirectional coverage. In this stage, the backmost view (i2) is used to condition the MVD model for the synthesis of the final set of novel views (2)) required complete the omnidirectional scene representation. As with the previous stages, these views are meticulously processed by the PVO module. Upon completion this stage, a comprehensive set of geometrically consistent and pose-accurate omnidirectional views is obtained.
Finally, in Stage IV, this complete collection of PVO-optimized views, along with their refined camera poses and intrinsic parameters, is leveraged to reconstruct the 3D scene. Specifically, we train a 3D Gaussian Splatting (3DGS) model using these views. The resulting 3DGS model facilitates freely rendering of novel views in omnidirectional angles.

---

3.2 Multi-View Diffusion (MVD)
For the default implementation of Omni3D, we follow [37 to employ LoRA-tunes CogVideoX [48] as MVD models. These models are configured to generate 48 novel views per orbit, in addition to the original input view. The MVD models were trained on carefully selected samples from the DL3DV-10K dataset [19], with strict separation maintained between the training and test sets. It is worth pointing out that Omni3D's effectiveness is not contingent on the choice of MVD model, and has the generalization capabilities across different MVD backbones, which is validated in Section 4.3
3.3 Pose- View Optimization (PVO)
This section introduces our Pose- Pose-View Optimization (PVO) module, a core component of the Omni3D method. The PVO module employs progressive pairing scheme systematically process sequences of generated novel views. Coupled with this is an iterative optimization process designed to syner- gistically refine both the content of these views and their corresponding camera poses. This process yields views with enhanced geometric consistency and more accurate pose information, playing a critical role in facilitating robust omnidirectional 3D scene reconstruction with 3DGS.
3.3.1 Progressive Pairing
The progressive optimization process, illustrated in Figure 2 (a), is applied to each view generation orbit. To illustrate, we describe the procedure for single orbit (e.g., the right orbit in Stage I), and views within all other orbits across Stages I, II, and III are processed analogously.
Let Bo denote the initial input view for orbit, and let {=r represent the sequence of T novel views generated along this orbit. The optimization proceeds in a sliding window manner, Initially, xo serves the reference view and is paired with the first N generated novel views, {ar,) For each pair (2o, x;) where i E {I...., .V. ). the novel view æ; undergoes our pairwise iterative Pose- View Optimization (PVO), detailed in Section 3.3.2. This step yields an optimized view & and its corresponding refined pose P;. After this initial set of N views is optimized, the N-th optimized view, &y (along with its pose Pvv) ). becomes the new reference view. This new reference, & &N. is then paired with the subsequent block of N views, i.e., (ÆN. x+++i for E {1,.... .N.. These pairs then undergo the same PVO process. This progressive, sliding- window optimization scheme continues until all 7 generated views within the orbit have been processed and refined.
hiding-window
This strategy is adopted to balance two competing factors. We note that consistently using the initial global input view as the reference for all pairs across an entire orbit would lead to progressively larger viewpoint disparities. Such large angular differences can significantly challenge the robustness of pose estimation and the efficacy of the PVO. Conversely, using each immediately preceding optimized view -;-1 as the reference for the current view æ; might introduce error accumulation and propagation along the generation path of orbits.
In Omni3D, address this trade-off by empirically setting the window size N 1/4. For our default setting where = 48 views are generated per orbit, N is therefore set to 12. This approach ensures that the maximum angular difference between the reference view and any target view within its optimization window remains manageable (e.g., approximately 22.5 under our default settings), facilitating stable PVO process.
3.3.2 Pairwise Iterative PVO
Framework. To simplify the notations, we use the pair (æa. ax)) as an example to introduce the pairwise iterative PVO network. The same process utilized to all other pairs. As Figure[ illustrated, we overfit lightweight (8,) network"" on view pair (æg. 2;) to learn Homography matrix (H), flow map (F) and residual (R) for 3D and 2D optimization (O) of the generated view æ,, The optimized view d can be expressed as:
c = Ozz, 0,) W(T(2, H), F) RR
'As Figure (b) shows, the Homography branch contains three CNN layers with strides 1, and respectivaly, and fully connected layer. The flow and residual branches have a residual block and an output layer in each. All CNN layers use the kernal size of with 32 channels, except the output layers for flow and residual, whose channel numbers are 2 and 3, respectively.
5

---

where T(-, H) and W(-, F) denotes the Homography transformation and 2D warping, respectively. worth point out that each network (6,) is overfit for each pair (Tg. F;) in an online training manner, and the weights do not share across pairs. Besides, it is important that the parameters in the
lighweight network is zero-initialized, expect the bias of the output layer for Homography matrix, which initially outputs 3333 As such, the refined view á. is initialized as the input 2;, L.e.
init 3;
given
13x3, F O, and R 0.
(2)
Next, employ the MAS 117 network to produce the pointmaps of wo and &; (initially w;)) Po and P., respectively. Note that Po and P; are presented in some coordinate, which denote as the world coordinate. Then, the Perspective-n-Poing with RANSAC [6] scheme is applied to estimate the camera poses defined as Po and Pr for the views To and e;, respectively. Meanwhile, we can also obtain the camera intrinsics K given the estimated poses containing focal values (fa. fy) and the coordinates of the principal point (Cz. Cy).
(PnP) [6] pose computation method
toneerr-tt-wordde which are
In the following, with the input view zu, its pointmap Po. the pose of the target view and camera intrinsics, we are able to reproject wo to the target view in the 3D space. Specifically, we first convert the camera pose P3 to the world-to-camera matrix Po,, i.e.
R 0
R 0
7
p
=
(3)
and then transform the pointmap Po to the target view's coordinate
P = RfPo f%
reproject it into the 2D screen coordinate (u,, 0z) of the target view using camera intrinsics, i.e.
U4
[12
0
Cx
X
X Y
U2
= KP
0
f
Cy
Y
where P
(5)
0
Z
and we define (us; v;) = (ū./ zz, 0./2) 2.). Finally, the RGB values Bg of each 3D point of Po can be mapped into its projected location (w, vs) in the target view's coordinate, considering the depth Z for visibility and blending overlapping points, iee,
To-s Render (u;, Uz), Fu. Z).
(6)
Similarity, the 3D reprojection from 2, to Co. i.e., 2.-s can be calculated in the same manner
After obtaining 2:-00 and Zp-si, we define the loss function
L = Mo ||00
a o + M, | To 2o- x112
Lo
where Mo and M, mask out the black pixels resulting from occlusion. The loss function is minimized overfit the lightweight network for refining the generated view in an online training manner. Note that, during the training process, the MASt3R network itself remains unchanged, but its differentiability is crucial as it allows error poc-ppropagttion.
Iterative optimization. In the proposed PVO method, we employ an iterative optimization scheme to jointly optimize the generated view and refine the estimated camera pose. Recall that at the beginning of optmization, we have a z; and we calculate the initial pose estimations (Pu and p.) and camera intrinsics. Given these parameters. we train the lightweight network 6, to optimize the generated view á; by minimizing the loss function defined in Equation (7). During this training phase, the estimated poses (po and (b)) and camera intrinsics are held constant, and only â is optimized. Once convergence observed for the view optimization, we update the poses Po and fú; based on the refined view, and also camera intrinsics. This updated pose and intrinsics are then used in the next iteration to further optimize d. This cycle of optimizing the view with fixed poses, followed
Forr the 3D reprojection operations, we utilize PyTorch3D 28 which provides differentiable rasterization and rendering functions.

---

by updating the poses and intrinsics based on the refined view, is repeated until the estimated poses converge.
This iterative refinement process simultaneously addresses geometric distortions and inconsistent content in ã; and improves the accuracy of the pose estimation. In our experiments, we empirically observed that the estimated poses consistently converge after three updates (in addition to the initial pose estimation). Consequently, we set the number of iterations in our proposed PVO method.
3.3.3 Parallelism
Leveraging the pairing scheme introduced in Section 3.3.1 the PVO process of view pairs that share the same reference view are independent from each other, allowing significant parallelism in computation. To be specifie, in Stage the PVO of most 4N pairs can be computed in parallel. In Stages II and III, there are at most 3N and 2N pairs can be optimized concurrently, respectively. In our experiments, the machine with 8 NVIDIA A100 GPUs allows the paralleled PVO of 24 pairs. Hence, given N = 12. we are able to parallelly compute PVO on N pairs of two orbits. This way, the entire framework requires only 24 serial PVO computations across all stages, i.e., 8, 12 and 4 serial PVOs in Stages I, II and III, respectively. This does not significantly increase the overall computational time. We analyze the time consumptions in Section 4.5
Experiments
4.1 Evaluation Protocol
The proposed Omni3D method represents the reconstructed 3D scene using 3D Gaussian Splatting (3DGS). Consequently, we evaluate the reconstruction performance by rendering views from the 3DGS model at the camera poses corresponding to the groundtruth views and comparing these rendered images to the groundtruth. To facilitate this comparison, is essential to align the 3D coordinates of groundtruth with the MASt3R coordinates employed in our method. Therefore, associate each groundtruth view with four specific reference views used in Omni3D, depicted as ih, t and A in Figure (a). and then utilize MASt3R to estimate the pose of each groundtruth view. During this pose estimation process, the poses of the four selected Omni3D reference views are held fixed. By doing so, effectively align the estimated poses of the groundtruth views to established MASt3R coordinate system. This alignment allows us to render images from 3DGS at the poses of the groundtruth for evaluation. It is crucial that after aligning the coordinates of groundtruth, the groundtruth views are not included for the training of 3DGS, and are only used in evaluation.
4.2 Experimental Setup
We quantitatively evaluate the 3D scene reconstruction quality of Omni3D on three distinct datasets: the Tanks and Temples [13., Mip-NeRF 360 [1] and DL3DV [9] datasets. For Tanks and Tem- ples [3] and Mip-NeRF 360 [1, we evaluate Omni3D on their whole test sets. For DL3DV [19] we randomly select test scenes non-overlapping with the training samples of MVD. In each test sample, we randomly select groundtruth views in the entire omnidirectional space. We compare the performance of the proposed Omni3D against previous state-of thet-arr open-sourced methods, including ZeroNVS B3i, ViewCrafter [51], and Liftlmage3D B3, in terms of PSNR, SSIM 44] and LPIPS [52). Furthermore, we conduct comprehensive ablation studies to validate the effectiveness of the proposed PVO and analyze the computational time required for each component of the Omni3D framework. More experimental results are in Appendix.
4.3 Main Results
Table I presents a quantitative summary of our results. As demonstrated in the table, our proposed Omni3D approach consistently outperforms all compared methods across all evaluated datasets and metrics. Notably, on the Tanks and Temples dataset, we achieve significant PSNR improvement of
1.45 dB compared to the recent Liftlmage3D 3, and approximately 2.4 dB over ViewCrafter [51]. For perceptual quality metrics SSIM and LPIPS, Omni3D also clearly surpasses the compared methods. Similar performance gains are also observed on the Mip-NeRF 360 and DL3DV datasets, where Omni3D advances the PSNR results of Liftlmage33 by 1.62 dB and 0.87 dB, respectively, and
7

---

Table Evaluation of rendered views in omnidirectional space.
Tanks and Temples
Mip-NeRF 360
DL.DV
Methods
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
ZemoNVS 33
12.67
0.4647
0,7506
13.40
0.2413
0.8299
11.28
0.4725
0.7074
ViewCrafter [
13.91
0.4714
0.5886
14.06
0.2420
0.7649
16.61
0.6185
0.3883
LiftImage3D
14.85
0.4841
0.5781
14.27
0.2491
0.6479
16.21
0.6020
0.4844
Our Omni3l
16.30
0.5308
0.5166
15.89
0.2859
0.6369
17.08
0.6649
0.3348
ZeroNVS
ViewCrafter
Liftimage3D
Our Omni3D
Groundtruth
Figure 3: Visual results of rendered views of our Omni3D and compared approaches.
Table 2: Results of user study on our Omni3D and compared methods.
Methods
Tanks and Temples
Mip-NeRF 360
DL3DD
ZeroNVS 33
1.0
1.3
0.8
ViewCrafter [51
4.3
4.7
7.4
LiftImage3D
5.1
4.5
5.8
Our Omni3D
7.6
7.9
8.2
also achieve superior perceptual quality. We illustrate the visual results in Figure 3 In comparison with pervious methods, our Omni3D approach achieves rendered views with higher visual quality, less distortions artifacts, and better geometrical accuracy to the groundtruth. These visual results align with the conclusion from the numerical evaluations, validating the effectiveness Omni3D.
We further conducted user study with 10 non-expert users, who are requested to rate the reconstructed 3D scenes with scores from 0 (poorest quality) to 10 (perfect quality). the user study, we render the images with omnidirectional trajectories from the 3DGS generated by our Omni3D and the compared methods and send them to the users video format, to reduce the hardware requirements for the users' personal computers. The average ratings are shown in Table It can be seen from Table that our Omni3D approach has obvious superior perceptual quality performance, which aligns with the numerical results.

---

Before PVO
After PVO
Reference view Z0
Øi-+0
Reference view
PVO
3D reproj.
3D reproj.
View #t
Bo
View a;
Figure 4: Ablation visual example of 3D-reprojected views before (left) and after (right) PVO.
Table 3: Ablation study on MVD and PVO
PSNR
SSIM f
LPIPS +
Omni3D w/o PVO
15.56 16.30
0.5198
0.5346
Omni3D
0.5308
0.5166
Liftlmage3D
14.85
0.4841
0.5781
Liftlmage33 PVO
15.28
0.4964
0.5446
4.4 Ablation study
Figure 4 visually illustrates the effects of the proposed PVO method. It can be seen from Figure that before PVO, the reprojected images 2;-+0 and Zp-si contain obviously difference on the objects positions in comparison with the target views. For example, the relative position between the women's head and the background in æn and æ0--0. and the relative position between the man's head and the tree in a; and 2n-ii (highlighted by red squares). These indicate the geometrical inconsistency in these views. This inconsistency may be not very noticeable in NVS, however, may significantly challenge the accurate reconstruction with 3DGS. After applying the proposed PVO, it can be seen that the geometrical error the 3D-reprojected views has been effectively corrected, reflecting the improved consistency between the optimized view &; and the reference view Øg. As such, the PVO facilitates our Omni3D achieving state-of of-te-of-the-ar performance on omnidirectional 3D reconstruction.
Table 3 shows numerical results for the ablation studies on the Tanks and Temples dataset [13] In comparison with the baseline model without PVO (Omni3D w/o PVO), the proposed Omni3D achieves 0.74 dB improvement in terms of PSNR, and also better SSIM and LPIPS performance. Moreover, Table 3 also indicated the generalizability the proposed PVO method various MVDs. We conduct experiments for utilizing PVO LiftImage3D [3), whose MVD employs MotionCtrl [45.

---

Table 4: Time consumption on 8XA100 GPUs.
MVD
Pose calc.
3DGS
Total
ZeroNVS [3
133.7 min
ViewCrafter [51
2.1 min
12.8 min
14.9 min
Liftlmage3D
3.5 min
1.5 min
67.4 min
72.4 min
Our Omni3D
10.8 min
10.5 min
12.8 min
34.1 min
Table 5: Time consumption on single A100 GPU.
MVD
Pose calc.
3DGS
Total
ZeroNVS 33
133.7 min
ViewCrafter [51
4.3 min
12.8 min
17.1 min
Liftlmage3D 3
12.0 min
1.5 min
67.4 min
80.9 min
Our Omni3D
21.6 min
83.9 min
12.8 min
118.3 min
Table shows that the proposed PVO method successfully advances the performance of Liftlmage3I by 0.37 dB in terms of PSNR, and also benefits the SSIM and LPIPS performance.
4.5 Time consumption
4able|[shows the analysis on the time consumption each component of our Omni3D and compared methods. We conduct experiments on machine with 8 NVIDIA A100 GPUs. In the compared methods, ZeroNVS [33] employs Score Distillation Sampling (SDS) for novel view generation. Since NeRF distillation is very time-consuming. it takes > hours for the inference of ZeroNVS. In our Omni3D, we utilize a standard 3DGS, whose training is much faster than the distortion- aware 3DGS in Liftlmage3D
(3). Most importantly, thanks the parallelism scheme of Omni3D introduced in Section 3.3.3 the proposed PVO method does not significantly increase the total time consumption. The paralleled PVO takes 10.5 min, which is less than the time used for 3DGS optimization. As a result, the whole framework of Omni3D takes about 34 min for an omnidirectional 3DGS reconstruction from single image. It obviously faster than ZeroNVS and LiftImage3D, Note that we also run the compared methods with maximal parallelism on the same hardware as ours.
In Table 5 We further show the results on single A100 GPU. It can be from Table that when the parallelism capability is limited to a single A 1100 GPU, the proposed Omni3D approach consumes additional 46.2% of computational time compared with Liftlmage3D B3 and still faster than ZeroNVS (33) (CVPR'22 24). However, our Omni3D approach is able to reconstruct the en- tire omnidirectional 3D space, instead of only the forward-facing views, and also achieves better reconstruction quality.
Conclusion
In this paper, introduced Omni3D, a novel framework specifically designed to enhance the geometric accuracy of diffusion-generated views for robust omnidirectional 3D Gaussian Splatting. Our key innovation lies in the synergistic Pose- View Optimization (PVO) process. By iteratively refining both the generated view content and their estimated camera poses, guided by geometric priors from techniques like MAS(3R and minimizing 3D reprojection errors, Omni3D produces set of geometrically consistent novel views with high pose accuracy. These refined views and poses serve as advanced foundation for constructing an explicit 3D Gaussian Splatting representation. As demonstrated by our experimental results, Omni3D achieves state-of-the-aa performance in single- image omnidirectional 3D scene reconstruction, yielding substantially improved rendering quality across a wide range of view angles compared to existing methods. This work represents significant step towards enabling accurate and high-quality 3D reconstruction of complex, omnidirectional environments from single image.
10

---

References
[1] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded anti- aoii-al neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5470-5479, 2022.
[2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[3] Yabo Chen, Chen Yang, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Wei Shen, Wenrui Dai, Hongkai Xiong, and Qi Tian. Liftlmage3D: Lifting any single image to 3D gaussians with video generation priors. arXiv preprint Xrx: 09527,999977. 2024.
[4] Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3D: Video diffusion models are effective 3D generators. arXiv preprint arXiv:2403.066 067338 2024.
[5] Jaeyoung Chung, Suyoung Lee. Hyeongjin Nam. Jaerin Lee, and Kyoung Mu Lee. LucidDreamer: Domain-free generation of 3D gaussian splatting scenes. arXiv preprini arXiv:2311.13384 2023.
[6] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):38 1981.
[7] Junlin Han, Filippos Kokkinos, and Philip Torr. Vfusion3d: Learning scalable 3d generative models from video diffusion models. In European Conference on Computer Vision (ECCV), pages 333-350. Springer. 2024.
[8] Yuxuan Han, Ruicheng Wang, and Jiaolong Yang. Single-view view synthesis in the wild with learned adaptive multiplane images. In ACM SIGGRAPH. 2022.
[9] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
[10] Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. CameraC- trl: Enabling camera control for text-to- generation. arXiv preprint 4rXiv:2404:02101 2024.
[11] Wenyi Hong, Ming Ding, Wendi Zheng. Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In International Conference on Learning Representations (ICLR), 2023.
[12] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküihler, and George Drettakis. 3D gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, +2(())13991, 2023.
[13] Arno Knapitsch, Jaesik Park. Qian-Yi Yi Zhou, and Vladlen Koltun. Tanks and Temples: Benchmarking large-scale scene reconstruction. ACM Trunsactions on Graphics, 16(4))1-13, 2017.
[14] Jeong-gi Kwak, Erqun Dong, Yuhe Jin, Hanscok Ko, Shweta Mahajan, and Kwang Moo Yi. Vivid-1-to-3: Novel view synthesis with vidco diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6775-6785, 2024.
[15] Dae Yeol Lee, Guan-Ming Su. and Peng Yin. OGRMPI: An efficient multiview integrated multiplane image based on occlusion guided residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 794-802, 2024.
[16] Vincent Lepetit, Francesc Moreno-Noguee and Pascal Fua. EPuP: An accurate O(n) solution to the PnP
problem. International Journal of Computer Vision, 81:155-166, 2009.
[17] Vincent. Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3D with MASt3R. In European Conference on Computer Vision (ECCV), pages 71 71-91, 2024.
[18] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3D: High-resolution text-to-3D content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 300-309, 2023.
[19] Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lui, et al. DL33V-10K: A large-scale scene dataset for deep learning-baad based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22160-22169, 2024.
[20] Fangfu Liu, Wenqiang Sun, Hanyang Wang. Yikai Wang. Haowen Sun, Junliang Ye, Jun Zhang. and Yueqi Duan. ReconX: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint arXiv:2408. 16767, 2024.
[21] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-t0-3: Zero-shot one image to 3D object. In Proceedings of the IEFF//VF International Conference on Computer Vision (CVPR), pages 9298-9309, 2023.
[22] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma. Song- Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3D: Single image to 3D using cross-doman diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9970-9988, 2024.
11

---

[23] Luke Melas- Kyriazi, Iro Laina. Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. IM-3D: Iterative multiview diffusion and reconstruction for high-quality 3D generation. International Conference on Machine Learning (ICML), 2024.
[24] Yiyu Pang. Tong Jia, Yichun Shi, Zimeng Tang. Jiaxiang Zhang, and Xiaohua Cheng. Envision3D: One
image to 3D with anchor views interpolation. arXiv preprint arXiv22403.0890 2024.
[25] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D) diffusion. In International Conference on Learning Representations (ICT.),, 2023.
[26] Guo Pu, Peng-Shuai
Shuai Wang, and Zhouhui Lian. SinMPI: Novel view synthesis from a single image with expanded multiplane images. In SIGGRAPH Asia, 2023.
[27] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.061 022 2022.
[28] Nikhila Ravi, Jeremy Reizenstein, David Novotny. Taylor Gordon, Wan-Yen Lo, Justin Johnson, and
Georgia Gkioxari. Accelerating 3D deep learning with PyTorch3D. In SIGGRAPH Asia, 2020.
[29] Chris Rockwell, David F Fouhey, and Justin Johnson. Pixelsynth: Generating a 3d-consistent experience from a single image. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pages 14104-14113, 2021.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pottern Recognition (CVPR), pages 10684-10695, 2022.
[31] Robin Rombach, Patrick Esser, and Bjöm Ommer. Geometry- free view synthesis: Transformers and no 3D priors. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), pages 14356-1436, 2021.
[32] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35:36479936494, 2022.
[33] Kyle Sargent, Zizhang Li. Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. ZeroNVS: Zero-shot 360-degree view synthesis from single image. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 9420-9429, 2024.
[34] Yichun Shi, Peng Wang, Jianglong Yc, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-viow diffusion for 3D generation. arXiv preprint arXiv:2308. 16512, 2023.
[35] Meng-Li Shih, Shih- Yang Su, Johannes Kopf, and Jia-Bin Huang. 3D photography using context-awan aware layered depth inpainting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8028 8038, 2020.
[36] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 175-184, 2019.
[37] Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. Dimen- sionX: Create any 3D and 4D scenes from single image with controllable video diffusion. arXiv preprint arXiv:2411.0423 2024.
[38] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-It-3D: High-fidelity 3D creation from single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22819-22833 2023.
[39] Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 551-560, 2020.
[40] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy 1lg. Alexey Dosovitskiy, and Thomas Brox Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5038-5047, 2017.
[41] Vikram Volcti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D generation from single image using latent video diffusion. In European Conference Computer Vision (ECCV), pages 439-457, 2025.
[42] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2D diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12619-12629, 2023.
[43] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Beris Chidlovskii, and Jerome Revaud. DUS:3R: Geometric 3D vision made casy. In Proceedings of the IEEE//VF Conference on Computer Vision and Patter Recognition (CVPR), pages 20697-20709 2024.
[44] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-6) 612, 2004.
12

---

[45] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li. Tianshui Chen. Menghan Xia, Ping Luo, and Ying Shan. MotionCtll: A unified and flexible motion controller for video generation. In ACM SIGGRAPH, 2024.
[46] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. SynSin: End-to-end view symthesis from single image. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 7467-7477, 2020.
[47] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. NeX: Real-time view synthesis with neural basis expansion. In Proceedings of the IEEE/CVF Conference
Computer Vision and Pattern Recognition (CVPR), pages 8534-8543. 8543, 2021.
[48] Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu. Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, et al. CogVideoX: Text-to-vide video diffusion models with an expert transformer. In International Conference on Learning Representations (ICLR), 2025.
[49] Donggeun Yoon and Donghyeon Cho. CORE-MPI: Consistency object removal with embedding multiplane image. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 20081-20090. 2024.
[50] Hong- Xing Yu, Haoyi Duan, Charles Hermann, William T Freeman, and Jiajun Wu. WonderWorid: Interactive 3D scene generation from single image. arXiv preprint arXiv:2406.00 09394, 2024.
[51] Wangbo Yu, Jinbo Xing. Li Yuan, Wenho Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-T'ii Wong. Ying Shan, and Yonghong Tian. ViewCrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409. 244400440 2024.
[52] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as a pereeptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 586-595, 2018
[53] Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox. Deeptam: Deep tracking and mapping. In European Conference on Computer Vision (ECCV), pages 822-838, 2018.
[54] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images, ACM Transactions on Graphics, 37(4), 2018.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper describes all details of the methodology, evaluation protocol,
experiments setup, and computer resources reproduce the experimental results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We will release codes upon internal approval. The datasets we used are publicly
available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssiinyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper includes all details on datasets, hyperparameters, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The experiments are conducted on datasets with sufficient samples and the results are statistically significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information of computer resources in Section 4.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We introduced the societal impacts in Section I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credited the original owners of assets well and properly respected their license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core methodology does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Appendix
A Detailed Network Architecture
Figure 5 illlustrates detailed architecture of the lightweight network in our PVO method. In this figure, the convolutional layers are denoted as "Conv, filter size, filter number", and we use + 2 to denote the stride of "GeLU" indicates the activation function of Gaussian Error Linear Unit. Besides, the output layer in the Homography branch is a dense layer with 8 nodes, whose outputs are defined as O1 Os, respectively. They form the Homography matrix H as
(0
O2
O3
H
O4 0
Os
O6
(8)
Os
Recall that all parameters in convolutional layers and weights in the dense layer are zero-initialized and the bias of the dense layer are initialized as [1, 0, 0, 0, 1, 0, 0, 0J". As such, we have
Ii.
given
H = 13x3, F 0, and R = 0
(9)
to initialize the refined view a with its original input æ; at the beginning of PVO.
View .;
Coev 3x3, 32, GeL.U
Cuny, _2. 3+3, 32, GeLU
Conv, 12, 3x3, 32, GoLU
Dense layer. 8
Homography matrix
H
Conv 12, GeLU
concat
Conv 3K3, 32, GeLU
ConY 3+3, 32, GaLU
CunYy 3w3, 2
Flow map
F
Reference To
3x1
3x3, GeLU
3x3, GeLL
Residual
R.
Lightweight network 0;
Figure 5: Detailed architecture of the lightweight network in PVO.
B Additional Ablation Studies
Ablation study on N in progressive pairing. Recall that in Section 3.3. we proposed a progres- sive pairing scheme with the window size of N, i.e., in each generation orbit, the first N generated views undergo PVO with the reference of æg. and then the next N views takes the optimized N-th view N as reference. This progressive scheme continues until all generated views within the orbit have been processed and refined. shows the ablation study on the selection of N on the Tanks and Temples dataset (13. It can be seen from Table that N 12 results in the best performance. This outcome is likely attributable to the fact that utilizing views with large disparities as references (e.g., 24 or 48) can significantly impair the robustness of pose estimation and diminish the efficacy of PVO. Conversely, employing each immediately preceding optimized view, Φ;-1 as the reference for the current view x; (N 1), may inadvertently introduce error accumulation and propagation along the generation path of the orbits. Besides, setting N = 1 also considerably limits parallelism.
we proposed a pairwise iterative PVO method. In the proposed PVO, given the initially estimated poses and intrinsics, we refine the generated view by minimizing the reprojection error until convergence, and then the poses and intrinsics are updated based on the refined views. This cycle iteratively conducted until estimated poses converge. Table 7 illustrates the performance on the Tanks and Temples dataset [13] with different iterations of pose updates, addition to the initial pose estimation.
Ablation study on iterations of pose updates in PVO. Recall that in Section 3.3.2
It can be seen that the performance converges at 3 iterations, i.e., optimize the views and then update
21

---

Table 7: Ablation iterations of pose updates in PVO, in addition to the initial pose estimation
Table 6: Ablation on progressive pairing
PSNR
SSIM f
LPIPS 4
Iterations
PSNR
SSIM t
LPIPS 4
w/o PVO
15.56
0.5198
0.5346
(wlo PVO)
15.56
0,5198
0.5346
16.24
0.5305
0,5170
15.62
0.5207
0.5325
N 12
16,30
0.5308
0.5166
2
15,91
0,5254
0,5296
N 24
16.19
0.5281
0.5179
3
16.30
0.5308
0.5166
N 15.98
0.5206
0.5254
4
16.33
0.5311
0.5162
poses for three time. When the iteration number comes 4, the performance negligibly increases. This verifies the reasonability for setting the iterations to 3 in our approach.
C Limitations and Future Work
Current 3D reconstruction techniques, including the proposed Omni3D, ViewCrafter, and Liftlmage. largely rely multi-stage process where 2D novel view synthesis acts crucial intermediary for generating 3D Gaussian Splatting (3DGS) reconstructions. This approach involves first synthesizing numerous 2D images from different perspectives and then using these synthesized views infer the underlying 3D structure and appearance. While effective, this indirect methodology often introduces computational overhead and can sometimes limit the fidelity of the final 3D output due to potential errors or inconsistencies introduced during the 2D synthesis phase. The reliance on this intermediate step means that the overall efficiency and quality of 3D reconstruction often constrained by the performance of the 2D view synthesis component.
The recent emergence of powerful foundation models across various domains presents transforma- tive opportunity for the field of 3D reconstruction. Instead of the current indirect approach, these advanced models could enable more direct paradigm: training models to generate 3DGS, even other sophisticated 3D formats, straight from a single 2D image input. This shift would fundamentally bypass the computationally expensive 2D intermediate steps, leading to several significant advantages. Not only could this potentially lead to a substantial improvement in the quality and realism of 3D reconstructions by eliminating potential bottlenecks introduced by the 2D synthesis. but it would also dramatically reduce the inference time. direct generation pipeline would streamline the entire process, making 3D reconstruction faster and more accessible for wider range of applications, from virtual reality to robotics.
Taking this vision even further, the ongoing development of world foundation models and AI agents hints at an even more revolutionary future. Imagine a scenario where complex 4D (3D over time) content could be generated directly from simple, high-level prompts, completely bypassing all 2D and 3D intermediaries. This would represent a paradigm shift, moving from data-driven reconstruction to concept-driven generation, where AI understands and creates dynamic 3D environments and objects based on abstract instructions. These ambitious directions, spanning from direct 3DGS generation to holistic 4D content creation, represent some of the most significant and exciting avenues for future research in artificial intelligence and computer graphics, promising to unlock unprecedented capabilities in digital content creation and spatial computing.
22


Reviews:
[{"review_id": "VFU4P06Ard", "reviewer": "Reviewer_Wu2D", "summary": "The paper proposes a novel method for single-image to 3D reconstruction, outperforming prior work. The method uses a multi-view diffusion model to create many views from an input image. The authors then propose a \"Pose-View Optimization module\" to refine both the camera poses (estimated via MASt3R) and the generated views. This process starts with nearby views and then expands to farther views. Finally, once this has been done for each region (both nearby and far views), 3DGS is used for 3D reconstruction.", "questions": "Overall, I like this paper and am inclined towards acceptance. My main concern is I did not notice much improvement in the qualitative results, but I still feel the ideas from the method are a useful contribution.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The proposed method seems well designed. In particular, the design of the progressive pairing strategy is well thought out and seems like a valuable contribution for future work to build upon.\n- In addition to the point above, I appreciated the ablation on the Pose-View Optimization module, as it helped support the benefit of the module.\n- The quantitative results support the efficacy of the proposed method.\n\n**Weaknesses**\n- I was a bit underwhelmed by the qualitative results. The paper would benefit from a user study to see if the reported metrics (PSNR/SSIM/LPIPS) align. For me, I did not notice significant improvements in the images from the proposed method. Perhaps identifying areas that are significantly better (with a box) and showing a zoomed in view of that region would be helpful for readers. \n- Additional ablations on the progressive pairing strategy and different hyperparameters would be informative.", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer very much for the constructive suggestions and insights into the qualitative results and the ablations on progressive pairing and hyperparameters.\n### **Qualitative results**\n* **User study.** According to the suggestion of the reviewer, we conducted a user study with 10 non-expert users, who are requested to rate the reconstructed 3D scenes with scores from 0 (poorest quality) to 10 (perfect quality). In the user study, we render the images with omnidirectional trajectories from the 3DGS generated by our Omni3D and the compared methods and send them to the users in video format, to reduce the hardware requirements for the users’ personal computers. The average ratings are shown in Table 1. It can be seen from Table 1 that our Omni3D approach has obvious superior perceptual quality performance, which aligns with the numerical results reported in the paper. We will add this experiment to the revised manuscript.\n\n    \\\n    Table 1: Results of user study on our Omni3D and compared methods\n\n    | Methods | Tanks and Temples$\\ \\ \\ \\ \\ $ | Mip-NeRF 360$\\ \\ \\ \\ \\ $ | DL3DV |\n    |:--------|:-----------------|:------------|:-----|\n    | ZeroNVS [33]|    1.0     |        1.3         |          0.8          |               \n    | ViewCrafter [51]|       4.3          |        4.7         |          7.4               |\n    | LiftImage3D [3]|         5.1        |        4.5         |          5.8          |           \n    | **Our Omni3D** |        **7.6**         |         **7.9**        |          **8.2**          |               \n    | | | | | |\n\n\n* **Visual examples in videos.** Besides, we have uploaded three visual examples in video format in the *Supplementary Material*. It can be obviously seen from the videos that the proposed Omni3D approach achieves 3D scene reconstruction in the omnidirectional space with higher visual quality and better geometry consistency than the compared approaches.\n* **Figure 3 in the paper.** In the revised manuscript, we will follow the reviewer’s suggestion to highlight the regions with obvious differences with zoomed in views to increase readability.\n### **Ablations on the progressive pairing strategy and hyperparameters**\nWe have provided additional ablation studies on the progressive pairing strategy and hyperparameters in Section C in the *Supplementary Material*. We also show the results in the following.\n* **Ablation study on progressive pairing.** \nRecall that in Section 3.3.1, we proposed a progressive pairing scheme with the window size of $N$, i.e., in each generation orbit, the first $N$ generated views undergo PVO with the reference of $\\boldsymbol{x}_0$, and then the next $N$ views takes the optimized $N$-th view $\\boldsymbol{\\hat{x}}_N$ as reference. This progressive scheme continues until all generated views within the orbit have been processed and refined. \n\n    Table 2 shows the ablation study on the selection of $N$ on the Tanks and Temples dataset [13]. It can be seen from Table 2 that $N=12$ results in the best performance. This outcome is likely attributable to the fact that utilizing views with large disparities as references (e.g., $N=24$ or $48$) can significantly impair the robustness of pose estimation and diminish the efficacy of PVO. Conversely, employing each immediately preceding optimized view, $\\boldsymbol{\\hat{x}}_{i-1}$ as the reference for the current view $\\boldsymbol{x}_i$ ($N=1$), may inadvertently introduce error accumulation and propagation along the generation path of the orbits. Besides, setting $N=1$ also considerably limits parallelism.\n    \n    \\\n    Table 2: Ablation on progressive pairing\t\n    | | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n    |---|---|---|---|\n    | w/o PVO | 15.56 | 0.5198 | 0.5346 |\n    | $N=1$ | 16.24 | 0.5305 | 0.5170 |\n    | $N=12$ | **16.30** | **0.5308** | **0.5166** |\n    | $N=24$ | 16.19 | 0.5281 | 0.5179 |\n    | $N=48$ | 15.98 | 0.5206 | 0.5254 |\n    | | | | | |\n\n* **Ablation study on iterations of pose updates in PVO.** \nRecall that in Section 3.3.2, we proposed a pairwise iterative PVO method. In the proposed PVO, given the initially estimated poses and intrinsics, we refine the generated view by minimizing the reprojection error until convergence, and then the poses and intrinsics are updated based on the refined views. This cycle is iteratively conducted until estimated poses converge. Table 3 illustrates the performance on the Tanks and Temples dataset [13] with different iterations of pose updates, in addition to the initial pose estimation. It can be seen that the performance converges at 3 iterations, i.e., optimize the views and then update poses for three time. When the iteration number comes to 4, the performance negligibly increases. This verifies the reasonability for setting the iterations to 3 in our approach.\n    \n    \\\n    Table 3: Ablation on iterations of pose updates in PVO, in addition to the initial pose estimation\n\n    | Iterations | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n    |---|---|---|---|\n    | 0 (w/o PVO) | 15.56 | 0.5198 | 0.5346 |\n    | 1 | 15.62 | 0.5207 | 0.5325 |\n    | 2 | 15.91 | 0.5254 | 0.5296 |\n    | 3 | 16.30 | 0.5308 | 0.5166 |\n    | 4 | 16.33 | 0.5311 | 0.5162 |\n    | | | | | |"}, {"review_id": "agC0DKlq4L", "reviewer": "Reviewer_ExWa", "summary": "This paper addresses the challenge of omnidirectional 3D scene reconstruction from a single image, identifying common issues in existing diffusion-based methods such as geometric distortion and inconsistencies. To overcome these, the authors propose Omni3D, which leverages pose estimation priors and a Pose-View Optimization (PVO) module to iteratively refine novel views and their estimated camera poses. Experimental results demonstrate significant improvement over existing state-of-the-art methods.", "questions": "1. This submission is very similar to LiftImage3D from motivation to method. Please clarify the difference between the two, which is very important to highlight the contribution of this submission.\n\n2. Are the experimental setup and comparison methods consistent? Why are the results reported in this article so much lower than those reported in the original paper? It is recommended to unify the evaluation, as this is crucial to enhancing the credibility of the submission. If the uncertainty is due to random selection, then repeated verification is necessary.\n\n3. Include more detailed discussions about computational costs and potential optimizations or approximations to reduce inference time.\n\nMy main concern is the contribution of this paper and the reliability of the experimental data. Currently, the contribution of this paper is incremental compared to previous methods, and the experimental results do not appear convincing. I am happy to see the author's clarification and response.", "limitations": "1. Omni3D may require substantial computational resources, potentially limiting deployment in real-time or resource-constrained scenarios.\n\n2. The method's iterative refinement strategy may accumulate errors, especially for extreme viewpoints or highly complex scene geometries.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The task of single-image 3D reconstruction is significant for future AI-powered immersive media. The paper clearly identifies critical issues in current methods and proposes targeted solutions.\n\n2. The iterative Pose-View Optimization (PVO) approach, which jointly optimizes pose estimation and generated views by minimizing 3D reprojection errors, is well-motivated.\n\n3. The video in the supplementary material shows good performance.\n\nWeaknesses:\n\n1. This submission is very similar to LiftImage3D from motivation to methodology. The differences between Omni3D and LiftImage3D must be explicitly clarified to effectively highlight the original contributions of this paper. Currently, the incremental nature of contributions diminishes the novelty of the submission.\n\n2. The reported experimental results are notably lower than those presented in the original papers being compared. This inconsistency raises concerns regarding experimental reliability and evaluation fairness. It is crucial to unify experimental setups and perform repeated evaluations to ensure the credibility of the results.\n\n3. Although parallelization strategies are mentioned, the computational complexity and inference time still appear substantial. The paper lacks detailed discussions on potential optimizations or approximations that could effectively reduce computational costs.", "was_revised": true, "final_justification": "My main concerns were addressed well. The core contribution of this submission is the proposed Pose-View Optimization (PVO) method, which enables 360-degree reconstruction of the scene. This is both meaningful and impressive, so I've increased my initial rating.", "rebuttal": "Thanks to the reviewer for the positive comments on the motivation of the proposed PVO method and the performance of the videos in the supplementary material. We would like to also thank the reviewer very much for the valuable questions, which are incredibly helpful in improving the quality of our manuscript. We provide point-by-point responses to the reviewer’s comments as follows:\n### **Novelty and contributions vs. LiftImage3D**\nLiftImage3D proposes a distortion-aware 3DGS method to reduce the distortion of 3DGS during the Gaussian optimization process with the generated multi-view images which suffer from content and geometric inconsistencies. Although the distortion-aware 3DGS mitigates the inconsistencies among multi-view images to a certain degree, it may be an indirect way with incremental effectiveness and low time efficiency. On the contrary, in this paper, we proposed a novel Pose-View Optimization (PVO) method to directly and jointly correct both the estimated poses and the generated contents by an iterative self-supervised learning. This way, the geometric inconsistencies and the error of pose estimations can be directly and effectively refined before the optimization of 3DGS. The experiments validated that our approach is a more effective way with better quality performance and also achieves higher time efficiency (when parallelism applies) than LiftImage3D. More importantly, thanks to the propose PVO method, we are able to achieve the challenging task of *omnidirectional* 3D reconstruction, instead of only reconstructing 3DGS in narrow angles in LiftImage3D.\n### **Experimental setup**\nThanks a lot for the valuable comments on the experimental setup. The reason for the lower reported numbers in our manuscript than those reported in the original papers is that we evaluate the quality of rendered images in the entire **omnidirectional** space, instead of only the surrounding views of the input (please see line 11 of Sec. 4.1 in the LiftImage3D paper). We will release the codes publicly with our evaluation protocol, which ensures the reproducibility of our approach, and furthermore may contribute to benchmarking the future works in this direction.\n\nWe agree with the reviewer that it is beneficial to conduct additional experiments with the settings aligned with the original paper of compared methods. Since the SOTA method LiftImage3D does not provide an evaluation protocol in their open-sourced codes, we are not able to exactly know their detailed experimental setup, such as which groundtruth views are selected in each sample. Hence, we tried our best to set up the settings where LiftImage3D performs similarly to the original numbers reported in their paper, and then evaluate our proposed Omni3D approach. The results are shown in Table 1 in this rebuttal below.  It can be seen from Table 1 that our Omni3D approach still achieves superior performance under this setting. We will add the results of this experiment to the revised manuscript.\n    \n* Table 1: Evaluation on the surrounding views of input (align with LiftImage3D)\n\n    | Methods | Tanks and Temples | Mip-NeRF 360 | DL3DV |\n    |:--------|:-----------------|:------------|:-----|\n    |         | PSNR $\\uparrow\\ \\ \\ \\ $   SSIM $\\uparrow\\ \\ \\ $   LPIPS $\\downarrow\\ \\ \\ $ | PSNR $\\uparrow\\ \\ \\ \\ $  SSIM $\\uparrow\\ \\ \\ $  LPIPS     $\\downarrow\\ \\ \\ $ | PSNR $\\uparrow\\ \\ \\ \\ $  SSIM $\\uparrow\\ \\ \\ $  LPIPS $\\downarrow$ |\n    | ZeroNVS [33]|         13.98 $\\ \\ \\ \\ \\ $\t0.4856 $\\ \\ \\ \\ $ 0.6225 |\t14.25 $\\ \\ \\ \\ \\ $\t0.2669 $\\ \\ \\ \\ $\t0.8657|\t13.73 $\\ \\ \\ \\ \\ $\t0.5380 $\\ \\ \\ \\ $\t0.6476\n    | ViewCrafter [51]| 15.79 $\\ \\ \\ \\ \\ $\t0.5054 $\\ \\ \\ \\ $\t0.5625\t | 15.99 $\\ \\ \\ \\ \\ $\t0.2717 $\\ \\ \\ \\ $\t0.7012\t | 17.85 $\\ \\ \\ \\ \\ $\t0.6459\t$\\ \\ \\ \\ $ 0.3843\n    | LiftImage3D [3]|  16.14 $\\ \\ \\ \\ \\ $\t0.5232 $\\ \\ \\ \\ $\t0.5346 | \t16.09 $\\ \\ \\ \\ \\ $\t0.2776 $\\ \\ \\ \\ $ 0.5553 |\t20.77 $\\ \\ \\ \\ \\ $\t0.6834 $\\ \\ \\ \\ $\t0.4557\n    | **Our Omni3D** | **18.32** $\\ \\ \\ \\ \\ $\t**0.5919** $\\ \\ \\ \\ $ **0.4689** |\t**16.96** $\\ \\ \\ \\ \\ $\t**0.2880** $\\ \\ \\ \\ $\t**0.4807** | \t**22.02** $\\ \\ \\ \\ \\ $\t**0.7649** $\\ \\ \\ \\ $ **0.3011**\n    | | | | | |\n\n### **Computational costs and potential optimizations**\nWe agree with the reviewer that although we employed significant parallelism in computation, the inference time is still substantial. This is mainly because the diffusion-based generation model (i.e., MVD) and Gaussian optimization are time-consuming. Other image-to-3D methods, e.g., ViewCrafter and LiftImage3D, also suffer from the same problem. As Table 3 in our manuscript shows, the proposed PVO method consumes only ~30% of the total computational time and the proposed Omni3D achieves faster speed than LiftImage3D. \nBased on above analysis, a potential way to effectively reduce time consumption is training a (diffusion-based) model to generate 3D Gaussian splats straight from a single image input, bypassing the 2D (multi-view images) intermediate steps. This way, it saves the computational time for generating hundreds of multi-view images and also eliminates the optimization process for 3DGS. As such, the computational complexity is expected to be significantly reduced. This can be seen as interesting and valuable future work.\n### **Error accumulation in iterative refinement**\nWe thank the reviewer very much for pointing out the possible risk of error accumulation. We analyze this issue from two aspects as follows:\n* **Error in the iterative PVO of each pair.** In the iterative PVO process of each pair, the generated views and estimated poses and intrinsics are jointly optimized by minimizing the reprojection error until convergence. This process should *not* suffer from error accumulation, since the error (loss) continuously reduces until nearly unchanged throughout the optimization. In Section C of the *Supplementary Material*, we have conducted the ablation study regarding the quality performance and the iterations of pose updates in PVO. We also show the results as follows. It can be seen from Table 2 in this rebuttal that the reconstruction quality continuously increasing along with the process of iterative PVO (the 0 to 4 iterations of pose updates). This confirms the continuous reduction of error without accumulation. Besides, the performance only negligibly increases at the 4-th iteration, hence it verifies the reasonability for setting the iterations of pose updates to 3 in our PVO method.\n    \n    \\\n    Table 2: Ablation on iterations of pose updates in PVO, in addition to the initial pose estimation\n    | Iterations | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n    |---|---|---|---|\n    | 0 (w/o PVO) | 15.56 | 0.5198 | 0.5346 |\n    | 1 | 15.62 | 0.5207 | 0.5325 |\n    | 2 | 15.91 | 0.5254 | 0.5296 |\n    | 3 | 16.30 | 0.5308 | 0.5166 |\n    | 4 | 16.33 | 0.5311 | 0.5162 |\n    | | | | | |\n\n* **Error in the progressive paring scheme.** Recall that in Section 3.3.1, we proposed a progressive pairing scheme with the window size of $N$, i.e., in each generation orbit, the first $N$ generated views undergo PVO with the reference of $\\boldsymbol{x}_0$, and then the next $N$ views takes the optimized $N$-th view $\\boldsymbol{\\hat{x}}_N$ as reference. This progressive scheme continues until all generated views within the orbit have been processed and refined. \n\n    This scheme is a trade-off between the risk of error accumulation and the robustness of PVO. That is, the large size of $N$ (consistently use the initial view as reference for a large range of other views) would lead to progressively larger viewpoint disparities and therefore challenge the robustness of PVO, and on the contrary, using immediately preceding optimized view (very small $N$) may introduce error accumulation. We have conducted the ablation study on this trade-off balance in Section C of the *Supplementary Material*, and we show the results in Table 3 in this rebuttal. It can be seen from Table 3 that utilizing views with large disparities as references (e.g., $N=24$ or $48$) can significantly impair the robustness of pose estimation and diminish the efficacy of PVO. Conversely, employing each immediately preceding optimized view, $\\boldsymbol{\\hat{x}}_{i-1}$ as the reference for the current view $\\boldsymbol{x}_i$ ($N=1$), may inadvertently introduce error accumulation and propagation along the generation path of the orbits. Therefore, setting $N=12$ in our approach results in the best performance, indicating a reasonable balance of the trade-off issue. \n\n    \\\n    Table 3: Ablation on progressive pairing\t\n    | | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n    |---|---|---|---|\n    | w/o PVO | 15.56 | 0.5198 | 0.5346 |\n    | $N=1$ | 16.24 | 0.5305 | 0.5170 |\n    | $N=12$ | **16.30** | **0.5308** | **0.5166** |\n    | $N=24$ | 16.19 | 0.5281 | 0.5179 |\n    | $N=48$ | 15.98 | 0.5206 | 0.5254 |\n    | | | | | |"}, {"review_id": "Ko1QOre7nJ", "reviewer": "Reviewer_yK7z", "summary": "This paper introduces a method to tackle the challenge of single-image 3D reconstruction by integrating the diffusion-based novel view synthesis methods and a pose refinement loop, named Pose-View Optimization (PVO). The proposed method progressively synthesizes sets of novel views around the input view, or around optimized views with a larger disparity to the input one. Then, the PVO module iteratively refines the pose pair of the novel views by PnP, where the 3D points are estimated from a pretrained MASt3R model. This core component yields a set of more geometry-consistent posed novel views. Finally, these pose-optimized views are fed into 3DGS pipelines to generate an omnidirectional scene representation.", "questions": "- Can this method generalize to scenes other than object-centric, e.g. forward-facing scene generation?\n- How long does it take to optimize the lightweight network?\n- How much does the optimized poses drift from the input ones? If not, why not register the four optimized poses (front, left, right, back) to the ground truth poses followed by evaluating the rendered views at registered poses?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- An innovative design to overcome the geometry-conflicting distortion introduced by diffusion methods. The lightweight network within this design estimates the residual on top of the diffused view, which is further improved by jointly optimizing its pose.\n- This method does not reply on the choice of the novel view diffusion methods, thereby providing a general framework that can be used for omnidirectional reconstruction from single view inputs.\n- The splitting strategy allows a good parallelism and makes this method applicable.\n- The manuscript is well written.\n### Weakness:\n- Highly dependent on the quality of MASt3R. If the point map estimated by the feed-forward method has poor quality, the whole pipeline may fail.\n- L173, \"expect\" -> \"except\"", "was_revised": true, "final_justification": "This work presents a systematic way to reconstruct a 3D scene from a single image. The iterative PVO module, being the core module of this work, allows this method to generalize to different scenes, showing its potentially broad impact.", "rebuttal": "We would like to thank the reviewer very much for the positive comments, and thanks a lot for the valuable questions and for pointing out the typo. We response to the reviewer's questions point-by-point as follows:\n### **Generalizability on various kinds of scenes**\nIn our experiments, we evaluated the proposed method on three commonly used datasets including various kinds of natural scenes, which are *not* limited to object-centric scenes. The generalizability of scene generation depends on training dataset of the Multi-View Diffusion (MVD). As described in Section 3.2, the MVD in our method is trained on DL3DV-10K, which is the largest scale dataset of 3D scenes as far as we know at the time point of submitting this manuscript. Employing larger-scale datasets, such as SceneSplat-49K (published in June 2025), may further enhance the generalizability of our approach. Besides, please note that the proposed Pose-View Optimization (PVO) method, which is the core contribution of this manuscript, is independent from the specific MVD, and can be generalized to various kinds of MVDs (as validated in Section 4.4).\n### **Time for optimizing the lightweight network**\nTable 3 in our manuscript shows the analysis on the time consumption of each component of our Omni3D approach. The lightweight network is optimized in an online training manner during the proposed PVO scheme in our approach, which averagely takes 10.5 min on a test sample.  This time consumption is less than the time used for 3DGS optimization, and therefore, does not significantly increase the overall time consumption.\n### **Evaluation protocol**\nThanks a lot for the suggestion regarding the evaluation protocol. During the proposed PVO process, the poses of generated views only slight drift in most cases, probably because of the relatively high capacity of the Multi-View Diffusion (MVD) model.\nIn theory, our evaluation protocol, that incorporate the groundtruth views into the coordinate of the generated and optimized views to align the coordinates, should be _equivalent_ to registering the optimized poses of generated views to the coordinate of groundtruth poses. However, our evaluation protocol has a benefit that it can be applicable to the evaluation with *unposed* groundtruth views. \n### **Dependency on MASt3R**\nWe agree to the reviewer that the proposed pipeline highly depends on a reliable pose estimation method. We utilize MASt3R in this manuscript, which performs effectively on all test datasets. Moreover, some advanced methods, e.g., MV-MASt3R, VGGT, etc, were proposed recently and have shown better performance than MASt3R, and therefore, they may potentially provide more reliable pose estimation to our proposed Omni3D approach.\n### **Typos**\nThanks a lot for pointing out the typo. We will undertake a thorough review of the entire manuscript to correct this and any other typographical errors before final submission."}, {"review_id": "juM9eN7K3H", "reviewer": "Reviewer_xEyg", "summary": "The paper introduces Omni3D, a framework for reconstructing a full 360° (omnidirectional) 3-D scene from a single RGB image.\n\nA multi-view diffusion (MVD) model (here a LoRA-tuned CogVideoX) first hallucinates a dense set of novel views arranged on three progressively expanding hemispherical “orbits.”\n\nA Pose–View Optimisation (PVO) module then iteratively refines each generated view and its camera pose simultaneously. PVO treats a pair of views, estimates an initial pose with MASt3R features, renders a 3-D reprojection between the pair, and trains a lightweight network to minimise reprojection error; the pose is re-estimated after each optimisation round.\n\nAfter all views in all orbits are corrected, the set of images with accurate poses and intrinsics is used to train a standard 3D Gaussian splatting (3DGS) scene, enabling real-time omnidirectional rendering.\n\nExperiments on Tanks & Temples, Mip-NeRF 360, and DL3DV-10K demonstrate that Omni3D outperforms ZeroNVS, ViewCrafter, and LiftImage3D, yielding better SSIM and LPIPS results.", "questions": "Please address the weakness listed above.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n- It is new to demonstrably reconstruct an omnidirectional 3-D Gaussian scene from a single image. The synergistic PVO loop (jointly updating geometry and camera) is an interesting way to marry diffusion priors with classical reprojection error.\n\n- Writing: The paper is easy to follow. Figures 2a/2b clearly step through the pipeline.\n\n- Experimental: Three datasets, multiple SOTA baselines, three metrics, plus runtime analysis (Table 3). Ablations isolate PVO and show it generalises to a different backbone (MotionCtrl).\n\nCons:\n- The entire pipeline is a bit cumbersome: MVD + per-pair refine network + MASt3R + PnP Ransac. Most of the components are prior art. The key component, PVO, is also a classical reprojection error-based optimization, although in the new context of point maps. Also, why do you design the refinement network according to the Homography transformation, which has a strong planar geometry assumption? \n\n- The MVD part is only briefly mentioned. How is camera pose being controlled in the MVD, aligning with the target orbit? If the MVD generation somehow follows the orbit, why don't you make use of this initial camera pose in the subsequent geometry correction? Also, is the MVD module frozen?\n\n- Per-pair “over-fitted” refinement network: The lightweight CNN used in PVO is independently over-fitted to every view pair. Because its parameters are never reused, it has zero cross-scene generalisation value. This raises the question: why not directly optimise geometric parameters (e.g., homographies, dense flow fields, or per-pixel depth offsets) and regularise them, instead of introducing an extra network?\n\nPlease clarify the motivation for choosing a network over a purely geometric parameterization. Add an ablation comparing (i) the current per-pair network, (ii) direct optimisation of a dense flow or homography with smoothness regularisation, and (iii) a shared network trained across multiple pairs;\n\n- Evaluation restricted to omnidirectional targets: All quantitative results measure image quality at 360° target views. It remains unclear whether the proposed PVO+3DGS pipeline improves reconstruction quality when the target views lie in other camera distributions—e.g., the forward-facing used by ViewCrafter and LiftImage3D.\n\n- The visual results in Fig. 3 still have many artifacts. This raises the question: Do we really need to make the task so hard? Some sparse-view based solutions like MVSplat360 can achieve much better 360 visual results, which seems more practical.\n\n- Runtime fairness and resource usage: Table 3 reports wall-clock time using 8 × A100 GPUs, but it is not stated whether competing methods were benchmarked with identical hardware or scaled resources. As written, the comparison may conflate algorithmic efficiency with the advantage of more GPUs. Please provide runtimes for all methods on a single A100 (or specify each method’s GPU count and GPU hours) so that computational efficiency can be assessed fairly.", "was_revised": true, "final_justification": "The authors’ substantial efforts in the rebuttal are commendable. Several of my concerns have been addressed—for instance, the evaluations on alternative camera distributions. The rebuttal also reinforces my initial impressions: the proposed pipeline appears overly complex, relies on unscalable per-pair “overfitted” refinement networks, and suffers from slow processing times (as shown in Table 3). Additionally, the requirement to train four separate MVDs further adds to the system’s complexity.\n\nOverall, this work falls into the category of a system-level integration contribution. Although the methodological novelty may be limited, the development of this first framework capable of reconstructing a complete 360° 3D scene from a single RGB image can be appreciated. Thus, I upgrade my initial rating to a borderline accept.", "rebuttal": "We would like to extend our sincere thanks to the reviewer for the valuable and insightful comments, which are instrumental in improving our manuscript. Please find our point-by-point responses to each of the reviewers' comments in the following.\n### **Overall pipeline and Homography**\n* **Overall pipeline.** We agree with the reviewer that the proposed pipeline has the potential to be simplified in future work. One possible way is generating 3D Gaussian splats straight from a single image input by a 3D fundamental model, bypassing the 2D (multi-view images) intermediate steps. This would not only simplify the overall pipeline but also significantly accelerate the inference speed.\n*  **Homography transformation.** Indeed, Homography transformation has a strong planar geometry assumption. However, it achieves image-level viewpoint transformation with only 8 parameters, and therefore can be used as an effective yet time-efficient way for the **coarse** correctness or alignment of camera views, especially when drifting viewpoint in a small angle. Beyond Homography, we also employ flow maps for pixel-level warping and learn residuals for the correctness of each pixel value, which play the role for **finer** correctness of generated views.  As such, we achieve a **hierarchical coarse-to-fine** process of pose-view optimization. Its effectiveness has been validated in the experiments.\n### **Details about MVD**\nThanks for the questions regarding MVD. We will describe the MVD more detailly in the revised manuscript. \n* **Training MVD for four orbits.** Instead of controlling MVD by camera poses, we have trained four MVD models for the four target orbits (left, right, up and down), respectively. Specifically, before training the MVD models, we first reconstruct the 3DGS of each sample in the training set, and then render it along the four target orbits to obtain the training samples of sequential images in the four orbits, respectively. Then, the MVD model for each orbit is trained by the samples of the corresponding orbit. \n* **Initial poses.** Although the MVDs show good performance for multi-view generation along the target orbits, it may inevitably occur slight errors between the generated orbits and the ideal orbits. Therefore, we employ MASt3R to calculate the initial poses of the generated views, instead of the poses predicted by the ideal orbits, to reduce the initial error of pose estimation before the PVO process.\n* **Frozen MVD.** After training the MVD models, they are frozen during the following processes, i.e., MASt3R, PVO and Gaussian optimization.  \n### **Design of the refinement network and ablation studies**\nThanks a lot for the insights on the refinement network. \n* **Lightweight CNN vs. direct optimization of parameters.** In the proposed PVO process, we use a lightweight CNN to overfit the geometric parameters instead of directly optimize the parameters, since we observed instability (loss may increase) during the iterative optimization when directly optimize the geometric parameters themselves (denoted as w/o lightweight CNN). This is probably because that the lightweight CNN increases the nonlinearity of the network and are more compatible with the Adam optimizer, and therefore it is easier to adjust the learning rate during training. In the following, we show the ablation results in Table 1 in this rebuttal. It can be seen from Table 1 that directly optimizing the parameters (w/o lightweight CNN) leads to worse performance than the proposed lightweight CNN due to the worse convergence of PVO. Note that, in the case of instability, PVO has to be early stopped when the loss starts to increase, to avoid the crash of the whole pipeline.\n\n    \\\n    Table 1: Ablation study on the lightweight CNN vs. direct optimization of parameters (w/o lightweight CNN)\n\n    | Methods | Tanks and Temples | Mip-NeRF 360 | DL3DV |\n    |:--------|:-----------------|:------------|:-----|\n    |         | PSNR $\\uparrow\\ \\ \\ $   SSIM $\\uparrow\\ \\ \\ $   LPIPS $\\downarrow\\ \\ \\ $ | PSNR $\\uparrow\\ \\ \\ $  SSIM $\\uparrow\\ \\ \\ $  LPIPS     $\\downarrow\\ \\ \\ $ | PSNR $\\uparrow\\ \\ \\ $  SSIM $\\uparrow\\ \\ \\ $  LPIPS $\\downarrow$ |\n    | Omni3D (w/o lightweight CNN)|    15.92 $\\ \\ \\ \\ \\ $ 0.5227 $\\ \\ \\ \\ $ 0.5175         |        15.21 $\\ \\ \\ \\ \\ $ 0.2605 $\\ \\ \\ \\ $ 0.7262         |        16.57 $\\ \\ \\ \\ \\ $ 0.6244 $\\ \\ \\ \\ $ 0.3890         |\n    | **Omni3D (w/ lightweight CNN)** |    **16.30** $\\ \\ \\ \\ \\ $ **0.5308** $\\ \\ \\ \\ $ **0.5166**  |   **15.89** $\\ \\ \\ \\ \\ $ **0.2859** $\\ \\ \\ \\ $ **0.6369**    |    **17.08** $\\ \\ \\ \\ \\ $ **0.6649** $\\ \\ \\ \\ $ **0.3348**   |  \n    | | | | | |\n   \n* **Replace by a shared network.** If using a pre-trained (shared) network to refine the generated views according to estimated poses, it should learn to extract features from initially estimated poses and the corresponding initial re-projected images, in addition to the input image pair. The current lightweight CNN takes the inputs only the pair of reference and generated views without pose-related conditions, and therefore this network (or even its upscaled version) cannot play the role of a pre-trained network for the refinement of generated views. To employ a shared network, it is necessary to re-design the network structure. During the rebuttal period, it is hard to re-design and propose the pre-trained shared network, which are largely different from the proposed method. This may be an interesting topic that is valuable to be explored in future research.  \n\n### **Evaluation in other camera distributions**\nThanks a lot for the valuable comments on the experimental setup. In this rebuttal, we have conducted additional experiments on the evaluation of rendering quality at forward-facing camera views, following the experimental setup of LiftImage3D. The results are shown in Table 2, where we can see that the proposed Omni3D approach also improves the reconstruction quality for the forward-facing camera views. \n\n\\\nTable 2: Evaluation of rendered views at **_forward-facing camera distributions_**\n\n| Methods | Tanks and Temples | Mip-NeRF 360 | DL3DV |\n|:--------|:-----------------|:------------|:-----|\n|         | PSNR $\\uparrow\\ \\ \\ \\ $   SSIM $\\uparrow\\ \\ \\ $   LPIPS $\\downarrow\\ \\ \\ $ | PSNR $\\uparrow\\ \\ \\ \\ $  SSIM $\\uparrow\\ \\ \\ $  LPIPS     $\\downarrow\\ \\ \\ $ | PSNR $\\uparrow\\ \\ \\ \\ $  SSIM $\\uparrow\\ \\ \\ $  LPIPS $\\downarrow$ |\n| ZeroNVS [33]|         13.98 $\\ \\ \\ \\ \\ $\t0.4856 $\\ \\ \\ \\ $ 0.6225 |\t14.25 $\\ \\ \\ \\ \\ $\t0.2669 $\\ \\ \\ \\ $\t0.8657|\t13.73 $\\ \\ \\ \\ \\ $\t0.5380 $\\ \\ \\ \\ $\t0.6476\n| ViewCrafter [51]| 15.79 $\\ \\ \\ \\ \\ $\t0.5054 $\\ \\ \\ \\ $\t0.5625\t | 15.99 $\\ \\ \\ \\ \\ $\t0.2717 $\\ \\ \\ \\ $\t0.7012\t | 17.85 $\\ \\ \\ \\ \\ $\t0.6459\t$\\ \\ \\ \\ $ 0.3843\n| LiftImage3D [3]|  16.14 $\\ \\ \\ \\ \\ $\t0.5232 $\\ \\ \\ \\ $\t0.5346 | \t16.09 $\\ \\ \\ \\ \\ $\t0.2776 $\\ \\ \\ \\ $ 0.5553 |\t20.77 $\\ \\ \\ \\ \\ $\t0.6834 $\\ \\ \\ \\ $\t0.4557\n| **Our Omni3D** | **18.32** $\\ \\ \\ \\ \\ $\t**0.5919** $\\ \\ \\ \\ $ **0.4689** |\t**16.96** $\\ \\ \\ \\ \\ $\t**0.2880** $\\ \\ \\ \\ $\t**0.4807** | \t**22.02** $\\ \\ \\ \\ \\ $\t**0.7649** $\\ \\ \\ \\ $ **0.3011**\n| | | | | |\n\n\n### **Task setting**\nThanks a lot for the insightful comment on the task setting. Indeed, sparse-view based solutions have been widely studied in the community and have achieved promising results on 360-degree 3D reconstruction. Lifting a single image to the 3D representation (e.g., 3DGS) of the entire omnidirectional space is much more challenging, and meanwhile has the potential to be practical for the scenarios of AI-powered immersive media, such as the 3D experience of online meeting with mobile or laptop camera, and immersive 3D experience of the scene in a photograph. In recent years, there are a number of works in this direction, however, the existing works, e.g., LiftImage3D, ViewCrafter, etc., only effectively reconstruct 3DGS in limited angles, encountering significant hurdles to reconstruct omnidirectional 3D scenes. In this work, we pioneered in omnidirectional 3D reconstruction from single image, thanks to the proposed pipeline and the PVO method in our Omni3D approach.\n### **Comparison of time consumption**\nThank the reviewer very much for the valuable comments on the comparison of time consumption. \n* **Fairness for the comparison on 8 x A100 GPUs.** In the original manuscript, the proposed Omni3D and all compared methods are tested on **identical hardware**, i.e., 8 x A100 GPUs. Moreover, we have also applied **parallelism computation to all compared methods** as significant as possible, such as paralleled MVDs for various orbits, to fully leverage the capabilities of the hardware. Therefore, it ensures fairness in the comparison of time consumption. We will declare this issue more clearly in the revised manuscript.\n* **Time consumption on 1 x A100 GPUs.** Indeed, as suggested by the reviewer, it is necessary to report the computational time on a single GPU as well. We show the results in Table 3 in this rebuttal as below. It can be seen from Table 3 that when the parallelism capability is limited to a single A100 GPU, the proposed Omni3D approach consumes additional 46.2% of computational time compared with LiftImage3D, and still faster than ZeroNVS (_CVPR'24_). However, our Omni3D approach is able to reconstruct the entire omnidirectional 3D space, instead of only the forward-facing views, and also achieves better reconstruction quality. In the revised manuscript, we will include the time consumption results on a single GPU in the paper.\n\n    \\\n    Table 3: Time consumption on a single A100 GPU\n    |                | MVD | Pose calc. | 3DGS | Total |\n    |----------------|-----|------------|------|-------|\n    | ZeroNVS [33]| -   | -          | -    |    133.7 min |\n    | ViewCrafter [51] |   4.3 min  | -          |   12.8 min   |    17.1 min   |\n    | LiftImage3D [3] |   12.0 min |      1.5 min      |  67.4 min    |   80.9 min   |\n    | **Our Omni3D** |  21.6 min   |      83.9 min      |   12.8 min   |   118.3 min    |\n    ||||"}]

Author rebuttal:

Meta-review: This paper introduces Omni3D, a framework for reconstructing omnidirectional 3D scenes from a single image. The key contribution is the Pose–View Optimization (PVO) module, which jointly refines diffusion-generated views and their estimated poses to mitigate distortions and inconsistencies before reconstruction with 3D Gaussian splatting. The method is evaluated on multiple datasets and demonstrates clear improvements over strong baselines, supported by quantitative metrics, qualitative comparisons, and a user study.

The reviewers appreciated the significance of the task, the clarity of presentation, and the systematic design of the proposed PVO. The work was also praised for its comprehensive experiments, generalizability to different diffusion backbones, and supplementary visualizations. Concerns were raised regarding the incremental nature of the contributions, as parts of the pipeline resemble LiftImage3D, as well as the complexity and efficiency of the overall system. However, the authors addressed these points in rebuttal with additional experiments (forward-facing views, ablations, runtime clarifications) and a perceptual user study, which strengthened confidence in the results and mitigated most concerns.

Overall, while efficiency remains a limitation, the paper makes a meaningful step forward by demonstrating that omnidirectional 3D reconstruction from a single image is feasible. The PVO framework provides a principled and generalizable solution, and the empirical results substantiate its effectiveness.  The final recommendation by the Area Chairs is to accept the submission.
Decision: Accept (poster)