Title: Omnidirectional 3D Scene Reconstruction from Single Image
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Reconstruction

Diffusion

Gaussian Splatting
Abstract: Reconstruction of 3D scenes from a single image is a crucial step towards enabling next-generation AI-powered immersive experiences. However, existing diffusion-based methods often struggle with reconstructing omnidirectional scenes due to geometric distortions and inconsistencies across the generated novel views, hindering accurate 3D recovery. To overcome this challenge, we propose Omni3D, an approach designed to enhance the geometric fidelity of diffusion-generated views for robust omnidirectional reconstruction. Our method leverages priors from pose estimation techniques, such as MASt3R, to iteratively refine both the generated novel views and their estimated camera poses. Specifically, we minimize the 3D reprojection errors between paired views to optimize the generated images, and simultaneously, correct the pose estimation based on the refined views. This synergistic optimization process yields geometrically consistent views and accurate poses, which are then used to build an explicit 3D Gaussian Splatting representation capable of omnidirectional rendering. Experimental results validate the effectiveness of Omni3D, demonstrating significantly advanced 3D reconstruction quality in the omnidirectional space, compared to previous state-of-the-art methods. Project page: https://omni3d-neurips.github.io.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Omni3D, a framework for reconstructing a full 360° (omnidirectional) 3-D scene from a single RGB image.\n\nA multi-view diffusion (MVD) model (here a LoRA-tuned CogVideoX) first hallucinates a dense set of novel views arranged on three progressively expanding hemispherical “orbits.”\n\nA Pose–View Optimisation (PVO) module then iteratively refines each generated view and its camera pose simultaneously. PVO treats a pair of views, estimates an initial pose with MASt3R features, renders a 3-D reprojection between the pair, and trains a lightweight network to minimise reprojection error; the pose is re-estimated after each optimisation round.\n\nAfter all views in all orbits are corrected, the set of images with accurate poses and intrinsics is used to train a standard 3D Gaussian splatting (3DGS) scene, enabling real-time omnidirectional rendering.\n\nExperiments on Tanks & Temples, Mip-NeRF 360, and DL3DV-10K demonstrate that Omni3D outperforms ZeroNVS, ViewCrafter, and LiftImage3D, yielding better SSIM and LPIPS results.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n- It is new to demonstrably reconstruct an omnidirectional 3-D Gaussian scene from a single image. The synergistic PVO loop (jointly updating geometry and camera) is an interesting way to marry diffusion priors with classical reprojection error.\n\n- Writing: The paper is easy to follow. Figures 2a/2b clearly step through the pipeline.\n\n- Experimental: Three datasets, multiple SOTA baselines, three metrics, plus runtime analysis (Table 3). Ablations isolate PVO and show it generalises to a different backbone (MotionCtrl).\n\nCons:\n- The entire pipeline is a bit cumbersome: MVD + per-pair refine network + MASt3R + PnP Ransac. Most of the components are prior art. The key component, PVO, is also a classical reprojection error-based optimization, although in the new context of point maps. Also, why do you design the refinement network according to the Homography transformation, which has a strong planar geometry assumption? \n\n- The MVD part is only briefly mentioned. How is camera pose being controlled in the MVD, aligning with the target orbit? If the MVD generation somehow follows the orbit, why don't you make use of this initial camera pose in the subsequent geometry correction? Also, is the MVD module frozen?\n\n- Per-pair “over-fitted” refinement network: The lightweight CNN used in PVO is independently over-fitted to every view pair. Because its parameters are never reused, it has zero cross-scene generalisation value. This raises the question: why not directly optimise geometric parameters (e.g., homographies, dense flow fields, or per-pixel depth offsets) and regularise them, instead of introducing an extra network?\n\nPlease clarify the motivation for choosing a network over a purely geometric parameterization. Add an ablation comparing (i) the current per-pair network, (ii) direct optimisation of a dense flow or homography with smoothness regularisation, and (iii) a shared network trained across multiple pairs;\n\n- Evaluation restricted to omnidirectional targets: All quantitative results measure image quality at 360° target views. It remains unclear whether the proposed PVO+3DGS pipeline improves reconstruction quality when the target views lie in other camera distributions—e.g., the forward-facing used by ViewCrafter and LiftImage3D.\n\n- The visual results in Fig. 3 still have many artifacts. This raises the question: Do we really need to make the task so hard? Some sparse-view based solutions like MVSplat360 can achieve much better 360 visual results, which seems more practical.\n\n- Runtime fairness and resource usage: Table 3 reports wall-clock time using 8 × A100 GPUs, but it is not stated whether competing methods were benchmarked with identical hardware or scaled resources. As written, the comparison may conflate algorithmic efficiency with the advantage of more GPUs. Please provide runtimes for all methods on a single A100 (or specify each method’s GPU count and GPU hours) so that computational efficiency can be assessed fairly.\n\nQUESTIONS:\nPlease address the weakness listed above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to tackle the challenge of single-image 3D reconstruction by integrating the diffusion-based novel view synthesis methods and a pose refinement loop, named Pose-View Optimization (PVO). The proposed method progressively synthesizes sets of novel views around the input view, or around optimized views with a larger disparity to the input one. Then, the PVO module iteratively refines the pose pair of the novel views by PnP, where the 3D points are estimated from a pretrained MASt3R model. This core component yields a set of more geometry-consistent posed novel views. Finally, these pose-optimized views are fed into 3DGS pipelines to generate an omnidirectional scene representation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- An innovative design to overcome the geometry-conflicting distortion introduced by diffusion methods. The lightweight network within this design estimates the residual on top of the diffused view, which is further improved by jointly optimizing its pose.\n- This method does not reply on the choice of the novel view diffusion methods, thereby providing a general framework that can be used for omnidirectional reconstruction from single view inputs.\n- The splitting strategy allows a good parallelism and makes this method applicable.\n- The manuscript is well written.\n### Weakness:\n- Highly dependent on the quality of MASt3R. If the point map estimated by the feed-forward method has poor quality, the whole pipeline may fail.\n- L173, \"expect\" -> \"except\"\n\nQUESTIONS:\n- Can this method generalize to scenes other than object-centric, e.g. forward-facing scene generation?\n- How long does it take to optimize the lightweight network?\n- How much does the optimized poses drift from the input ones? If not, why not register the four optimized poses (front, left, right, back) to the ground truth poses followed by evaluating the rendered views at registered poses?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of omnidirectional 3D scene reconstruction from a single image, identifying common issues in existing diffusion-based methods such as geometric distortion and inconsistencies. To overcome these, the authors propose Omni3D, which leverages pose estimation priors and a Pose-View Optimization (PVO) module to iteratively refine novel views and their estimated camera poses. Experimental results demonstrate significant improvement over existing state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The task of single-image 3D reconstruction is significant for future AI-powered immersive media. The paper clearly identifies critical issues in current methods and proposes targeted solutions.\n\n2. The iterative Pose-View Optimization (PVO) approach, which jointly optimizes pose estimation and generated views by minimizing 3D reprojection errors, is well-motivated.\n\n3. The video in the supplementary material shows good performance.\n\nWeaknesses:\n\n1. This submission is very similar to LiftImage3D from motivation to methodology. The differences between Omni3D and LiftImage3D must be explicitly clarified to effectively highlight the original contributions of this paper. Currently, the incremental nature of contributions diminishes the novelty of the submission.\n\n2. The reported experimental results are notably lower than those presented in the original papers being compared. This inconsistency raises concerns regarding experimental reliability and evaluation fairness. It is crucial to unify experimental setups and perform repeated evaluations to ensure the credibility of the results.\n\n3. Although parallelization strategies are mentioned, the computational complexity and inference time still appear substantial. The paper lacks detailed discussions on potential optimizations or approximations that could effectively reduce computational costs.\n\nQUESTIONS:\n1. This submission is very similar to LiftImage3D from motivation to method. Please clarify the difference between the two, which is very important to highlight the contribution of this submission.\n\n2. Are the experimental setup and comparison methods consistent? Why are the results reported in this article so much lower than those reported in the original paper? It is recommended to unify the evaluation, as this is crucial to enhancing the credibility of the submission. If the uncertainty is due to random selection, then repeated verification is necessary.\n\n3. Include more detailed discussions about computational costs and potential optimizations or approximations to reduce inference time.\n\nMy main concern is the contribution of this paper and the reliability of the experimental data. Currently, the contribution of this paper is incremental compared to previous methods, and the experimental results do not appear convincing. I am happy to see the author's clarification and response.\n\nLIMITATIONS:\n1. Omni3D may require substantial computational resources, potentially limiting deployment in real-time or resource-constrained scenarios.\n\n2. The method's iterative refinement strategy may accumulate errors, especially for extreme viewpoints or highly complex scene geometries.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes a novel method for single-image to 3D reconstruction, outperforming prior work. The method uses a multi-view diffusion model to create many views from an input image. The authors then propose a "Pose-View Optimization module" to refine both the camera poses (estimated via MASt3R) and the generated views. This process starts with nearby views and then expands to farther views. Finally, once this has been done for each region (both nearby and far views), 3DGS is used for 3D reconstruction.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The proposed method seems well designed. In particular, the design of the progressive pairing strategy is well thought out and seems like a valuable contribution for future work to build upon.
- In addition to the point above, I appreciated the ablation on the Pose-View Optimization module, as it helped support the benefit of the module.
- The quantitative results support the efficacy of the proposed method.

**Weaknesses**
- I was a bit underwhelmed by the qualitative results. The paper would benefit from a user study to see if the reported metrics (PSNR/SSIM/LPIPS) align. For me, I did not notice significant improvements in the images from the proposed method. Perhaps identifying areas that are significantly better (with a box) and showing a zoomed in view of that region would be helpful for readers. 
- Additional ablations on the progressive pairing strategy and different hyperparameters would be informative.

QUESTIONS:
Overall, I like this paper and am inclined towards acceptance. My main concern is I did not notice much improvement in the qualitative results, but I still feel the ideas from the method are a useful contribution.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03