Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images

## Abstract

Abstract This paper presents Omni-View, which extends the unified multimodal understanding and generation to 3D scenes based on multiview images, exploring the principle that “generation facilitates understanding”. Consisting of understanding model, texture module, and geometry module, Omni-View jointly models scene understanding, novel view synthesis, and geometry estimation, enabling synergistic interaction between 3D scene understanding and generation tasks. By design, it leverages the spatiotemporal modeling capabilities of its texture module responsible for appearance synthesis, alongside the explicit geometric constraints provided by its dedicated geometry module, thereby enriching the model’s holistic understanding of 3D scenes. Trained with a two-stage strategy, Omni-View achieves a state-of-the-art score of 55.4 on the VSI-Bench benchmark, outperforming existing specialized 3D understanding models, while simultaneously delivering strong performance in both novel view synthesis and 3D scene generation.

## 1 Introduction

Recently, unified multimodal models (UMMs) (team2024chameleon; xie2024show; deng2025bagel) have emerged as a pivotal area of research. The primary goal is to empower multimodal large language models (MLLMs) to both understand and generate visual signals present in our world, laying the foundation for artificial general intelligence.

Numerous methods (team2024chameleon; wu2024janus; wu2024vila; xie2024show; chen2025blip3; deng2025bagel) have achieved coexistence between 2D image understanding and generation. Among them, some studies (wu2024liquid; tong2024metamorph; yan2025can) aim to improve model performance by exploring the synergies arising from the interaction between understanding and generation. For example, Metamorph (tong2024metamorph) has extensively validated the role of understanding in improving generation performance. However, the potential and effectiveness of generation to improve understanding capabilities within unified models remain underexplored.

Due to the intrinsic geometric and spatiotemporal nature of 3D scenes and their multiview images, generative tasks such as geometry estimation and novel view synthesis are particularly well-suited for facilitating understanding in the 3D domain. This is attributable to the fact that 3D understanding tasks (yang2024thinking; zhang2025flatland) inherently necessitate robust geometric and spatiotemporal modeling capabilities, which can be effectively acquired through these generative tasks. Moreover, biomedical evidence (maus2013motion; nortmann2015primary) suggests that human understanding of 3D environments is governed by the capacity to generate and imagine future sensory and geometric data (keller2012sensorimotor; leinweber2017sensorimotor) in the observed scene. These findings provide us with a guidance: the “generation facilitates understanding” paradigm presents a promising approach to building a unified model for 3D scene understanding and generation.

Inspired by the above analysis, this paper aims to fully unlock and maximize the benefits of generation to understanding, thereby constructing a unified model for 3D scene understanding and generation, called Omni-View. We emphasize that geometry estimation and novel view synthesis can leverage their inherent geometric measurement and spatiotemporal modeling capabilities to improve 3D scene understanding, localization, and spatial reasoning. Specifically, we achieve this through two key aspects, architectural design and training strategy.

The architectural design aims at unifying 3D scene understanding and generation, leveraging geometry estimation and novel view synthesis to advance 3D scene understanding. Our Omni-View is built on Bagel (deng2025bagel), a strong unified framework in which the interaction between understanding and generation is facilitated by its shared multimodal self-attention. However, Bagel’s generative capacity is limited to RGB images, thus only capturing texture information, whereas 3D scene generation necessitates inclusion of both texture and geometric structure. In accordance with the dual generative objectives, the generation model in Omni-View is split into two distinct modules: a texture module and a geometry module. The texture module receives reference images, a list of targeted camera poses, and prompt tokens encoded by the understanding model to generate novel views of the scene. Meanwhile, the geometry module employs the hidden features from the understanding model and the latent output of the texture module to infer geometric information of novel views, such as depth maps and camera poses. This dual-pathway architecture empowers the model to develop both geometric and spatiotemporal modeling capabilities, which are essential for 3D scene understanding tasks.

The principal goal of training strategy is to comprehensively improve the performance of the model. A two-stage training strategy is employed. The first stage aims to augment the benefits of generation for understanding 3D scenes, as introduced by the proposed architecture. The subsequent stage is intended to refine the generation performance. In stage 1, the understanding model, geometry module, and texture module are trained simultaneously. Geometry estimation assists the model in comprehending the relative positional relationships among objects, thus directly enhancing the model’s capability to evaluate the relative distances and directions of objects in 3D scenes. The autoregressive generation forces the understanding model to discern the spatiotemporal relations between the generated novel views, thus improving its understanding capabilities. As iterations progress in stage 1, the number of reference images gradually decreases. This progressive shift from dense to sparse views supports a curriculum-like, easy-to-difficult training approach, ultimately enhancing the performance of the understanding model. In stage 2, the understanding model is forzen. The generation model is finetuned via RGB-Depth-Pose joint generation, thereby enhancing its capabilities in 3D scene generation.

We evaluate Omni-View on scene understanding, spatial reasoning, and novel view synthesis tasks. The model achieves an impressive score of 55.4 on the VSI-Bench, exceeding current MLLMs designed for visual reasoning. It manifests particularly notable improvements in subtasks such as Relative Distance and Appearance Order, which require spatiotemporal modeling and the estimation of geometry acquired through generation tasks. It also exhibits superior performance compared to existing 3D understanding MLLMs in the area of 3D question answering (ma2023sqa3d; azuma2022scanqa). Furthermore, it effectively narrows the performance gap between unified models and specialized models focused on 3D visual localization (chen2020scanrefer). Furthermore, we attain robust results in the domain of novel view synthesis and scene generation, with particular emphasis on enhanced perceptual quality.

## 2 Related work

Scene understanding. Recent advances in understanding 3D scenes have been greatly provided by incorporating 3D or video input and 3D reconstruction prior. LLaVA-3D (zhu2024llava3d) and GPT4Scene (qi2025gpt4scene) function within the voxel space and BEV. Video3DLLM (zheng2024video3dllm) improves localization ability by encoding 3D coordinates as position embeddings, while Ross3D (wang2025ross3d) improves 3D understanding through visual-centric reconstruction. However, the dependency on 3D input of these methods poses practical application challenges. To alleviate this issue, VG-LLM (zheng2025learning) and SpatialMLLM (wu2025spatial) use the features of VGGT (wang2025vggt) as input, embedding the 3D piror in the model.

Scene generation. A vital element of scene generation methods is the presence of an efficient proxy to represent the 3D scene, with panoramic images (team2025hunyuanworld), point clouds (yu2024viewcrafter), and Gaussian Splatting (yu2024wonderworld) among the viable options. In consideration of the advances in image and video diffusion models (rombach2022high; wan2025wan), contemporary 3D scene generation strategies reconstruct the scene’s geometry from a single view, employing conditioned video diffusion models to render the scene’s texture. ViewCrafter (yu2024viewcrafter) brings explicit 3D information (point cloud) to generation models through iterative reconstruction. WonderJourney (yu2024wonderjourney) generates a comprehensive set of view sequences. Voyager (huang2025voyager) reconstructs the scene from a single view and uses it as conditions for inpainting.

Unified understanding and generation. In the domain of 3D scenes, a unified model for understanding and generation applicable in general scenarios remains absent. Hermes (zhou2025hermes) uses BEVs to design a unified understanding and generation model for autonomous driving. In contrast, significant progress has been made in 2D vision (zhang2025unified). These methods show primarily variations in architectures and training strategies. Chameleon (team2024chameleon) utilizes VQ-VAE for image tokenization, thereby improving generation competencies. However, its understanding capabilities are inferior to those of Janus (wu2024janus), which employs SigLIP (zhai2023sigmoid) as visual understanding encoder. VILA-U (wu2024vila) integrates understanding and generation within the image encoder, bypassing multitask gradient conflicts during MLLM training. BAGEL (deng2025bagel) implements task-based hard routing for MLLM, which also avoids gradient conflicts. Harmon (wu2025harmonizing) takes advantage of MAE’s reconstruction ability and downstream understanding enhancement. BLIP3o (chen2025blip3) introduces “understand first, then generation” training paradigm, thus achieving performance gains. Building on the progress in the 2D domain, this paper investigates a unified model in 3D scenes and explores how generation aids the understanding scheme within the framework.

## 3 Method

### 3.1 Archtitectures

Figure: Figure 1: Architecture of Omni-View. Building upon Bagel (deng2025bagel), Omni-View consists of an understanding model and a generation model. The generation model is further composed of two specialized modules: one for texture and one for geometry. Trained via a two-stage process, Omni-View shows high effectiveness in scene understanding and novel view synthesis. Crucially, it unlocks the benefits of its generative capabilities to enhance the model’s understanding performance.
Refer to caption: x1.png

The architecture of Omni-View, as illustrated in Figure [1](https://arxiv.org/html/2511.07222v1#S3.F1), is divided into two distinct components. The left component, termed the understanding model, is tasked with understanding the multiview images of 3D scenes and producing text feedback. The right component is comprised of two integral parts: the texture module and the geometry module, which facilitate the synthesis of the RGB image and the associated geometric data.

Understanding Model. The understanding model is required to perform 3D scene or spatial comprehension tasks, including question answering, localization, and reasoning. This necessitates the model to extract semantic information from spatial-temporal data while measuring the geometric characteristics of the 3D scene. Specifically, given $n$ multiview images and a question $T_{ques}$, the understanding model produces the textual answer $T_{ans}$ through the next token prediction. In this process, the images are encoded by SigLIP (zhai2023sigmoid) and the text is tokenized by a frozen vocabulary $\tau(\cdot)$. Subsequently, tokens of multiview images and question $\tau(T_{ques})$ will be sequentially concatenated and fed into the understanding model. Moreover, the understanding model can also produce intermediate features, which will be used in the geometry module.

Texture Module. The texture module in the generation model is tasked with novel view synthesis using flow matching (lipman2022flow). This module processes a textual description $T_{des}$ and some reference images $I_{ref}$ of the current scene, together with a specified camera pose, to produce consistent novel views of this scene. Within this module, the reference images $I_{ref}$ are encoded using the FLUX-VAE encoder (flux2024) $\varepsilon(\cdot)$, and the vocabulary used to tokenize $T_{des}$ aligns with that used by the understandimg model. This process incorporates the camera pose control as delineated in MV-AR (hu2025auto), embedding the Plucker-Ray encoding $r_{i,j}=(o\times d,d)$ of the camera pose as the absolute position encoding in the model, where $o$ and $d$ donate the origin and direction of the ray, $(i,j)$ represents the pixel coordinate. This camera pose embedding exhibits adaptability to various image resolutions and demonstrates significant flexibility. The above process can be described as:

$$ $F_{tex}=\text{Texture Module}\ ([\text{LM Head}(\tau(T_{des}));[\varepsilon(I_{ref});N_{tex}]+r]),$ (1) $$

where $F_{tex}$ is the predicted image noise of the texture module, $N_{tex}$ is the random input noise, and LM Head is the text processing module of the understanding model.

Geometry Module. The geometry module in the generation model constructs the geometric aspects of the generated images in the texture module. It synthesizes depth maps through flow matching and employs a learnable query strategy to accurately estimate camera poses. It receives the latent of novel view images $F_{tex}$ from the texture module as input, which is concatenated with random depth noise $N_{dep}$ and a learnable camera pose query $q_{cam}$ along the frame dimension. After processing, $N_{dep}$ will be denoised as depth maps’ latent, while $q_{cam}$ will be decoded to reveal the intrinsic and extrinsics of the camera. The features of novel view images $F_{und}$, output from the understanding model’s block, is also integrated into the geometry module through cross-attention. The process in the geometry module can be described as follows.

$$ $[F_{dep};\hat{g}]=\text{Geometry Module}\ ([F_{tex};N_{dep};q_{cam}],F_{und}),$ (2) $$

where $F_{dep}$ is the predicted depth noise of the geometry module. $\hat{g}$ is the predicted intrinsics and extrinsics of the camera, which is decoded by VGGT’s camera decoder (wang2025vggt).

The geometry module has independent parameters and maintains architectural connections with both the texture module and the understanding model. Unlike the connection method in BAGEL, the geometry module uniformly extracts features from the understanding model at the layer dimension as conditions. It utilizes cross-attention for fusion and ensures that the gradients of geometry estimation can be backpropagated to the understanding model. Currently, the geometry module’s input only relies on the last-layer output latent of the texture module. This approach guarantees that it acquires information closest to the image modality, thereby providing finely aligned features for the estimation of depth map and camera pose.

### 3.2 Training Recipe

The Omni-View training recipe has two separate stages, as shown in the upper left of Figure [1](https://arxiv.org/html/2511.07222v1#S3.F1).

Stage 1: unify 3D understanding and generation. In stage 1, we train the understanding model, the texture module, and the geometry module simultaneously. It can leverage the fine-grained geometry estimation capability in the geometry module and combines it with the spatial-temporal modeling ability within the texture module, thus improving the 3D understanding performance.

Understanding model. For the understanding model, we use the next token prediction to predict the distribution of the answer text given the distribution of the multiview images and query text. The loss function of the understanding model can be expressed as follows.

$$ $L_{und}=-\sum^{T}_{i=1}logP_{\theta}(y_{i}|y_{<i}),$ (3) $$

where $y$ is the multimodal sequence that contains tokenized multiview images, query text, and answer text.

Texture module. The loss function for the texture module is defined as the Mean Squared Error (MSE) loss between each predicted texture noise $F_{tex}$ generated by the texture module and the provided texture noise $N_{tex}$.

$$ $L_{tex}=||F_{tex}-N_{tex}||_{2}.$ (4) $$

In contrast to the majority of novel view synthesis methods, the texture model employs a autoregressive generation framework. Specifically, during the generation of the $n$-th frame, the model is exposed to the visual data of the preceding $n-1$ frames, while excluding any subsequent frames. This autoregressive methodology enables the model to fully grasp the concept of temporal sequences, thereby enhancing its spatiotemporal modeling proficiency and effectively improving scene understanding. To optimize the 3D consistency of the sampled images, diffusion forcing (chen2024diffusion) is employed when training the texture module.

To acquire this complex spatiotemporal generation capability incrementally, we gradually adjust the reference images. As iterations progress, the reference images are systematically reduced in a stepwise manner, transitioning from encompassing all input images, excluding the first image, to including only the first image. This implies that for the model, the reference confidence transitions from dense to sparse. We designate this progressive training approach as dense-to-sparse (D2S). This strategy has been shown to be highly effective in facilitating improved understanding.

Geometry module. In stage 1, the geometry module is tasked with estimating the depth and pose of the camera from both the provided images and those synthesized by the texture module. The loss function for the geometry module comprises a sum of the depth estimation loss and the camera pose estimation loss. In terms of depth estimation, we apply the MSE loss to the comparison between the depth noise $F_{dep}$ predicted by the geometry module and the given depth noise $N_{dep}$. The estimated intrinsics and extrinsics of the camera $\hat{g}$ are optimized directly through the Huber loss.

$$ $L_{geo}=||F_{dep}-N_{dep}||_{2}+||\hat{g}-g_{gt}||_{\epsilon},$ (5) $$

where $g_{gt}$ is the ground truth of the camera pose and $||\cdot||_{\epsilon}$ donates the Huber loss.

Ultimately, a weighted summation of the aforementioned losses is conducted to derive the loss function in stage 1 $L_{s1}$.

$$ $L_{s1}=\lambda_{und}L_{und}+\lambda_{tex}L_{tex}+\lambda_{geo}L_{geo},$ (6) $$

where $\lambda_{und},\lambda_{tex},\text{and }\lambda_{geo}$ represent the weighting coefficients and their default values are 1, 1, and 0.1 respectively.

Stage 2: advance generation. In stage 2, the texture module and the geometry modules are trained. The RGB-Depth-Pose (RGBDP) joint learning methodology is used for training, capitalizing on the geometry prior obtained from depth-pose estimation to enhance the ability to generate consistent appearances for novel views.

In the texture module, the reference single-view image is used along with its depth map to reconstruct the initial point cloud of the scene. The images rendered from different views, projected through this point cloud, serve as conditions following Voyager (huang2025voyager). For the geometry module, it generates the depth map and camera pose from images synthesized by the texture module. Concurrently, it no longer relies on the features of the understanding model as conditions for cross-attention.

## 4 Experience

### 4.1 Experimental Setup

In our experiments, the understanding model and the texture module in the generation model are initialized using the pre-trained BAGEL-7B (deng2025bagel). The geometry module in the generation model is configured to have the same dimensions as the texture module, with a depth of four layers. For 3D scene understanding, a filtered dataset comprising 780k valid items, sourced from SQA3D (ma2023sqa3d), ScanQA (azuma2022scanqa), 3DOD (zheng2025learning), ScanRefer (chen2020scanrefer), VLM-3R (fan2025vlm), SPAR (zhang2025flatland) (234k subset) and llava-hound4 (zhang2024llavavideo) (64k subset), is meticulously curated for training. Regarding novel view synthesis, 61k video clips are carefully selected from re10k (zhou2018stereo). The corresponding depth maps are synthesized using the Voyager data pipeline (huang2025voyager) and captions are synthesized using the QwenVLMax (Bai2025Qwen25VLTR). During training, we do not use images from the scene understanding task to train the generation model to fully demonstrate that the understanding performance improvement brought by Omni-View does not come from memorizing the data in understanding tasks.

Model training was performed using the AdamW optimizer, characterized by $\beta_{1}=0.9,\beta_{2}=0.95$, the peak learning rate of $1\times 10^{-5}$. The warm-up phase that constitutes 5% of the whole training iterations. The training process is completed after one epoch of the understanding dataset. For the understanding model, we do not rely on any 3D scene input to support both 3D scene understanding and spatial reasoning tasks. In main comparisons, we use the same checkpoint for testing all tasks.

### 4.2 Main Comparisons

#### 4.2.1 3D scene understanding

Benchmarks. We evaluate the 3D understanding performance of the model on question answering (ma2023sqa3d; azuma2022scanqa) and localization (chen2020scanrefer; zheng2025learning). During inference, we set the frame numbers as 32 following Video3DLLM (zheng2024video3dllm).

Comparison baselines. We compare Omni-View with models specifically designed for 2D or 3D visual understanding tasks (internvl2; wang2024qwen2vl; zhang2024llavavideo; huang2023leo; zhang2024chatscene; chen2024grounded; zhu2024llava3d; zheng2024video3dllm; qi2025gpt4scene; wang2025ross3d), as well as with some unified models applicable to video modalities (deng2025bagel). Unified models are evaluated after fine-tuning with the same data. Within the results table, we differentiate between models that incorporate explicit 3D input and those that do not. Although the incorporation of explicit 3D input improves model performance, it concurrently restricts applicability (zheng2024video3dllm).

**Table 1: Evaluation of 3D scene understanding. “–” indicates the number is not available for us. Bold and underline denote the best and second-best models without 3D scene input, respectively. For ScanRefer, the content in “()” indicates results without proposal refinement (zhang2025flatland).**
| Methods | 3D Input | SQA3D${}_{\text{test}}$ | ScanQA${}_{\text{val}}$ | 3DOD | ScanRefer |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EM | EM-R | C | B-4 | M | R | EM | F1 | Acc@0.25 | Acc@0.5 |  |  |
| \rowcolor[gray]0.9      Task-specific Models |  |  |  |  |  |  |  |  |  |  |  |
| LEO | ✔ | 50.0 | 52.4 | 80.0 | 11.5 | 16.2 | 39.3 | 21.5 | – | – | – |
| ChatScene | ✔ | 54.6 | 57.5 | 87.7 | 14.3 | 18.0 | 41.6 | 21.6 | – | 55.5 | 50.2 |
| Grounded 3D-LLM | ✔ | – | – | – | – | – | – | – | – | 47.9 | 44.1 |
| Video-3D-LLM | ✔ | 58.6 | – | 102.1 | 16.4 | 20.0 | 49.3 | 30.1 | – | 58.1 | 51.7 |
| GPT4Scene-HDM | ✔ | 59.4 | 62.4 | 96.3 | 15.5 | 18.9 | 46.5 | – | – | 62.6 | 57.0 |
| Ross3D | ✔ | 63.0 | 65.7 | 107.0 | 17.9 | 20.9 | 50.7 | 30.8 | – | 61.1 | 54.4 |
| LLaVA-3D | ✔ | 60.1 | – | 103.1 | 16.4 | 20.8 | 49.6 | 30.6 | – | 50.1 | 42.7 |
| InternVL2-8B | ✘ | 33.0 | 45.3 | 62.5 | 3.3 | 14.5 | 34.3 | – | – | – | – |
| Qwen2-VL-7B | ✘ | 48.5 | – | 53.9 | 3.0 | 11.4 | 29.3 | – | – | – | – |
| LLaVA-Video-7B | ✘ | 48.5 | – | 88.7 | 3.1 | 17.7 | 44.6 | – | – | – | – |
| SPAR-7B | ✘ | 58.1 | – | 90.7 | 15.3 | – | – | – | – | 48.8 (31.9) | 43.1 (12.4) |
| VG-LLM-4B | ✘ | – | – | – | – | – | – | – | 47.0 | 53.5 (36.4) | 47.5(11.8) |
| SpatialMLLM-4B | ✘ | 55.9 | 58.7 | 91.8 | 14.8 | 18.4 | 45.0 | – | – | – | – |
| black!30black<br>\rowcolor[gray]0.9             Unified Models |  |  |  |  |  |  |  |  |  |  |  |
| BAGEL-7B-FT | ✘ | 57.2 | 59.7 | 95.5 | 14.7 | 18.7 | 46.3 | 27.0 | 41.3 | 46.9 (28.0) | 41.6 (7.7) |
| Omni-View-7B | ✘ | 59.2 | 61.9 | 103.0 | 16.2 | 20.1 | 49.0 | 29.5 | 46.4 | 50.8(32.5) | 45.0(9.9) |

#### 4.2.2 3D spatial reasoning

Benchmarks. We evaluate the 3D spatial reasoning ability of the model in VSI-Bench (yang2024thinking). During inference, we follow the VSI-Bench to set frame numbers ranging from 8 to 32 and frame resolution to 640p.

Comparison baselines. We compare our Omni-View with models specifically designed for 2D or 3D visual understanding tasks (Xue2024LongVILASL; Zhang2024LongCT; Bai2025Qwen25VLTR; ray2024sat; zhang2025flatland; zheng2025learning; wu2025spatial), as well as with some unified models applicable to video modalities (gpt4o; team2024gemini; deng2025bagel; xie2025show). Unified models are evaluated after fine-tuning with the same data we used.

**Table 2: Evaluation of spatial reasoning on VSI-Bench. “–” indicates the number is not available for us. Bold and underline denote the best and second-best models, respectively.**
| Methods | Numerical Qusetion | Multiple-Choice Question | Avg. |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Obj. Cnt. | Abs. Dist. | Obj. Size | Room Size | Rel. Dist. | Rel. Dir. | Route Plan | Appr. Order |  |  |
| \rowcolor[gray]0.9    Task-spefic Models |  |  |  |  |  |  |  |  |  |
| LongVILA-8B | 29.1 | 9.1 | 16.7 | 0.0 | 29.6 | 30.7 | 32.5 | 25.5 | 21.6 |
| LongVA-7B | 38.0 | 16.6 | 38.9 | 22.2 | 33.1 | 43.3 | 25.4 | 15.7 | 29.2 |
| LLaVA-OneVision-72B | 43.5 | 23.9 | 57.6 | 37.5 | 42.5 | 39.9 | 32.5 | 44.6 | 40.2 |
| LLaVA-Video-72B | 48.9 | 22.8 | 57.4 | 35.3 | 42.4 | 36.7 | 35.0 | 48.6 | 40.9 |
| Qwen2.5VL-7B | 40.9 | 14.8 | 43.4 | 10.7 | 38.6 | 38.5 | 33.0 | 29.8 | 33.0 |
| Qwen2.5VL-72B | 25.1 | 29.3 | 54.5 | 38.8 | 38.2 | 37.0 | 34.0 | 28.9 | 37.0 |
| SAT-LLaVA-Video-7B | – | – | – | 47.3 | 41.1 | 37.1 | 36.1 | 40.4 | – |
| SPAR-8B | – | – | – | – | – | - | - | – | 41.1 |
| VG-LLM-4B | 66.4 | 36.6 | 55.2 | 56.3 | 40.8 | 43.4 | 30.4 | 39.5 | 46.1 |
| Spatial-MLLM-4B | 65.3 | 34.8 | 63.1 | 45.1 | 41.3 | 46.2 | 33.5 | 46.3 | 48.4 |
| \rowcolor[gray]0.9    Unified Models |  |  |  |  |  |  |  |  |  |
| GPT-4o (API) | 46.2 | 5.3 | 43.8 | 38.2 | 37.0 | 41.3 | 31.5 | 28.5 | 34.0 |
| Gemini-1.5 Pro (API) | 56.2 | 30.9 | 64.1 | 43.6 | 51.3 | 46.3 | 36.0 | 34.6 | 45.4 |
| BAGEL-7B-FT | 62.8 | 36.3 | 56.4 | 49.7 | 46.1 | 49.4 | 26.8 | 43.1 | 46.3 |
| Omni-View-7B | 70.3 | 46.4 | 68.6 | 54.7 | 65.9 | 54.4 | 33.5 | 49.0 | 55.4 |

#### 4.2.3 3D scene generation

Benchmarks. The model’s generation capacity is validated under two tasks: novel view synthesis (NVS) from a single view and scene generation. NVS from a single view necessitates that the model generate the subsequent 25 frames from the first image. 3D scene generation evaluates scenes that are reconstructed from the videos generated to 3DGS. The test scenes are randomly selected from the Re10k test set following chen2025flexworld.

Comparison baselines. We compare with scene generation methods (chung2023luciddreamer; wang2024motionctrl; ma2025you; yu2024viewcrafter; chen2025flexworld; huang2025voyager) and a unified model (deng2025bagel). To evaluate the performance of scene generation, we use Dust3R to reconstruct the generated videos, ensuring fairness in the comparison, according to chen2025flexworld. The evaluation is only conducted on the first 25 frames generated by the model following yu2024viewcrafter; zhai2025stargen.

NVS from single view and scene generation. Omni-View achieved the highest PSNR and SSIM, and lowest LPIPS score, indicating that its image quality could surpass that of other methods. However, in terms of pixel-level fidelity, Omni-View shows only slight improvements over popular scene generation models. This discrepancy may be attributed to Omni-View’s challenges in being precisely controlled via the camera pose. Visualization results are presented in the Appendix due to page limit.

**Table 3: Evaluation of novel view synthesis from single view and scene generation on Re10k.**
| Methods | NVS from Single View | Scene Generation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| \rowcolor[gray]0.9    Task-specific Models |  |  |  |  |  |  |
| LucidDreamer | $22.27$ | $0.766$ | $0.204$ | $21.98$ | $0.698$ | $0.290$ |
| MotionCtrl | $15.86$ | $0.520$ | $0.431$ | $15.33$ | $0.479$ | $0.590$ |
| See3D | $22.37$ | $0.781$ | $0.199$ | $21.60$ | $0.744$ | $0.238$ |
| ViewCrafter | $22.60$ | $0.754$ | $0.195$ | $22.25$ | $0.709$ | $0.204$ |
| FlexWorld-5B | $23.05$ | $0.788$ | $0.182$ | $22.86$ | $0.756$ | $0.198$ |
| Voyager-13B | $23.12$ | $0.793$ | $0.175$ | $22.93$ | $0.768$ | $0.194$ |
| \rowcolor[gray]0.9    Unified Models |  |  |  |  |  |  |
| BAGEL-7B-FT | 21.76 | 0.703 | 0.288 | 21.04 | 0.599 | 0.403 |
| Omni-View-7B | 23.22 | 0.817 | 0.114 | 23.12 | 0.801 | 0.146 |

### 4.3 Ablation studies

We perform ablation studies on the proposed architecture and training strategy to ascertain their efficacy. The data used and the hyperparameters applied during the ablation studies remain consistent.

Effect of two modules in the generation model. Both the texture module and the geometry module can improve understanding performance, but they focus on different aspects. The results presented in Table [4](https://arxiv.org/html/2511.07222v1#S4.T4) demonstrate that the integration of the texture module facilitates notable advancements in tasks dependent on spatiotemporal modeling, exemplified by an increase of 4.1 points in Appr. Order. Conversely, the introduction of the geometry module markedly enhances performance in tasks contingent upon relative position information, notably in Rel. Dist. However, because of the lacking absolute metric in the synthesized depth maps, improvements in tasks pertaining to absolute metric comprehension, such as Abs. Dist., are constrained. Incorporating the task of accurately predicting camera pose can mitigate the reduction resulting from the imprecise depth prediction. Furthermore, our results corroborate that the segregation of the texture and geometry modules results in superior understanding performance compared to employing a unified architecture that concurrently learns both texture and geometry.

**Table 4: Ablation on modules in the generation model. The gray row denotes that, in this experiment, both the texture module and the geometry module use the same architecture and parameters.**
| Texture | Geometry | SQA3D | ScanQA | ScanRefer | VSI-Bench (subset) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| depth | camera | EM | C | B-4 | Acc@0.25 | Obj. Cnt. | Abs. Dist. | Obj. Size | Rel. Dist. | Appr. Order |  |
| ✘ | ✘ | ✘ | 57.2 | 95.5 | 14.7 | 46.9 (28.0) | 62.8 | 36.3 | 56.4 | 46.1 | 43.1 |
| ✔ | ✘ | ✘ | 58.3 | 97.4 | 14.7 | 48.8 (31.5) | 67.7 | 44.6 | 59.0 | 53.2 | 47.2 |
| ✔ | ✔ | ✘ | 58.9 | 100.5 | 16.0 | 48.2 (31.2) | 69.0 | 44.9 | 69.7 | 63.0 | 47.9 |
| \rowcolor[gray]0.9<br>✔ | ✔ | ✔ | 58.7 | 99.2 | 15.0 | 49.0 (31.6) | 69.5 | 45.9 | 67.8 | 63.8 | 48.2 |
| ✔ | ✔ | ✔ | 59.2 | 103.0 | 16.2 | 50.8 (32.5) | 70.3 | 46.4 | 68.6 | 65.9 | 49.0 |

Effect of the autoregressive generation. Autoregressive generation significantly improves tasks requiring spatiotemporal modeling, like Appr. Order. Table [5](https://arxiv.org/html/2511.07222v1#S4.T5) compares the understanding performance with and without autoregressive generation. Although bidirectional generation offers some enhancement, autoregressive generation provides greater improvements. Specifically, it boosts performance by 5.8 and 4.4 points in Abs. Dist. and Appr. Order tasks, respectively, compared to when not used. This demonstrates that autoregressive generation strengthens spatiotemporal modeling and enhances related scene understanding tasks.

**Table 5: Ablation on the autoregressive generation in stage 1. “None” means we only train understanding model in this experiment.**
| Autoregressive | SQA3D | ScanQA | ScanRefer | VSI-Bench (subset) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EM | B-4 | EM | Acc@0.25 | Obj. Cnt. | Abs. Dist. | Obj. Size | Rel. Dist. | Appr. Order |  |
| None | 57.2 | 95.5 | 14.7 | 46.9 (28.0) | 62.8 | 36.3 | 56.4 | 46.1 | 43.1 |
| ✘ | 57.2 | 98.8 | 15.0 | 47.0 (28.0) | 68.3 | 40.6 | 69.0 | 60.6 | 44.6 |
| ✔ | 59.2 | 103.0 | 16.2 | 50.8 (32.5) | 70.3 | 46.4 | 68.6 | 65.9 | 49.0 |

Effect of the D2S (dense-to-sparse). The efficacy of D2S training is substantiated during stage 1. The results show that D2S significantly improves the model understanding performance, exceeding the visual reconstruction method advanced in Ross3D (wang2025ross3d) for scene understanding tasks necessitating spatiotemporal ordering. In this ablation, the geometry module is kept trainable.

**Table 6: Ablation on the D2S mechanism in stage 1 (S1). “None” means we don’t train generation model in this experiment. The “random mask” denotes the visual reconstruction method in Ross3D (wang2025ross3d).**
| Condition in S1 | SQA3D | ScanQA | ScanRefer | VSI-Bench (subset) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EM | B-4 | EM | Acc@0.25 | Obj. Cnt. | Abs. Dist. | Obj. Size | Rel. Dist. | Appr. Order |  |
| None | 57.2 | 95.5 | 14.7 | 46.9 (28.0) | 62.8 | 36.3 | 56.4 | 46.1 | 43.1 |
| randon mask | 58.9 | 98.9 | 15.5 | 49.4 (31.7) | 65.7 | 38.9 | 55.5 | 51.6 | 46.5 |
| dense | 57.3 | 94.7 | 15.0 | 47.2 (28.3) | 67.7 | 42.3 | 60.1 | 65.6 | 46.6 |
| sparse | 57.9 | 97.3 | 15.7 | 50.1 (32.0) | 69.0 | 44.9 | 65.5 | 63.1 | 48.3 |
| dense $\to$ sparse (Ours) | 59.2 | 103.0 | 16.2 | 50.8 (32.5) | 70.3 | 46.4 | 68.6 | 65.9 | 49.0 |

Effect of training stage 2. The results in Table [7](https://arxiv.org/html/2511.07222v1#S4.T7) show that stage 2 can significantly improve the scene generation performance of the model.

**Table 7: Ablation of the stage 2.**
| Stage 2 | NVS from single view | Scene Generation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| ✘ | $21.90$ | $0.705$ | $0.265$ | $21.44$ | $0.683$ | $0.307$ |
| ✔ | $23.22$ | $0.817$ | $0.114$ | $22.93$ | $0.768$ | $0.194$ |

## 5 Conclusion

We introduce Omni-View, a unified 3D scene understanding and generation model. By decomposing the generation model into distinct texture and geometry components, we establish the feasibility and efficacy of employing generation processes to enhance 3D scene understanding and spatial reasoning. Our method illustrates that a unified understanding and generation model is capable of achieving performance on par with leading specialized understanding models. We posit that Omni-View can serve as a foundational model across 3D and multiview domains, thereby advancing the development of downstream applications such as spatial and intelligence.

Ethics Statement. This paper aims to develop unified models to understand and generate 3D scenes. In light of the ongoing advancements in scene generation technology, we emphasize the importance of preventing its misapplication, such as the fabrication of deceptive scenes or the creation of scenes with nefarious intents.

Reproducibility Statement. We state that Omni-View is highly reproducible. Implementation details on our main experiences are provided in Section [4.1](https://arxiv.org/html/2511.07222v1#S4.SS1). It is anticipated that these descriptions can sufficiently demonstrate the reproducibility of Omni-View. We plan to open-source the code and weight files after the paper passes peer review.

Limitations and future work. Although Omni-View has demonstrated a unified advance in the domains of question answering, spatial reasoning, and novel view synthesis, its grounding capabilities remain to be substantiated. Additionally, its generation model currently lacks the capability for long-range world generation. Future efforts will be concentrated on using reinforcement learning to augment Omni-View’s performance in 3D visual grounding and long-range generation. At the same time, since the training data used by its geometry module is a synthetic depth map, its actual geometry prediction ability may not be accurate enough.

## Appendix A Qualitative results.

We show some qualitative results for 3D scene understanding and generation tasks in the appendix.

### A.1 3D scene understanding

[Uncaptioned image]: x2.png

### A.2 Spatial Reasoning

[Uncaptioned image]: x3.png

### A.3 NVS from single view

We show an effective result (3f2bf7371b72e40f) and a bad case (e7020d449f50c737) for NVS from single view task. It can be seen that when the angle changes greatly, Omni-View’s inter-frame consistency still has room for improvement, for example, the object on the left side of the door is inconsistent between View-20 and View-24 in sample e7020d449f50c737.

|  |  |  |
| --- | --- | --- |
| Reference (3f2bf7371b72e40f) |  |  |
|  |  |  |
| View-4 | View-8 | View-12 |
|  |  |  |
| View-16 | View-20 | View-24 |
|  |  |  |
| Reference (e7020d449f50c737) |  |  |
|  |  |  |
| View-4 | View-8 | View-12 |
|  |  |  |
| View-16 | View-20 | View-24 |

The Use of Large Language Models (LLMs). LLMs are used to correct potential grammatical inaccuracies in the manuscript. LLMs do not participate in research ideation.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Clarify the novelty beyond architectural decoupling: In what ways is the texture/geometry split more than a clean engineering separation compared to prior “appearance vs. structure” decouplings (e.g., ViewCrafter, Voyager) and unified frameworks (BAGEL, VILA-U, BLIP3o, Harmon)? Could you provide a controlled comparison to a single-branch generator with two prediction heads (texture, geometry) at equal parameter count?
- The ablations show AR improves spatiotemporal reasoning. Can you report exposure-bias analyses at inference time, e.g., teacher-forcing vs. free-running rollouts? Does diffusion forcing mitigate compounding errors, and how does performance vary with rollout length (8/16/32 frames)? Have you tested scheduled sampling or token-level AR only on camera poses while keeping texture bidirectional?
- Provide qualitative and quantitative failure analyses: cases where geometry improves understanding but harms texture fidelity (and vice versa), per-category VSI error tied to pose/depth errors, and sensitivity to large viewpoint changes where you noted inconsistencies. Would integrating a small, explicit 3D proxy (e.g., Gaussians or sparse point clouds) at training but not inference close these gaps?
soundness: 3
strengths: - The paper presents a unified 3D understanding–generation framework that cleanly separates texture and geometry, a simple yet original design that operationalizes “generation facilitates understanding.”
- The two-stage recipe with dense-to-sparse curriculum and autoregressive NVS is well-motivated, technically sound, and shows careful loss design and gradient routing to benefit the understanding model.
- Writing is clear and structured, with concrete training details, datasets, metrics, and ablations that isolate the contribution of each module and training choice.
- The empirical significance is strong, with SOTA results on VSI-Bench and competitive 3D QA/localization, plus solid NVS and scene generation metrics, demonstrating broad impact across 3D reasoning and generation.
summary: The paper presents Omni-View, a unified model for 3D scene understanding and generation from multiview images that tests the “generation facilitates understanding” hypothesis. Built on Bagel, it splits generation into a texture module (flow matching with Plücker pose encoding, autoregressive NVS) and a geometry module (depth and camera pose via flow matching with cross-attention to understanding features). A two-stage training recipe first jointly trains understanding/texture/geometry with a dense-to-sparse curriculum, then fine-tunes generation with RGB-Depth-Pose joint learning. Omni-View achieves SOTA on VSI-Bench (55.4), improves QA/localization versus unified baselines without 3D inputs, and delivers strong NVS/scene generation results on Re10k.
weaknesses: - Limited novelty relative to prior unified frameworks (Bagel, VILA-U, BLIP3o, Harmon)  
The core idea of leveraging generation to aid understanding has precedents in 2D unified models and recent 3D works that inject reconstruction priors (e.g., Ross3D; VG-LLM/Spatial-MLLM via VGGT features). The split into texture vs. geometry resembles established “appearance vs. structure” decouplings in 3D pipelines (e.g., ViewCrafter, Voyager). Clarify what is fundamentally new beyond integrating these pieces within Bagel, and compare to a “single-branch with multi-heads” backbone.

- Ambiguity in camera control and absolute metric grounding   
The paper reports strong perceptual metrics but acknowledges difficulty in precise camera control and absolute depth scale. Because the gains on VSI categories like Abs. Dist. hinge on metric grounding, add analyses: scale consistency across scenes, depth-scale calibration via known baselines, and camera-pose accuracy vs. ground truth under diverse motions.

- Dataset overlap and generalization concerns  
Though the authors state they avoid using understanding images for generation training, several datasets share scene domains with Re10k-like indoor content, risking leakage of priors. Please report cross-dataset generalization (e.g., ScanNet -> Replica, RealEstate10K -> ACID/CO3D subsets) to support robustness claims.

- Incomplete ablations on design choices and routing  
The geometry module conditions only on the last-layer texture latent and uses cross-attention to the understanding model. Test alternatives: multi-scale latents, earlier-layer features, and gating that controls gradient flow to avoid potential interference. Provide compute/latency breakdowns for stage 1 vs. stage 2, and show sensitivity to $\lambda_{geo}$, pose-query design, and Plücker vs. other pose encodings.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Mentioned in the weaknesses
soundness: 3
strengths: - This paper demonstrates that generative 3D tasks (novel view synthesis, geometry estimation) can actively enhance 3D scene understanding, rather than being separate objectives.

- This paper has a unified architecture for 3D reasoning, with separate texture and geometry modules allow complementary learning of appearance and spatial structure, leading to better localization, spatial reasoning, and depth-aware Q&A.

- This paper outperforms specialized models in 3D understanding benchmarks while maintaining competitive NVS and scene generation performance, closing the gap between multimodal understanding and 3D generative models.

- A systematically organized evaluation and ablation study would strengthen the credibility of this paper.
summary: - This paper proposes Omni-View, a unified multimodal model that jointly performs 3D scene understanding, geometry estimation, and novel view synthesis from multiview images, based on the principle that generation facilitates understanding.

- This paper introduces a dual-path architecture consisting of a texture module (appearance generation) and a geometry module (depth/pose estimation), enabling bidirectional synergy between generative and understanding tasks.

- This paper uses a two-stage training strategy where joint training enhances understanding via generative signals, followed by refinement for high-quality 3D scene generation; achieves state-of-the-art performance on VSI-Bench and strong results in NVS and 3D Q&A.
weaknesses: - It would be beneficial to include a diagram that more precisely illustrates the functionality of each module and the architecture, compared to the current version.

- Additionally, visualizing 3D scene understanding / spatial reasoning / NVS from a single view as a video could also be an effective way to present the capabilities of the system.

- Is there a reason why you refer to Texture Module and Geometry Module in the equations (e.g., (eq. 1), (eq. 2)) without using italics? Also, I believe writing them as TextureModule and GeometryModule (without a space) would improve readability and be more suitable for mathematical notation.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Could the authors provide qualitative results of the geometry module’s depth estimation, camera pose estimation with metric-scale? Without any depth estimation results or broader test cases, it is difficult to assess whether the geometry module truly learns meaningful 3D structure rather than overfitting to training priors, without any generalizability.
2. How well does Omni-View generalize to unseen or real-world multi-view scenes, rather than the well captured ones in the appendix? Have the authors tested its performance to verify the robustness and effectiveness of the learned spatial reasoning?
3. Could the authors clarify key implementation and training details—such as dataset splits, optimizer configurations, training epochs, and computational cost—to ensure reproducibility? Including more specifics or releasing supplementary materials would improve transparency.
soundness: 3
strengths: 1. Clear intuition and solid empirical validation.
The paper builds upon a clear and intuitive idea — that generation can facilitate understanding — and the overall logic is easy to follow. Quantitative results across multiple benchmarks convincingly demonstrate the benefits of the proposed design, especially in spatial reasoning and novel view synthesis.
2. Architectural innovation.
By decomposing the generation process into texture and geometry modules, the authors present a meaningful and modular architecture that captures both appearance and structure. This decomposition aligns well with human visual reasoning and can be viewed as an innovative contribution for the community.
3. Comprehensive ablation studies.
The ablation results thoroughly verify the contributions of the proposed contributions. These analyses effectively demonstrate that each component contributes to the final understanding and reasoning performance.
summary: This paper introduces Omni-View, a unified model for 3D scene understanding and generation that explicitly investigates the hypothesis that “generation facilitates understanding.” The contributions of the paper are:
1. Unified architecture integrating 3D scene understanding and generation, composed of three main components:
1.1 Understanding model (for spatial reasoning and QA)
1.2 Texture module (for novel view synthesis)
1.3 Geometry module (for depth and pose estimation)
2. Proposed a novel two-stage training strategy:
2.1 Jointly trains understanding and generation to encourage mutual benefits through geometry and spatiotemporal modeling.
2.2 Fine-tunes generation with RGB-Depth-Pose joint learning for better geometric consistency.
3. Empirical validation showing state-of-the-art (SOTA) performance on the VSI-Bench (score 55.4), outperforming both specialized and unified 3D models in reasoning tasks.
Overall, Omni-View demonstrates how generative modeling (novel view synthesis, geometry estimation) can enhance 3D reasoning, localization, and understanding—a conceptually elegant and empirically supported contribution.
weaknesses: 1. The qualitative results in the appendix are sparse, and there are no depth estimation visualizations or broader test cases. This makes it difficult to verify the model’s generalization and effectiveness beyond the reported metrics. For instance, the quality and consistency of metric-scale prediction from the geometry module remain uncertain — the reported results could be influenced by selective visualization or data bias, since the paper lacks convincing examples that demonstrate accurate geometric reasoning across varied real-world scenes.
2. The technical details provided for training, implementation, and comparison setups are relatively limited. Without clearer supplementary material (e.g., dataset statistics, architecture specifics, or convergence behavior), it is challenging to fully reproduce the reported results or assess robustness under different conditions.
3. The absence of released code or live demonstrations restricts the ability of other researchers to validate or extend this work. Although acceptable for review, the paper would be strengthened by open-sourcing its checkpoints or providing additional evaluation on long-range world generation and 3D visual grounding.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Strengthen theoretical analysis or visual explanation of the "generation promotes understanding" mechanism;

Compare with state-of-the-art methods on more 3D grounding tasks and analyze the sources of the gap.
soundness: 3
strengths: Unified modeling is novel: This is the first systematic exploration of the "generation-driven understanding" mechanism in 3D scenes, which is both inspiring and forward-looking.

Rational modular design: The generation module is split into texture and geometry components, modeling appearance and structure respectively, effectively improving understanding capabilities.

Effective training strategy: The two-stage training (unified training + generation fine-tuning) balances understanding and generation performance, and the D2S mechanism improves robustness.

Comprehensive experiments: The effectiveness of the method is verified on multiple 3D understanding, spatial reasoning, and generation tasks, with results significantly outperforming existing unified models.

No 3D input required: Relying solely on multi-view images, this improves the model's practicality and generalization capabilities.
summary: This paper proposes Omni-View, a unified 3D scene understanding and generation model based on multi-view imagery, exploring the principle of "generation promotes understanding." The model, composed of an understanding module, a texture module, and a geometry module, jointly models scene understanding, novel view synthesis, and geometry estimation. Using a two-stage training strategy, Omni-View achieved a state-of-the-art score of 55.4 on the VSI-Bench, surpassing existing dedicated 3D understanding models while also performing well in novel view synthesis and scene generation tasks.
weaknesses: Weak theoretical analysis: Although "generation promotes understanding" has been proposed, there is a lack of theoretical or interpretable analysis of its underlying mechanisms.

Generation quality still has room for improvement: Despite leading in PSNR/SSIM, inter-frame consistency under large viewpoint variations remains suboptimal (see Appendix visualization).

The geometry module relies on synthetic data: The depth map is synthesized by Voyager, which may limit the realism and accuracy of geometry predictions.

Limited long sequence generation capability: The model currently does not support long sequence scene generation, limiting its application in open-world scenarios.

There is still a gap compared to state-of-the-art dedicated models: In particular, in the 3D grounding task, there is still a significant gap compared to methods that rely on 3D input.Strengthen theoretical analysis or visual explanation of the "generation promotes understanding" mechanism;

Compare with state-of-the-art methods on more 3D grounding tasks and analyze the sources of the gap.


Decision: Accept (Poster)