UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
Authors: Yueming Xu
Jiahui Zhang
Ze Huang
Yurui Chen
Yanpeng Zhou
Zhenyu Chen
Yu-Jie Yuan
Pengxiang Xia
Guowei Huang
Xinyue Cai
Zhongang Qi
Xingyue Quan
Jianye HAO
Hang Xu
Li Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Unified understanding and generation; Large language models; 3D generation; 3D vision; Spatial understanding
Primary area: foundation or frontier models, including LLMs
TL;DR: Unified 3D Understanding and Generation via Geometric-Semantic Encoding

Abstract: Despite the impressive progress on understanding and generating images shown by the recent unified architectures, the integration of 3D tasks remains challenging and largely unexplored. In this paper, we introduce UniUGG, the first unified understanding and generation framework for 3D modalities. Our unified framework employs an LLM to comprehend and decode sentences and 3D representations. At its core, we propose a spatial decoder leveraging a latent diffusion model to generate high-quality 3D representations. This allows for the generation and imagination of 3D scenes based on a reference image and an arbitrary view transformation, while remaining supports for spatial visual question answering (VQA) tasks. Additionally, we propose a geometric-semantic learning strategy to pretrain the vision encoder. This design jointly captures the input's semantic and geometric cues, enhancing both spatial understanding and generation. Extensive experimental results demonstrate the superiority of our method in visual representation, spatial understanding, and 3D generation.

Published as a conference paper at ICLR 2026
UNIUGG: UNIFIED 3D UNDERSTANDING AND GENER- ATION VIA GEOMETRIC- COMMERRC-SEMANTIC SEMANTIC ENCODING
Yueming Xu
Jiahui Zhang" Ze Huang Yurui Chen' Yanpeng Zhou? Zhenyu Chen? Yu-Jie Yuan Pengxiang Xia Guowei Huang? Xinyue Cai² Zhongang Qi Xingyue Quan
Jianye Hao
Hang Xu² Li Zhang"!
Fudan University 2 Huawei Noah's Ark Lab
https: //te:///uddn-- /tt///uuannzzaaa wet.nnnhgggggggghbb.g. github.hhb...... ie/iio/nnniii UniUCCG
ABSTRACT
Despite the impressive progress understanding and generating images shown by the recent unified architectures, the integration of 3D tasks remains challenging and largely unexplored. In this paper, we introduce UniUGG, the first unified un- derstanding and generation framework for 3D modalities. Our unified framework employs an LLM to comprehend and decode sentences and 3D representations. At its core, we propose a spatial decoder leveraging latent diffusion model to gener- ate high-quality 3D representations. This allows for the generation and imagination of 3D scenes based on a reference image and an arbitrary view transformation, while remaining supports for spatial visual question answering (VQA) tasks. Addi- tionally, we propose a geometric-semantis learning strategy to pretrain the vision encoder. This design jointly captures the input's semantic and geometric cues, enhancing both spatial understanding and generation. Extensive experimental results demonstrate the superiority of our method in visual representation, spatial understanding, and 3D generation.
INTRODUCTION
Recent work on unified 2D understanding and generation has made significant strides (Sun al.
2023
2024 Ye et al. 22224] Dong et al. 20244; Wu et al. [2024] Team 2024) Wang et al. 2024a] 4a] Liu
al.
2024a Wu al. 2025
| Chen et al. 2025 Huang et al. 2025). Early works (Sun al. [023]
2024
Ye et al. 2024 Dong et al. 2024) build unified frameworks that couple an autoregressive LLM
with diffusion image decoder. The LLM consumes text-vision inputs and produces fixed set of learnable queries whose features are regressed into the diffusion latent space; the diffusion model
then synthesizes the image conditioned on these latents. Subsequent works (Team) 2024 Liu et al.
2024a
Wu al. 2024) employ VQ tokenizers for the unified generation of texts and images.
Although the aforementioned works have made significant progress in images, the challenge of achieving unified understanding and generation for 3D modalities remains largely unexplored. Several
benchmarks (Zhang et al.) 2025b, Fu et al., 2024b Ma et al. 2024, Yang et al. 2025) integrate large-scale public datasets and design spatial VQA tasks to enhance the spatial reasoning capability of LLMs. However, these efforts focus solely on spatial understanding and take a rather brute-force approach-directly fine-tuning LLMs with large amounts of spatial data, which has shown limited effectiveness. Other works (Chen et al. 2024a Cheng et al. 2024 Hong et al., 2023 Driess et al.) 2023 Chen al. [222ce] Zhu al. 2024a) incorporate additional modalities, such depth, point clouds, or scene graphs, to handle spatial understanding, Unfortunately, these methods introduce additional disadvantages, as they often require specialized data acquisition devices or necessitate explicit spatial modeling of the entire scene. These drawbacks hinder potential development for practical applications.
We identify two main challenges that bottleneck progress for the unified 3D frameworks. The first issue is the limitation of visual representations. Current L.MSs typically rely on vision encoders
Equally contributed Corresponding author (lizhanffd@fudan (ud....d.uun))

---

Published as a conference paper at ICLR 2026
(R) UniLGGG demo
(C) UnlUIG performance
UNiUGG
Geseulei spent ssurking fieee insigcaed
Spatial undentanding
Lnnified spatial undersaaddnng,om peneration
View Kie lell 4ar:
Besokle tiv gesmiied
kingneare with a forelquttonn eesttoodd scla pongursiie tua
grneration
Beadhiio with blidd.
image
Geveate sooe fooes poitbe desvite
Oves
Depth
FeuthSS beachnark
pou
ond Mosbeoi qui
Anmann
A Loy bes hooe wi l-gl- ull decarloutty limpard the ay fanies Toply cus fooe de mooder Feor ficet of
bood' to the Iobuett tie Boucl cteerve
The Moad cor lik of
00
Figure 1: We introduce UniUGG, the first unified framework for spatial understanding and generation. (A) UniUGG supports spatial-level VQA and generates geometrically consistent 3D scenes. (B) Given a reference image, it can creatively generate 3D variations and describe them accurately. (C) UniUGG outperforms baselines in both spatial understanding and generation, with our specially tuned vision encoder excelling in downstream tasks.
pretrained on 2D image semantic tasks, which lack the capability to model 3D geometries. This limitation creates performance bottleneck, particularly in spatial understanding tasks. The second issue is the incompatibility between 3D generation and LLMs. Here, LLMs are built on the basis of tokenization methods to autoregressively generate the next token. This scheme adapts well to image generation, as images are regular and can be tokenized into a fixed number of elements. However, such tokenization approaches not easily applicable 3D data such point clouds due to the irregular nature of representations. This tokenization gap makes it more challenging for LLMs to handle autoregressive 3D generation tasks effectively.
Recently, a line of works (Wang & Agapito 2024 Leroy et al. 2024 Zhang et al. 2024 Wang ctt al.) 2025b a) initiated by DUS(3R (Wang et al. 2024b) have introduced a new perspective on spatial representation by aligning pixels across multi-view into a unified global coordinate system. The multi-view geometry training paradigm enables models to reconstruct 3D scenes from visual inputs, along with predicting spatial relationships. Inspired by this, we introduce UniUGG, the first framework for unified spatial understanding and generation, marking a significant step by solving the aforementioned two issues.
For the first issue, we design a geometric-semante learning strategy for vision encoder pretraining. This strategy incorporates semantic information from teacher model while integrating the encoder with a spatial decoder for end-to-end multi-view geometric training, thereby enhancing its spatial modeling capabilities. The resulting ViT representations significantly improve both understanding and generation within the unified framework and yield better results a variety of downstream tasks. The learning strategy not only provides the LLM with an enriched representation but also bridges the gap between vision and 3D using the spatial decoder. This decoder, a byproduct of the pretraining process, decodes visual representations into 3D scenes corresponding the two inputs. Upon these, we solve the second issue by designing UniUGG. UniUGG takes a reference visual representation and an encoded target-view raymap as input, producing conditional features. These features are then used with a diffusion model to generate the visual representation of the target-view. To enhance this process, we design the Spatial- which effectively compresses coametric-semante information, enabling more accurate and efficient representation generation. Additionally, it links the spatial decoder for end-to-end fine-tuning, enhancing information compression while mitigating the negative impact of discrepancies between the reconstructed and original representations on 3D scene decoding. Finally, both the original and generated visual representations are passed through the fine-tuned spatial decoder to decode the 3D scene. Thanks the L.LM-based architecture. UniUGG simultancously learn understanding and generation tasks, enabling its 3D scene inference, while maintaining spatial VQA capabilities for both real and generated representations.
2

---

Published as a conference paper at ICLR 2026
Our main contributions summarized as follows: (i) We propose the first LLM- unified generation and understanding framework for 3D scenes, UniUGG, which enables spatial-level VQA and generates geometrically consistent rich 3D environments. (ii) We introduce novel geometric- semantic vision encoder pretraining strategy. Here, our ViT encodes geometric cues from input image pairs and preserves semantic features from 21) priors. (iii) We present a Spatial- VAE as the of our 3D scene representation generation scheme. Our Spatial-VAF compresses the 3D geometric-semantie representations from input image pairs and helps producing sharper 3D point clouds as output. (iv) Our method achieves top performance on multiple spatial reasoning benchmarks, surpassing baselines on VSI-Bench by 17.9% in particular, and maintaining significant superiority in 3D generation.
2 RELATED WORKS
Language models for spatial reasoning and generation There has been growing interest in
applying large multimodal language models to spatial reasoning tasks. Recent models (Alayrac et al.
[222sDriesss et al. 2023 Liu et al., 2023 Li et al. 2024a| [Bai et al. 2023) have shown impressive
results in language-guide
visual understanding, but they often focus primarily on semantic alignment. As result, they exhibit clear limitations when it comes understanding spatial relations, viewpoint
changes, and structural consistency. Several benchmarks (Zhang et al. 2025b Fu et al. 2024b; Ma
ct al. [024]] Yang al. (2025) have been proposed evaluate spatial reasoning abilities, and existing
methods (Chen et al. 2024a; Cheng et al., 2024: Hong et al. 2023 Driess et al., 2023 Chen et al.
2024e| Zhu et al. 2024a) typically enhance performance by increasing training data or incorporating additional structural inputs, such as depth, point clouds, or scene graphs. However, these structural inputs are often used without explicit modeling of spatial consistency, and structure-aware visual representations remain underexplored. In addition, while recent works have achieved unified 2D
Ye et al., 2024 Dong et al. 2024] Wu et al. 2024 Team 2024 Wang et al. 2024a] Liu et al. 2024a Wu et al. 2025 Chen et al. 2025 Huang
understanding and generation (Sun et al. 2023 2024
et al. 2025). there is limited research on applying this concept to the spatial level. In contrast, we
propose the first unified spatial framework, which not only handles spatial reasoning tasks but also
generates 3D) scenes based on a reference image and a specified view transformation.
Semantic and geometric representation Vision encoders Liang et al. 2024 Zhai et al. 2023
Cherti et al. 2023 Jia et al. 2021, Li et al. 2021 2022 2023 Zhu al., 2024b) trained with language supervision have demonstrated strong capabilities in semantic understanding, particularly in tasks involving open-vocabular recognition and image-text alignment. However, these models typically lack spatial awareness and fail to capture geometric consistency across views. In contrast, geometric methods (Wang et al. 2024b Leroy et al. 2024 Wang et al. 2025b) based on multi-view consistency learning focus on spatial correspondence and 3D reconstruction, but typically lack seman- tic understanding and are difficult to generalize to language guided tasks. Other works (Ranzinger et al. 2024b a 2025 Heinrich al. 2025 Sarryildz et al. (0255) explore multi-teacher feature distillation to combine semantic and geometric knowledge, but their training objectives focus on general -prrpse representation fusion rather than geometric awareness.In contrast, we aim pretrain the vision encoder with both semantic and geometric awareness, tailored for spatial unification.
METHODOLOGY
3.1 PIPELINE OVERVIEW
Fig. 2 presents the overall workflow, with the training pipeline on the left and the inference process on the right. During training, UniUGG adopts a three-stage training strategy. First, as shown in Fig. (a), we pretrain the vision encoder to learn geometric-senee semanticc visual representations in stage 1, detailed in Sec. 3.2 Next, as illustrated in Fig. 22 (b). we pretrain the Spatial-VAB in stage 2, which information into a compact latent space. This module enhances efficient generation and facilitates producing sharper 3D point clouds, as discussed in Sec. 3.3 Finally, we perform unified training for spatial understanding and 3D generation in stage 3, while keeping both the ViT and the VAE encoder frozen, as depicted in Fig. (c). The unified training procedure is elaborated in Sec. 3.3) During inference, UniUGG takes images, questions, or view trewtrranfoormation as input and generates text answers and point clouds, as shown in Fig. (d).
compresses seometrie-semantiee
3

---

Published as a conference paper at ICLR 2026
(a) Vision encoder pretraining
(c) UniUGG training
(d) UniUGG inferencing
Spatial
transformation
answer
transformation
Questton
VIT
decoder
ViT
VIT
Encode
Encode
Large language model
Large language model
(b) Spatial-VAE pretraining
Spatial-VAE
Spatial-VAE decoder
Denoising
Spatial-VAE decoder
encodar
Spatial-VAE encoder
Denoising U-Net
U-Net
Answer
Spatial decoder
Answer token prediction
VIT token
Recon token
Noise prediction
Point cloud
Matchings
Confidences
Figure 2: Pipeline overview of UniUGG. The left illustrates the three-stage training process, and the right shows the inference pipeline for spatial reasoning and 3D generation.
3.2 VISION ENCODER PRETRAINING
In this section, we introduce our geometric-semant semantic learning strategy for vision encoder pretraining, shown in Fig.
Encoder architecture Following the design of RADIOv2.5 (Heinrich et al. 2025), we adopt ViT- L/16 (osovitskiyy et al. 2020) our basic vision encoder to match the architecture of teachers, which allows us to benefit from its pretraining. Given an input image I R # x3, the vision encoder first partitions it into fixed-sized patches of size p pixels, then embeds them into hidden features of dimension d with learnable positional embeddings. After a series of Transformer blocks, the model finally produces set Z 2 of visual representations, where 2 E R N
Multi-view geometric learning To enhance our encoder's spatial modeling, we adopt the MASt3R (Leroy al., (022)) framework, retaining its decoder and spatial losses, while replacing the original encoder with ours. As shown in Fig. paired images I' I' are encoded by our shared-weight encoder, A two-layer visual projector fx (.) with GeLU activation processes each representation inde- pendently. Next, the projected features are fed into dual cross-attention decoders, yield- ing Hi, H = Decoder( [ (x nally, pointmaps X/, X RB x3 and confi- dence maps C³, Ci are regressed from [H' HJ,. along with dense matching descriptors D D R xdy, via spatial head.
producing two representations Z and Z
(a) Semantic guiding
Image 2
Fi-
E
F
E
shared
shared
MSE
Spatial decoder
To extend our encoder for color awarenes,, we implement RGB head to reconstruct color in- formation from encoded representation 2, con- strained by composite loss:
Projector
shared
Projector
MASI3R
Colerbad
Oporial head
shared
Spwtilhead
Color head
Ligb = AL, || Z[l,, + LAr-LPIIS((I,)) (1)
Spatial
RGB losses
denotes the LI-norm, and LPIPS (Zhang et al., 2018) captures perceptual similarity based deep networks. The final training loss in the spatial branch is defined as:
where
Pointeloud
Confidence
Confidence Point Point-loud
(b) Spatial representation learning
Figure 3: Overview of our encoder pretraining pipeline. (a) During semantic guiding, our student encoder learns to mimic the teacher's visual repre- sentations. (b) In spatial representation learning, the spatial decoder jointly refines predictions using information from both views.
L. = Lecnt + X1 Lmatch X X2L
where Lcout and Lmatch are confidence-awee gression loss and matching loss, respectively, defined in MAS13R.
4

---

Published as a conference paper at ICLR 2026
Latent token learning
(b) Unified understanding and generation
Generation
Understanding
View transformation
Thessee feglure punounded four chatn....
Image
Iosaee
Image
Image
VII
VIT
VII
VIT
Transformation cuceding
T Tekeniace
Image representation
regrosntation
Incefeetration
Quslien bolamt Answer Insere
Spatial-VAE encoder
Spatial-VAE encoder
Spatial-VAE encoder
Large language model
Latent takee
Lanert bokan 2
Natted
Conditiional
perdiction
Spatial-VAE decoder
shared
é Spatial-VAE decoder
Add noise
Denoising U-Net
talamt sda
PONU-
Spattal- VAE docoder
Spatial Jucoder
Recindracted representsiion and repersentattee
Gauscian
Notie prodiction
Lusent tokenn prediction
tape.
Spatial decoder
Legends-
Cioo-Tatrppy RGB loss, Regrassion loes
Confilence loss,
Point Confidence
Inferencing
Training
Figure 4: Overview of UniUGG training and inferencing, (a) In the latent token learning stage (stage 2), visual representation is compressed using the Spatial-VAE, while the spatial decoder linked for fine-tuning. (b) In the unified learning stage (stage 3), the reference image's visual representation and view transformation
L.L,, which outputs conditional features for noise prediction on latent token. The LLM also performs VQA-related training to maintain its understanding capability. During inferencing, UniUGG generates the visual representation of the target view, which, together with the reference representation, is decoded into the 3D scene.
input
Note that in the following, the visual projector, the MASt3R decoder, and the prediction heads collectively referred to as the spatial decoder, as illustrated in Fig.
Semantic knowledge guiding To enhance semantic understanding,
the pretrained RADIOv2.5
teacher guide our encoder. Given an input image, semantic tokens 2 Nxd are extracted from the teacher and aligned with student tokens Z via a weighted sum of cosine distance and smooth L, losses. To improve robustness, the loss computed over a randomly sampled subset of tokens. The guiding loss is defined as:
LKD = a:1 -
- cos(2,, (0,,,22)) 2,)) + 8- Σ - - &1l
n IEC
iEC
where |C| denotes the number of tokens. This alignment enforces consistency in both feature direction and magnitude.
3.3 UNIFIED UNDERSTANDING AND GENERATION LEARNING
Leveraging prior knowledge, humans can imagine both geometric and semantic details of unobserved areas from reference image. With this goal, we aim to enable LLMs understand and reason scenes through spatial question answering, and imagine novel 3D structures under view changes.
Overall learning target Based our encoder pretraining, design UniUGG for unified spatial understanding and generation, detailed in Fig. 4 (b). For 3D generation, we leverage an LLM in combination with a diffusion model to generate the visual representation of the target view conditioned on reference image and view transformation. The pretrained spatial decoder then processes the visual representations of both the reference image and the target view, decoding the 3D scene. For understanding, also perform supervised fine-tuning on the LLM at spatially grounded VQA tasks.
Latent token learning Directly generating high-dimensional representations costly and unstable. To address this, we design and pretrain the Spatial- VAE with an encoder-decoder architecture to compress visual representations into a compact latent space, enabling efficient generation, shown in Fig. 4 (a). Given image pair I I', our pretrained vision encoder extracts visual representations Z E N, N, xd which are encoded into 4-dimensional latent tokens T B x and then reconstructed back 2 Zi.
5

---

Published as a conference paper at ICLR 2026
The Spatial VAE optimization is guided by three loss terms: (i) Reconstruction loss Lase (ii) KL loss fx (q 2 ll P(T))) + (xL(a(T)) |2)) || p(T))), where DKL. denotes the Kullback-Leibler divergence for latent distribution regularization. (iii) Spatial loss C., defined in Eq. Due to discrepancies between reconstructed and original representations, the pretrained spatial decoder may struggle deal with the reconstructed representation. To address this and further guide compression, we feed the reconstructed representations into the spatial decoder and fine-tune it jointly with the Spatial-VAE in an end-to-end manner. The overall latent token learning loss fuve = L. + Lisse + LKL. where is the weight for the KI. loss term.
1 2
Spatial generation learning As shown in Fig. 4 (b), with the pretrained Spatial-VAE, the 3D generation can be modeled as generating the latent token conditioned on a reference image and a view transformation. This latent token is then decoded into the visual representation of the target-view using the VAE decoder. Subsequently, the 3D scene is decoded by the fine-tuned spatial decoder to the visual representations from both the reference and target views. Therefore, spatial generation learning is naturally the process of conditional noise prediction on the noisy latent token.
During training, the relative view transformation between I' and I' is encoded as Plücker raymap (Plucker) 1865). represented as P E IR. N N., This raymap is then transformed into queries q using an MLP, so that suitable for processing by LLM. Subsequently, we feed the visual representation of I', Zz, along with the transformation queries q. into the LLM, which generates the conditional features
The next step involves training the model to predict the noise in the noisy latent tokens. We encode I''s visual representation Z into the latent token T via the pretrained VAE encoder. Gaussian noise is progressively added to the latent token T over several timesteps, creating noisy latent token TP' for each timestep Specifically, the noise is added according each timestep. The noisy latent token at each timestep is then passed through the denoising diffusion model along with the corresponding conditional features C. At each timestep i, the model learns to predict the noise e (T²² |C, t) added to the noisy latent token. The training target minimizing the discrepancy between the predicted noise and the ground-truth noise E:
schedule that increases with
Egen ETT ||ca(TT C,t) -cl =c1]]
At inference, we start from a noisy latent token Tr ~ ~V 1) and iteratively denoise it using the reverse diffusion process. At each timestep t, the model predicts the noise to be removed, updating the latent token by T = T, c (T.(C, t). Here, C represents the conditional feature generated by the LLM, which takes the reference visual representation Z, and an arbitrary view transformation as input. After T steps, the final latent token To is decoded into target view's ' visual representation Z, by the Spatial- VAE decoder, and the full 3D structure is then decoded by the fine-tuned spatial decoder using both 2, and Z,..
Spatial understanding learning Given an input image I and question Q, the model predicts an answer sequence a = at........ ax) } in an autoregressive manner. Firstly, the question Q is tokenized into language embeddings q = {go, q1 .l.,....... Together with the image representation 2, we feed these language tokens into the LLM. At step L, the LLM produces a distribution over the next token conditioned on ground-truth prefix acc. UniUGG is trained with teacher forcing using a token-level cross-entropy loss faya = -n' log log po(a,) |2, q. azz). Here, 0g is the ground-truth token at t. At inference, the prefix a=z is replaced by the previously generated tokens åzz.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Vision encoder pretraining We initialize our vision encoder with RADIOv2.5-L (Heinrich et al.. 2025), ViT-Large model with 24 Transformer layers and a hidden size of 1024. The encoder is followed by a ViT-Base decoder initialized from MASt3R (Leroy et al. 2024). We pretrained our encoder on a mixture of ARKitScenes (Baruch al.) (222)) ) and Scan Yeshwanth al.) (023) to capture geometric capabilities, and diversity. More training details can be found in Appendix A.2
et al. 2021) to capture semantic

---

Published as a conference paper at ICLR 2026
Training UniUGG The Spatial VAE is trained on 2M co-viewing pairs from ARKitScenes and ScanNet++,, with the KL loss weight ? = 0.0001. The detailed Spatial- VAE architecture can be found in Appendix A.2 For unified training in stage 3, the vision encoder and the Spatial-VAE are frozen, with only the LTMM, projector, and diffusion model optimized. The training process follows three steps: (i) Projector is trained LCS-558K QLiu al.. 2023) align patch-level features with the LLM embedding space; (ii) LLM, diffusion model, and projector are jointly optimized on 2.4M spatial cnstructtionfoolowing samples from ShareGPT4V (Chen et al., 2024d) and ALLaVA (Chen et al. 2024b), along with 2M co-viewing pairs from ARKitScenes and ScanNet++; (iii) Model is further finetuned on SPAR (Zhang et al. (025a),, EMOVA (Chen et al. (024c). and an additional 2M spatial sample pairs to enhance generalization in spatial QA and 3D generation tasks.
We use Qwen2.553B-Img IIstructt (Bai et al., 2023) as the L.LM backhone and stable-diffusion-il 5 (Rombach et al. (002) as the diffusion model. The A optimizer with cosine learning rate decay and a warm-up ratio of 0.03 is used. The learning rate set
10-3 for step (i) and 2 x 10-5 for steps (ii) and (iii) in unified training. The global batch size is set to 256. Additional 0.25M samples are used for generation comparison, separate from the training data.
Training cost and computational resources Our pipeline adopts a three-stage training strategy as illustrated in Fig. 2 In stage 1, we pretrain the geometric-semanee encoder for 25 hours using 8x NVIDIA A6000 GPUs. In stage 2, the Spatial-VAE module is pretrained for 12 hours the same 8x A6000 GPUs In stage 3, we train the full UniUGG model on both spatial understanding and 3D) generation tasks for about 46 hours During inference, we run our model on a single NVIDIA A6000 GPU. For spatial reasoning, given 16 input images at resolution of 224x224, UniUGG achieves an inference latency of 350ms, utilizing 6 precision and FlashAttention for acceleration. For 3D generation, UniUGG takes approximately 1.2s to generate a point cloud from a single reference image.
cluster with 8 nodes, each equipped with 8 Ascend NPUs.
Casual
MipNoRF360
Geometry
Texture
All
Geometry
Texture
All
Feature
|SNRT
SSIM
LPIPS.]
LPIPS_ SSMM
|SNKT
LPIPS.
PSNRT
SSIMT
LPIPS.|PSNK SSIMT
LPIPS.
|SNNK
LPIPS_
PSNRT
SSIM
LPIPS.
PPNKKT
SSIM
LPIPS
DINOv2
19.42
.6544
.3698
17.64
.5701
.3754
19.21
.6535
4023
20.81
.496
.3953
19.05
.4495
.3821
20.75
.424
.6844
CLIP
19.21
.6552
.3719
17.46
.5699
.3743
19.05
.6582
4084
20.80
.4982
.3913
19.28
.4543
.3807
20.88
.4984
.4733
DUSI/
19.29
.652
.3580
17.54
.5693
.3750
19.19
6556
4050
20.82
.5088
.3795
19.10
.4489
.3866
21.02
.5048
.4522
VGGT
19.36
6590
3549
17.47
5645
3351
19.23
6604
4103
20.93
5120
3639
19.25
.499
3828
21.17
5102
4892
RADIO
19.54
6545
3465
17.52
.6666
33748
18.67
6533
4216
20.87
5100
3620
19.35
4550
.819
20.91
.5667
.5277
MASER
19.30
6650
3576
17.59
5708
3722
19.37
6588
4027
20.92
5093
3745
19.21
4540
3803
20.92
5084
4749
Ours
19.80 .6643
.3449
17.81
.5850
.3599
19.18
.6693
.3955
21.28
5537
.3562
19.73
.4848
.3595
21.31
5264
4698
Table 1: Results of novel view synthesis metrics on Feat2GS benchmark. Our encoder outperforms others in most datasets across Geometry, Texture, and All probing modes. The best results are marked in bold, and the second best in underlined.
4.2 EVALUATION OF THE GROMETRIC-SEMANT ENCODER
Evaluation on Feat2GS benchmark The Feat2GS benchmark (Chen et al. 2024g) evaluates novel view synthesis as a proxy task for assessing 3D awareness, which defines three evaluation modes: (i) Geometry: only geometry parameters are predicted from encoder features. while the texture is free-optimized for novel view rendering; (ii) Texture: only texture is predicted, with the geometry free- optimized: (iii) All: both geometry and texture are predicted from features. The encoders compared include DINOv2 (Oquab et al. (023). CLIP (Radford et al., 202T). DUST3R encoder (Wang et al.) 2024b), VGGT encoder (Wang et al. 2025a), AM-RADIC (Ranzinger et al.) 2024b), and MAS13R encoder. As shown in Tab. [ our vision encoder outperforms baselines in all three probing modes, achieving significant improvements. Detailed results are provided in Appendix(A.33
Evaluation semantic perception and 3D vision tasks We comprchensively evaluate our pre- trained vision encoder across a diverse set of tasks, including monocular and video depth estimation, lmage-level reasoning, and pixel-level visual understanding. Our method achieves highly competitive performance across all evaluations. Detailed results are provided in Appendix |A.3.
Downstream task performance We assess the spatial understanding performance of pretrained encoder by integrating it into a unified Eision-Language Model architecture based on Qwen2.5-3B. Instruct (Bai et al. 2023). We evaluate wide range of lasion-language reasoning benchmarks.
7

---

Published as a conference paper at ICLR 2026
Method
Para.
VSI
BLINK
SPAR
Sced!
Real
World
(M)
Low
Med.
High
Qwen2.5-ViT
669
35.56
37.81
36.50
36.67
39.89
41.81
44.97
CLIP-L/14
305
40.08
40.45
44.13
43.67
52.33
69.14
54.38
Sigl 1
316
23.81
39.08
41.75
34.67
43.11
56.31
45.23
MASER Enc.
303
39.14
40,93
50.00
42.33
48,22
56,96
50.07
R.DIOv2.5-L.
320
39.75
42.92
50.44
47.95
52.13
72.09
$7.38
UniUGG Enc.
320
42.18
44,40
50.82
49.07
$1.89
21.65
58.56
Table 2: Comparison of encoder performance on downstream vision-language reasoning bench- marks. The VLM architecture is based on Qwwen..5-3B-nnsrruce and all encoders are trained under the same settings ensure faimess.
Spatial and geometric abilities are assessed through VSI-Bench (Yang et al. 2025). SPAR Zhang et al., (025a) and BLINK (Fu et al.] 2024b), while general language understanding is tested on Real (Miyanishi et al. 2021) and SEED-I (Li et al. 2024b). Compared models include both semantic-oriented encoders C/14, SigLIP-L/I6 (Zhai et al. 2023), ) RADIOv2.5-L) and geometry-aware design (MASt3R encoder). All encoders are initialized from pretrained checkpoints and fine-tuned with the LLM under the same settings for fair comparison.
As shown in Tab. 2 on VSI-Bench, BLINK, and SPAR, our encoder (UniUGG Enc.) demonstrates clear advantages spatial reasoning tasks, which require spatial relational understanding and geometric abstraction. Moreover, our method also achieves competitive performance on general QA benchmarks like RealWorldQA and SEED-I, showing that spatial enhancement does not significantly impair semantic generalization. Compared to the coomeryy-ocuseed MASt3R encoder, our encoder shows more consistent performance across modalities. In general, our encoder can bridge geometry and semantics in unified representation, balancing spatial perception and high-level semantics.
4.3 EVALUATION OF SPATIAL UNDERSTANDING
To evaluate UniUGG's spatial reasoning ability, ie., VSI-Bench, BLINK, 3DSRBench (Ma et al.
we assess it on representative benchmarks, 2024), and SPAR. We use three open-source
LMMs-LLaV. (Liu et al. 2024b). Intern VL2.3 (Chen et al.) 2024h). Qwen2.5VL. (Bai et al.)
2025)-and one proprietary LMM, GPT-4o (Achiam al. 2023). along with the ofate-of-the-art 2D unified framework Janus-Pro (Chen et al. 2025) for comparison. Results shown in Tab. 3 our UniUGG achieves superior performance across most benchmarks. In particular, on VSI-Bench, our model outperforms the second-bes one by 17.9%. It demonstrates that UniUGG can capture fine- grained spatial relations by jointly modeling 3D structure and ussual-anguage reasoning. Additional evaluations on SQA3D (Ma et al. 2022). ScanQA (Azuma et al. 2022). and ScanRefer (Chen et al., 2020) are provided in Appendix A.322 A..322 It should be noted that UniUGG designed for multi-view spatial understanding, where the model learns geometry from 2D inputs. Therefore, on these 3D benchmarks, our model still shows a performance gap compared to 3D-enhanced methods.
4.4 EVALUATION OF 3D GENERATION
Quantitative generation comparison and ablations We compare UniUGG with baselines and perform ablation studies to evaluate the quality of the generated outputs. Given reference image and view transformation, the setup generates the corresponding spatial structure for the novel view. The generated point cloud is then projected back onto the image plane, producing a colored 2D image. These generated images are compared the real images (Ground truth) using the Fréchet inception distance (FID), kernel inception distance (KID), and LPIPS. Quantitative and qualitative results are presented in Tab. 3 and Fig. respectively.
The encoder from our pretraining strategy (ID g) significantly improves generation quality, out- performing both the RADIOv2.5-L (ID a) and MASt3R encoder (ID b). This shows that simply incorporating geometric or semantic information is insufficient, and fusing both is more effective in unified framework. From the UniUGG settings, we observe a notable performance drop when the spatial decoder is not fine-tuned during VAE training (ID c). Additionally, omitting the Spatial- VAE and diffusion models (ID d), and having the LLM directly predict the target-view representation. re- sults in suboptimal performance. By the way, removing the Spatial-VAE. only and training generation
8

---

Published as a conference paper at ICLR 2026
SPAR
Method
VSI BLINK 3DSR
ARKitScenes
ScanNet++
Method
Low
Med.
High
Avg.
FIDJ
KIDP
FIDI
KIDI LID|LPPSS
LLaVA-vI.5-7B
18.0
37.1
38.1
10.9
26.5
34.1
23.7
w/ RADIO
64.16
.0518
.4904
.614
.4229
LLaVA-NeXT-7B
20.6
41.8
48.4
8.5
4.8
20.2
13.2
(b) w/ MAS(3R Enc.
81.18
.0691
.5776
86.79
0803
.5242
IntemVL2..588
(c) w/o Dec. finetune
149.97
.147
.301
168.05
1686
.945
Qwen2.5-VL-7B
30.3
56.4
48.4
28.8
23.0
40.3
33.1
(d) Diffusion
87.51
.345
GPT-4o
34.0
60.0
44.2
36.9
26.5
43.8
38.1
(c) CUT3R
138.54 1128
130.76
1051
.5637
+JanusPro--I
38.9
50.0
10.7
24.7
30.8
20.6
-Janus-Pro-7B
40.5
24.6
33.9
LVSM
269.45
.3088
.5067
414.63
5117
.8655
UniUGG-3B (Ours) 40.1
43.6
52.1
50.8
41.7
45.7
47.2
(g) UnIUGG (Ours)
55.01 .0425
4849
55.64 .0442 .2633
Table 3: Comparison of 3D understanding and generation performance. Left: 3D understanding performance on various spatial reasoning benchmarks. *denotes 2D understanding and generation method. Right: Quantitative spatial generation comparison ARKitScenes and ScanNet++ datasets. (a(a) to ID(d) represent the ablation of our model.
Reference
Ground truth
UniUGC(Ours)
RADIO encoder
w/ MASI5R encoder
w/o Decoder finetume
w/o Diffasion
Figure 5: Qualitative ablation on 2D projected views from 3D generation. Our UniUGG, including the geometric-semante encoder, Spatial- and associated training paradigm, leads to noticeably better generation results, in terms of geometric accuracy and color consistency.
directly on the original representation also fails generate valid results. These results demonstrate the Spatial-VAl VAE and related training paradigms are the key to successful 3D generation. Finally, we compare UniUGG with baselines, CUT3R (Wang al. 20255) and LVSM (Jin al. 2025). While CUT3R (ID e) predicts 3D structures from pre-observed data and raymap. and LVSM (ID generates target-view 2D images, both fall short in performance due their lack of imaginative capabilities. This highlights the superiority of our method in 3D generation.
Qualitative understanding and generation comparison We further qualitatively assess our UniUGG with CUT3R, shown in Fig. 6 From the perspective of the generated area, UniUGG accurately identifies which parts of the geometric structure need to be generated. Additionally, in terms of texture and semantic details, UniUGG effectively leverages the reference image to plausibly infer new structures, such as windows and chairs. We also demonstrate the understanding capabilities of UniUGG by generating captions for the scene from the generated visual representations. UniUGG can provide accurate descriptions of the 3D structure, even for parts that were previously unseen. In contrast, the baseline struggles complete missing regions and lacks coherence in structural details, let alone understanding the scene. These results highlight the strengths of UniUGG unified spatial understanding and 3D generation. We provide additional iealiizationsiinclldui feature matching, evaluation under extreme view transformations. and failure cases cases-in Appendix A.3.3
5 CONCLUSION AND LIMITATIONS
In this paper, introduce UniUGG, the first unified framework for spatial generation and under- standing, capable of spatial-level VQA and generating 3D scenes. We propose a geometric-semanti learning strategy to pretrain the vision encoder, enhancing its spatial modeling capabilities. This

---

Published as a conference paper at ICLR 2026
Reference image
right
Merge
Reference image
Yaw right
Merge
CUT3R
CUT3R
Caption
Caption
A window with two
batheooom
centered the
fottarig
imigio llanked by frostad glass panels the windowall th
shiff Newo
partully cuble, The nulls
thee and the overtl byyatt demes the witdfow the back
smail devengtrve them.
UnilIG (Ours)
UniLIGG (Ours)
Reference image
30"
Merge
Reference image
Fitch
right
left
CUT3R
CUT3R
Caplion
Caplion
A while bankshelf filled with books and hamed pictures, small decorabive item veuble on the top shelf of boolshelf
gray seckonil snfa peritiorned night with small table in frort of holding at the background, diningg anos is visibie
UnilIGG
UniLIGG (Ours)
Reference image
Fitch down 30"
Merge
Reference image
Pitch 40"
Merge
CUT3R
CUT3R
Caption
Caption
A clvie with a blrk
A avall effice gooze atth
bay der pusitionall
Jesk on the left. off diab behind ii.
belge rug, with
amall cabinet
a wiltehoond
the wall to left clis door i
left, The scane
no the rigt, lodting
indledes patterned the
another toom.
UniLIGG (Ours)
UniLIGG (Ours)
Figure 6: Qualitative 3D generation comparison. UniUGG accurately captures the input view transformation and leverages the reference image 'imagine' fine-grained spatial structures, and outputs correct captioning. In contrast, the baseline method only produces coarse and fuzzy geometry.
significantly improves both the generation and understanding aspects of our unified framework and yields strong performance on downstream tasks. Moreover, we design the Spatial-VAi VAE for achieving efficient 3D generation, and link the spatial decoder for fine-tune munituning to ensure sharper 3D scene decoding. Extensive evaluations showcasing UniUGG's ability to handle both 3D generation and spatial VQA tasks effectively. However, further enhancing 3D generation capabilities in unified models remains key challenge for future research. Our framework still has several limitations, including the lack of controllable generation driven by language and the inability to perform freeform editing of the generated content. As preliminary exploration toward unified 3D modeling, method does not yet support interactive multi-round scene generation and editing, which also needs to be solved in the future.
ACKNOWLEDGMENTS
This work supported in part by New Generation Artificial Intelligence- -ational Science and Technology Major Project (20252D0123004) Ningbo grant (2025Z038) and National Natural Science Foundation of China (Grant No. 62376060).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint, 2023.
Jcan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. NeurlPS, 2022.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint, 2025.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitscenes a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In NeurIPS Dalasets and Benchmarks Track (Round 1), 2021.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision- langugee models with spatial reasoning capabilities. In CVPR, 2024a.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV, 2020.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang. Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesieed data for lite vision-language models. arXiv preprint, 2024b.
Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint, 2024c.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, 2024d.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. NeurlPS, 2022.
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In CVPR, 2024e.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint, 2025.
Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-lIm with referent tokens. arXiv preprint, 2024f.
Yue Chen, Xingyu Chen, Anpei Chen, Gerard Pons-Moll, and Yuliang Xiu. Feat2gs: Probing visual foundation models with gaussian splatting. arXiv preprint, 2024g.
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries open-source multimodal models with model, data, and test-time scaling. arXiv preprint, 2024h.
11

---

Published as a conference paper at ICLR 2026
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint, 2024.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive kunuuag-image learning. In CVPR, 2023.
MMSegmentation Contributors. Mmsegmentation: Openmmlab semantic segmentation toolbox and benchmark, 2020.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang. Kaisheng Ma, and Li Yi. DreamLLM: Synergistic multimodal comprehension and creation. In ICLR. 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Mauthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition at scale. arXiv preprint, 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, al. Palm-e: An embodied multimodal language model. 2023.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015.
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint, 2024a.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang. Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In ECCV, 2024b.
Greg Heinrich, Mike Ranzinger, Yin Hongxu, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavlo Molchanov. Radiov2. 5: Improved baselines for agglomerativ vision foundation models. In CVPR, 2025.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NeurlPS, 2023.
Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. arXiv preprint, 2023a.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang. Qing Li, Song-Chun Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint. 2023b.
Runhui Huang. Chunwei Wang, Junwei Yang. Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al. Illume+: Illuinnaing unified milm with dual visual tokenization and diffusion refinement. arXiv preprint, 2025.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In CVPR, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- Lsioun-angguag representation learning with noisy text supervision. In ICMI., 2021.
12

---

Published as a conference paper at ICLR 2026
Haian Jin, Hanwen Jiang. Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang. Fujum Luan, Noah Snavely, and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. In ICLR, 2025.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023.
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In ECCV, 2024.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint. 2024a.
Bohao Li, Yuying Ge, Yixiao Ge. Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In CVPR, 2024b.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang. Qi Wang. Qiang Fu, and Ke Liu. A survey of multimodel large language models. In CAICE, 2024.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurlPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024b.
Wufei Ma, Haoyu Chen, Guofeng Zhang, Celso M de Melo, Alan Yuille, and Jieneng Chen. 3dsrbench: A comprehensive 3d spatial reasoning benchmark. arXiv preprint, 2024.
Xiaojian Ma. Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint, 2022.
Taiki Miyanishi, Takuya Mackawa, and Motoaki Kawanabe. Sim2realqa: Using life simulation to solve question answering real-world events. IEEE Access, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint, 2023.
Julius Plucker. Xvii. on a new geometry of space. Philosophical Transactions of the Royal Society of London, 1865.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Mike Ranzinger, Jon Barker, Greg Heinrich, Pavlo Molchanov, Bryan Catanzaro, and Andrew Tao. Phi-s: Distribution balancing for label-free multi-teacher distillation. arXiv preprint, 2024a.
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In CVPR, 2024b.
13

---

Published as a conference paper at ICLR 2026
Mike Ranzinger, Greg Heinrich, Pavlo Molchanov, Jan Kautz, Bryan Catanzaro, and Andrew Tao. Featsharp: Your vision model features, sharper. arXiv preprint, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Mert Bülent Sarıyldız, Philippe Weinzaepfel, Thomas Lucas, Pau de Jorge, Diane Larlus, and Yannis Kalantidis. Dune: Distilling a universal encoder from heterogeneous 2d and 3d teachers. In CVPR, 2025.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint, 2021.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint, 2023.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In CVPR, 2024.
Chamelcon Team. Chameleon: Mixed-modal early- feily-fusion foundation models. arXiv preprint, 2024.
Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your llms to see, draw, and self-enhance. arXiv preprint, 2024a.
Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint, 2024.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In CVPR, 2025a.
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continu- ous 3d perception model with persistent state. In CVPR, 2025b.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024b.
Chengyue WI, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In CVPR, 2025.
Yecheng Wu. Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint, 2024.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han. Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In CVPR, 2025.
Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality alignment for large language model. arXiv preprint, 2024.
Chandan Yeshwanth, Yuch-Cheng Liu, Mathhis Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of indoor scenes. In ICCV, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang. Jilin Mei. Junhui Chen, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, et al. From flatland to space: Teaching vision- languaee models to perceive and reason in 3d. arXiv preprint, 2025a.
14

---

Published as a conference paper at ICLR 2026
Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yujie Yuan, Xinyue Cai, Guowei Huang, Xingyue Quan, Hang Xu, and Li Zhang. From flatland to space: Teaching dision-languel models to perceive and reason in 3d. NeurIPS, 2025b.
Junyi Zhang, Charles Hermann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming- Hsuan Yang. Monst3r: simple approach for estimating geometry in the presence of motion. arXiv preprint, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning position-aware video represen- tation for 3d scene understanding. arXiv preprint, 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJC,, 2019.
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway empowering Imms with 3d-awareness. arXiv preprint, 2024a.
Yi Zhu, Zhou Yanpeng, Chunwei Wang, Yang Cao, Jianhua Han, Lu Hou, and Hang Xu. Unit: Unifying image and text recognition in one vision encoder. NeurIPS, 2024b.
A APPENDIX
In Appendix, provide additional technical details and more detailed experimental validations in terms of our method, comparison, and visualization.
A.1 USE OF LARGE LANGUAGE MODELS
We used GPT-4o polish the language and improve the clarity of our manuscript, such as improving grammar and phrasing. The model did not contribute to the core research content of the manuscript, including mathematical formulations, algorithms, and related results.
A.2 ADDITIONAL IMPLEMENTATION DETAILS
Vision encoder pretraining In the experiments, the patch size is set to 16x 16, producing 768 tokens of dimension 1024. The training uses AdamW optimizer with cosine decay over epochs, with loss weights Åz, 1.0, ALP 0.5.A1 1.0, X2 1.0, and semantic guiding loss weights a 0.9, 3 0.1.
Downstream task evaluation To assess the effectiveness of the pretrained encoder across downstream tasks, we adopt an experimental configuration in which the encoder, projector, and LLM are jointly trained in end-to-end fashion. While this setting follows the stage training pipeline as described in the Sec. 4.1 "Training UniUGG' part of the main text, it differs in two key aspects: the encoder updated during training, and no additional co-viewing sample pairs are included. This design allows us to evaluate the performance of different encoders on both spatial understanding and general reasoning tasks fairly.
We assess the downstream performance of our pretrained encoder and other encoders by integrating them into a unified Vision-Lann Languuge Model (VLM) architecture based on Qwwen2.5-BBInntrrice 3B-Instruct (Bai et al. 2023). All models are trained with the same pipeline, jointly optimizing the encoder, visual projector, and L.L.. This design ensures fair comparison under identical supervision and model capacity. All encoders are initialized from their respective pretrained checkpoints and jointly finetuned with the LLM under the same settings to ensure fair comparison.
Spatial-VAB VAE architecture. The detailed Spatial-VAB architecture is provided in Tab. The architecture follows an encoder-decoder design tailored for compressing and reconstructing visual representations. The encoder first reshapes the visual representation inputs into a 2D feature map, applies a series of convolutional and attention layers, and finally outputs the latent mean and variance for sampling. The decoder mirrors this process by reconstructing the visual representations from the
15

---

Published as a conference paper at ICLR 2026
Layer
Description
Output shape
Sptatial- VAE encoder
Reshaped imput
[b. 1024, 14, 14]
Initial convolution
[b. 256, 14, 14]
Upsample 1 (convtranspose)
128, 28, 28
Upsumple 2 (convolution)
[b. 128, 28, 28]
4
Transformer blocks
[b. 784, 128]
Flattened attention output
[b. 128, 28, 28]
j convolution
28, 28]
Log- Log-variance convolution
[b, 4, 28, 28]
Reparameterization
[b. 4, 28, 28]
9
KL divergence loss
scalar (mean of log variance)
Sptatial-VAE decoder
Input
[b. 28, 28]
Pre-convolution
[b, 128, 28, 28]
2
Flattened attention output
[b, 784, 128]
Transformer hlocks
[b, 784, 128]
Reshaped attention output
[b, 128, 28, 28]
Downsample (convolution)
[b, 128, 28, 28]
6
Downsample 2 (convolution)
[b, 256, 14, 14]
Final convolution
[b, 1024, 14, 14]
Output reshaped
196, 1024]
Table 4: Detailed Spatial-VAE architecture. Our model follows an encoder-decoder design.
sampled latent features using combination of attention blocks and convolutional layers. Transformer- based attention modules are used both the encoder and decoder to model long-range dependencies across spatial positions, enhancing semantic fidelity during compression and reconstruction.
A.3 MORE EXPERIMENTAL RESULTS
A.3.1 EVALUATION OF THE GEOMETRIC-SEMANN
ENCODER
Evaluation on Feat2GS benchmark We provide comprehensive and detailed results on Feat2GS benchmark (Chen et al. 2024g), as shown in Tab. 5 Results indicate that our encoder leads to the best performance on most datasets in Geometry, Texture, and All probing modes.
Single-frame and video depth estimation Following MonST3R (Zhang et al., 2024 we evaluate single-frame depth on the NYU-v2 (Silberman et al.] 2012) dataset and video depth on the BONN (Sil berman al., 2012) dataset, which cover dynamic and static scenes. These datasets are excluded from training, enabling zero-shot performance evaluation across domains. Our evaluation metrics include absolute relative error (Abs Rel) and percentage of predicted depths within a 1.25-factor of true depth (Ō < 1. 1.25). Following (Wang et al. 2024b). single-frame evaluation adopts per-frame median scaling, and video evaluation aligns a single scale and/or shift factor per sequence.
We compared our methods with DUSt3R (Wang et al.) 2024b). MASt3R (Leroy et al. 2024). Spann3R (Wang Agapito| 2024), and MonST3R. where these basclines are specially designed for 3D tasks. As shown in Tab. (leff), our method achieves competitive results compared to baselines, and even outperforms MAST3R in both single-fran frame and video depth evaluation.
Image/pixel level evaluation To assess the performance of our encoders, we adopt set of repre- sentative metries following (Ranzinger al., 2024b). For image-level reasoning, we evaluate our encoder using Top-1 k-NN accuracy and zero-shot accuracy on the ImageNet- KK dataset (Deng et al.) 2009). The zhro-shot accuracy is computed using the CLIP language model (Radford et al. 2021). For the k-NN evaluation, we first extract the summary feature for all training images. Then, for each validation image, we identify the k nearest neighbors the feature space and predict the label based on weighted vote of these neighbors. We also evaluate the generalization performance of our encoder on lixel-level visual tasks, including segmentation mIOU on ADE20k (Zhou et al.) 2019) and PascalVOC2012 (Everingham et al. 2015) datasets.
The encoders compared include SAM-H/16 (Kirillov et al. 2023). OpenAI CLIP-L/14 Radford
et al. 2021), SigL.IP-L/14 (Zhai al. 2023 DINOv2-g/ 4-reg (Darcet et al. 2023 DUSBR
16

---

Published as a conference paper at ICLR 2026
LLBY
BILDVV
Cannal
Oconndry
Toslass
Testace
A3
Guonatry
Touare
Feauur
[SNE
180%
SSIN)
190%
1.78%)
PSYRI
LMPS)
PINRI
LIFESL
PSN
5010
M5N81
175%
199%
(1500)
55:31
1P9%)
CLIP
15:39
-2378
2221
15/02
20115
19.04
71336
15.2
19.55
1295
.3334
15.05
.671]
3235
19,22
2200
3353
15.21
A352
5710
11.40
5660
5763
19.05
36882
4664
7990
16:01
77.79
2765
19.00
7190
2861
19.61
1166
2811
19:33
7794
3158
8987
7981
11-44
5093
3790
19:19
1040
RADIO
15:39
2207
19000
7101
2301
19.50
4500
1152
19.45
1513
3329
18.00
6548
.2254
19,25
2206
3634
19334
A343
3605
11.52
5000
5748
13.67
3655
4316
MASTR
7940
16:01
2761
7134
1184
18
8811
2373
1994
6991
5710
19:77
1998
1007
MiyNoRF340
Trasks and Torghk
Touars
Feauee
[15N
5500
1/85-
SSNI
LF85
1.785)
UMPS)
PINRI
20150
USFSL
PONS1
551M
11155,
1558
55MM
5500
1F85L|
55341
L7F51
D
70.81
.986
1951
19:05
1621
20.75
0904
15
3716
16:88
5115
19:11
9411
1670
18:71
6113
17:54
A714
1348
18:03
66.85
6164
CLP
20.30
.082
3913
15:23
A543
3000
20.00
A/64
4113
13.41
5942
.308
16.56
3362
.358
10.31
.993
-993
18:92
6963
5721
11:01
3023
2016
13.75
4515
4083
VOXT
2003
-3120
15:25
A407
3636
1::17
S1G
19.45
8019
2915
12.00
3333
J346
19.3
3953
.346
1821
11.25
A221
3319
2934
3898
4011
RADIO
20.37
-300
363
1833
A330
3919
2501
306)
3127
1936
2540
16.59
3333
J358
1983
.933
J746
10,13
2012
3690
31384
4023
3821
1230
3676
4100
2333
1672
3590
23:11
3288
4668
59.84
8300
2342
1331
3333
3714
18117
3577
0024
1855
3070
Table 5: Per-dataset results of novel view synthesis metrics on Feat2GS benchmark. Results indicate that our encoder leads to the best performance on most datasets in Geometry, Texture, and All probing modes. The highest, second-hipe hocond- and third-highor scores in each category are highlighted with light red. light orange and light yellow respectively.
Params ImageNetIK Segmentation
Method
Method
NYU-22 (Samel)-frame)
BONN (Video)
(M)
Zero-shot k-NN ADE20k VOC
Abs Rel
8 < 1.25
Abs Rel
1.25
SAM-H/16
637
22.12
28.08
34.34
OpenAl CLIP-L/14
305
75.54
79.96
36.51
67.04
DUSER
0.080
90.7
0.155
83.3
SigLIP-L/14
428
82.61
85.16
40.53
70,31
MonST3R
0.102
88.0
0.067
96.3
DINOv2-g/14-re
1,137
83.41
48.68
82.78
Spann3k
0.122
84.9
0. 144
81.3
DUS:3R Enc.
303
32.10
46.02
DUNE- BENNE-B114444 B/14-448
420
45,60
MASB3
0.129
84.9
0.252
70.
MASI3R Enc,
303
32.54
48.58
Ours
0.070
93.9
0.086
91.4
#RADIOv2.5-L,
320
80.55
83.16
50.68
85.60
UniUGG Enc. (Ours)
320
80.06
83.13
50.12
85.43
Table 6: Left: Depth evaluation results. We report single-frames depth evaluation performance on the NYU-v2 dataset and video depth evaluation performance on the BONN dataset. Right: Comparison of encoder performance on the image/pixel level. 'Zero-Shot' and k-NN are computed on ImageNet- 1K. ADE20K and PascalVOC220 2 refer to linear probe semantic segmentation mIOU. edenotes teachers used to pretrain our encoder.
encoder Wang et al. 2024b DUNNE-B144448 (Saryyldız et al. 2025) .MAS(3R encoder Leroy et al. 2024) and RADIOv2.5-L Ranzinger et al. 2024b).
Following (Ranzinger et al. 2024b), we freeze the vision encoders and train a linear head on top of the frozen features. The linear probe is conducted in the MMSeg (Contributors) 2020) framework. We train the linear head for 20k steps using a total batch size of 64, base learning rate of 5e -3 and the Adam-W optimizer.
As shown in Tab. 66) (right), while our method does not outperform the teacher baseline, it yields competitive results that validate the potential of our encoder. More importantly, it establishes a strong foundation for unified spatial reasoning and 3D generation.
Downstream task performance As shown in Tab. we provide more detailed results about Tab. of the main text. Our encoder (UniUGG Enc.) demonstrates clear ad advantages on spatial reasoning tasks and two general QA benchmarks. It shows that our encoder has more consistent performance across modalities, balancing spatial perception and high-level semantics.
A.3.2 EVALUATION OF SPATIAL UNDERSTANDING
Detailed spatial understanding scores As shown in Tab. g we present more fine-grained spatial understanding scores on 3DSRBench and VSI-Bench (corresponding to Tab, of the main text). We jointly train UniUGG for both spatial understanding and 3D generation tasks. The model utilizes our encoder as the visual backbone and employs the large language model. The results demonstrate that UnIUGG capture precise spatial relations by jointly modeling 3D structure and visual- visual-language reasoning. It should be noted that the LLM used in UniUGG has size of only 3B parameters.
pretrained coonetric-semanee
1--22.53---Insttru as
17

---

Published as a conference paper at ICLR 2026
Params
VSI-Bench
SPAR
Method
(M)
Count
Obj.Size
Room
Rel.
Rel.
Route
Order
Avg.
Low
Medium
High
Qwen25-Vii VIT
669
58.00
51.12
31.04
37.31
29.44
25.26
21.20
35.56
36.50
36.67
39.89
OpenAI CLIP-1/114
305
61.43
49.30
49.31
39.45
36,20
30.93
16.18
40.08
44.13
43.67
52.33
SigLIP-L/16
316
11.93
42,06
0.73
38.95
27.89
28.87
9.71
23.81
41.75
34.67
43.11
MAS:3 Encoder
303
58.12
39.81
48.82
43.70
33.24
32.47
19.58
39.14
50.00
42.33
48.22
RADIOV2.5-I.
320
59.91
53,49
50.17
37.08
32.54
16.18
39.75
47.95
52.13
UniUGG Enc. (Ours)
320
62.69
51.70
49.34
42.14
34.37
32,99
27.51
42.18
50.82
49.07
51.89
Params
BLINK
Real
Method
Seed-1
World
(M)
Fun.Corr.
Vis.Corr.
Local.
Jigsaw
Depth
Spatial
Simi.
Art
Avg.
Qwen25-Vii
669
15.38
26.16
54.92
46.67
44.35
52.45
46.67
$2.99
37.87
41.81
44.97
OpenAl CLIP-L/14
305
24.62
24.42
52.46
56.00
45.97
63.64
43.70
52.99
40.45
69.14
54.38
SigLIP-L/16
316
20.00
20.35
61.48
51.33
46.77
53.85
53.33
56.41
39.08
56.31
45.23
MAS:3 Encoder
303
22.31
27.33
55.74
48.67
44.35
67.13
44.44
45.30
40.93
56.96
50.07
RADIOV2.5-L.
320
23.08
31.98
47.54
43.33
65.53
47.31
53.85
72.09
$7.38
UniUGG Enc. (Ours)
320
33.85
33.14
55.74
48.67
69.35
67.83
51.11
59.85
44.40
71.65
58.56
Table 7: Detailed comparison results of encoder performance downstream vision-language reasoning benchmarks. The VLM architecture is based on Qwen2.5-9BBIInsrruec, and all encoders are trained under the same settings to ensure fairness.
3DSRBench
VSI-Bench
Method
Height Loc. Orient. Multi, Avg. Obj Count Dist. Ohj Size RoomSize Rel. Dir. RoutePlan Appr Avg.
LLaVA-v1.5-7B
1-BBB
39.1
46.9
28.7
34,7
38.1
6.2
4.9
32,6
2.7
29.6
30.7
26.3
10.5
18.0
L.aVA-NeNT-7B
50.6
M6.
484
7.5
27.7
25.8
31.2
29.7
V.temVVL2.58B
45.9
68.1
38.7
43.3
50.9
7.7
32.6
42.9
34.6
39.6
40.0
24.7
37.7
32.5
Qwen2.5-VL.-77
44.1
62.7
40.6
40.5
48.
26.7
10.8
35.4
31.0
35.2
38.2
35.1
29.6
30.3
GPT-4o
53.2
59.6
21.6
39.0
442
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
34.0
UniUGG-3B (Ours)
43.4
63.2
49.1
50.4
30.3
Table 8: Detailed spatial understanding scores on 3DSRBench and VSI-Bench. Our UniUGG is
jointly trained for both spatial understanding and 3D generation tasks. Note that the LLM used in UniUGG has a size only 3B parameters.
Evaluation on 3D-specific benchmarks We further evaluate the 3D reasoning capability of our UniUGG on SQA3D (Ma et al.) 2022), ScanQA (Azuma et al.) (022), and ScanRefer (Chen et al., 2020), which are widely adopted benchmarks in 3D VLMs. Due to the differences in QA formats, and following the practice of prior works, we apply supervised fine-tuning on 3D tasks to better assess UniUGG's spatial capabilities. First, we evaluate spatial understanding using SQA3D and ScanQA, which focus on object attributes, spatial relations, and deeppoint-oonditooned reasoning. Compared models include 3D-LLM (Hong et al., 2023), Chat-3D v2 (Huang et al.) 2023a). LEO (Huang al., 2023b). LL3DA (Chen al. (024e)), Scene-LLMM (Fu et al.) 2024a). LLaVA-3D (Zhu et al. 2024a and Video-3D L.M (Zheng et al. 2024). As shown in Tab. 9 (left), although our model does not incorporate explicit 3D information, it achieves competitive performance compared to 3D 3D-enhanced LLMs. Additionally, we assess 3D grounding
the ScanRefer dataset, where the task requires localizing target objects based on textual descriptions. Compared models include ScanRefer, MVT (Huang et al. 2022). ViL3DRel (Chen et al. 2022). 3D-LLM, Chat-3D v2, Grounded 3D- LLM (Chen et al.) 2024h). LLaVA-3D, and Video-3D 3D LLM. As shown in Tab. 9 (right), our model may struggle to predict depth accurately due to the absence of 3D information, resulting in poor grounding results. To address this, we apply refinement strategy to mitigate this issue and improve grounding performance. These results highlight the potential of our UniUGG to perform spatial reasoning and grounding without relying on explicit 3D inputs. while still supporting more complex generative tasks. It should be noted that our model employs backbone with 3B parameters, while other methods, for example, LLaVA-3D and Video-3D-LLMM, use 7B 1--aammmtter backbones.
A.3.3 EVALUATION OF 3D GENERATION
More qualitative results of 3D generation We provide more visualization results to further demon- strate the 3D generation and understanding capabilities of UniUGG. Given reference image, randomly sample plausible relative view transformations and let UniUGG generate the corresponding 3D scenes. UniUGG further captioned the generated 3D scenes. As shown in Fig. 7 Fig. 8 Fig.
18

---

Published as a conference paper at ICLR 2026
Methods
Acc@0.25
A.c@0
SQA3D
ScanQA
Method
3D
ScanRefer
37.3
EM
BLEU-4
CiDEr
EM
MVT
40.8
33.3
3D-LLM
12.0
69.4
20.4
ViL3DRel
47.9
37.7
Chat-M
54.7
14.0
87.6
3D-LLM
30.3
LEO
50.0
13.2
101.4
21.5
Chat-3D v2
35.9
30.4
LL3DA
13.5
76.8
Grounded 3D-LLM
47.9
44.1
Seene-LLM
54.2
12.0
80.0
27.2
LLaVA-3D
54.1
42.4
LLaVA-3D
55.6
14.5
91.7
27.0
Video-3D LLM
58.1
51.7
Vidco-3D LLM
58.6
16.2
102.1
30.1
UniUGG (Ours)
51.3
12.1
85.6
24.4
+UniUGG (Ours)
23.2
7.8
UniUGG (Ours)
41.9
36.6
Table 9: Perforeeform comparison SQA3D and ScanQA. "3D" indicates whether the model is infused with 3D information. Right.Performance comparison across different models on ScanRefer. sindicates results without refine.
ARKitScenes
ScanNet++
Method
VSI BLINK 3DSR SPAR
Method
FIDJ, KIDJ LPIPS) FID4 LIDI
UniUGG (full)
40.1
43.6
52.1
47.2
UniUGG (full)
55.01 .0425
.4849
.0422 .4263 55.46 .0454 .3586
55.64
UniUGG (und. only) 42.2
44.4
51.3
49.8
UniUGG (gen. only) 54.35 .0409 .4566
Table 10: Performance comparison with separate optimization.
and Fig. 10 UniUGG consistently produces geometrically coherent and semantically meaningful 3D content, along with accurate scene captions.
Feature matching visualization By taking only a reference image and the relative view transforma- tion input, UniUGG predicts the ViT tokens of the target image, which, together with the reference tokens, are then decoded into point clouds and the corresponding cross-view feature matchings. We visualize these feature matching results in Fig. T1 and compare them with the baseline method MASI3R (Leroy al. 2024). which requires both the reference and ground-truth transformed-view images as input to obtain feature matching. Our method achieves highly accurate feature matching, which is highly consistent with those produced by MASt3R. This demonstrates that UniUGG not only generates the spatial structure under the novel view but also ensures that the generated content is generally consistent with the input view transformation condition.
Robust generation under extreme view transformation We illustrate the generation performance of UniUGG under extreme view transformation conditions, shown in Fig. 12 We evaluate the model performance conditioned on rotation angles of 60%, 80", 100", 120%, and 140 As observed, within the range of 600- UniUGG still produces high-quality, semantically rich, and structurally coherent 3D point clouds. However, at 140% or beyond, the generation quality drops noticeably. with the point clouds exhibiting blurred textures and distorted structures. This degradation arises from two factors. First, during training, the new-transformatten view overlap ratio 0.4, rel-translation m, and rel-rotation < 120". Consequently, when evaluated under extreme view transformation (e.g., rotation > 140"), our model struggles to maintain generation quality. Second, such large viewpoint changes significantly reduce the overlap between the reference and target images, resulting in insufficient cross-view constraints and finally leading to failures in 3D generation.
conditions are usually constrained within:
Failure case Due the lack of training samples with large view transformations, UniUGG would generate point clouds with blurred textures and distorted structure under extreme viewpoint changes, shown in Fig. 12 What's more, we also provide some examples of failure cases in Fig. B In some cases, UniUGG generates point clouds with color distortions, where vivid green regions are mistakenly inferred grayish colors. This may be caused by conflict between color decoding and semantic representation tasks during encoder pretraining.
A.3.4 PERFORMANCE OF SEPARATELY OPTIMIZED
As shown in Tab. 10 we separately optimize the spatial understanding task and the 3D generation task, and compare their performance with that of the jointly trained model. Benefiting from the Spatial- VAE and diffusion model, joint training (UniUGG full) does not significantly degrade the performance of 3D generation (UnUUGG gen. only) and only leads to moderate reduction in
19

---

Published as a conference paper at ICLR 2026
Caption
pls spaat sira viecaly sollndd Earigi madile
adaad sal an portone caleen Tlesde coverssiy wirecattonna iotte
Reference image
Refeeence
Kandom
Caption
Marge
Soo puckor us to doogl bille sabiecand
wilker ciam on
war au di sn
Random seed
Caption
Menge
Random seed
Capticn
Mergee
buteet aakie walk wiia ply
worath pohlas cine
Reference image
Eal bestec tike
Referenee image
potilly williii
Caption
Marge
Kandom
Caption
Mengo
th upmp ailcki
wenden gdlin parieed nar
Ma veth aucuptac
Torel otpere fhreal
ancrieeblle lgft
Figure 7: Additional visualization samples. We present 3D scene generations and the corresponding captions produced by our UniUGG under varying random seeds.
Random sexal
Capeion
Merge
Random soed
Caption
Merge
rechonal wile
aill wok undll andernediin covele
Had pirll nidt no cs uod morkey
Reference image
Reference image
fis en pelocied mothe
Randem seed
Bandon sced
Caption
s-d forund, meak cutereen in neti ville fires
gill seca
gaim
the cs to withe
Langun
clowrng adled lig hore
Random
Caption
Merge
Random seed
Caption
Merge
Aurie well careeii be more firiih indi lusk thar
Agin sululioh accenal with un anl codiow tharggt.
Refereace
Reference
anciia sovn rall care
Randan veul
Caption
Merge
Randon sood
Caption
Merge
Alomgnom lncwn
you ie clened caaents theneuthi dak Ric ands tolk
sit oe Deright uppa br
i- oden,
Figure 8: Additional visualization samples. We present 3D scene generations and the corresponding captions produced by our UniUGG under varying random seeds.
Capiion
Mergo
Random
Caytton
Merge
Aloeee fertoe with
huloattan on Dalon sosnteed wablo patall
Reference Inage
image
Random seed
Capriion
Mesge
Randon
Captica
Mergee
Anoly raached with und medinil chliroo affillingg dull dus wi plake
Ramlan vood
Caprioon
Mesge
Caption
Merge
Armel pastonidio dhomulee
otteat usd tren fallinga bet fowe lemied heuceee sublur the Seiiiial
Refernne image
Reference
podee becia asside
Random sned
Capiion
Margo
Ranlon
Caption
Merge
bec ahe
mater wilh from dan an cas nnt
cove vilcow sei moreii wilte solle
ald
Figure Additional visualization samples. We present 3D scene generations and the corresponding captions produced by our UniUGG under varying random seeds.
20

---

Published as a conference paper at ICLR 2026
Reference Generated part
Generated
Ariid son fate
fellff tnd commina morrs
right uctood to menddr Anle colerciien corro
abovn dai boll homn.
opea wtth corpatt selvete opes by and
aet Riiecbop
gueetdeed cas lovd
law ca dinse vos voidos
wonk aekkard baldng compra solerd
on searos
rellorril Piuemoed lubo.
heral adiite catains
cilling
socs and to socil voodan
A Aorrect laidd withs wate mum led kities nucde
ontrr i Surody than
Figure 10: Additional visualization samples. We present 3D scene generations and the correspond-
ing captions produced by our UniUGG.
Reference image
Reference image
Unseen
MASI3R
UniUGGG(Ors)
Figure 11: Feature matching visualization. Without access to the target image, our method takes the reference image and the relative view transformation as input and still produces accurate feature matchings, while baseline MASt3R takes image pairs input.
image
point
target view
image
point
target
View transformation 60*
View transformation angler 80"
View traneformation 100"
View transformation angle: 120
View transformation angle: 140-
View transiormation 240-
Figure 12: 3D generation under extreme view transformation. When the rotation angle is below 120%, UniUGG can still produce high-quality 3D point clouds under the target viewpoint. As the rotation angle increases to 140% or beyond, the quality of the generated results degrades noticeably.
21

---

Published as a conference paper at ICLR 2026
Reforence image
Generated point cloud Generated target
Reference image
cloud
Generated target
Figure 13: Other failure cases of our model.
Simplified generation-uneeeeee inwrrtiionuuderrsadddngggg learning
Simplified generation-undersanding inference
Understanding
Novelalew
Question
turnfornnctinn
G: How many crairs
Shown in this pictune! A. Them are four chains
Reference amage
Que toluro
Image
crepdling
Tokeniar
Encode
Encodo
Large model
Imaga reprostaiion
Tenefomalio quaries
Qunikaa
Prodited
Large language model
Spatial
Refererce cloud
Thers
MAE Joss
Cross-entoppy Joss
Nowelview
point cloud
tible.
Jerage
Image Iepresntation
Figure 14: Overview of our simplified model variant pipeline. In the simplified variant, we do not use the diffusion model and Spatial-VAL. for scene generation but directly produce the novel-view tokens by LLM, which is supervised with ground truth tokens.
spatial understanding (UniUGG und. only) performance. We believe this performance trade-offi is acceptable: although small amount understanding performance is sacrificed, we obtain a unified 3D model for both understanding and generation. This unification substantially reduces training cost and computational overhead while broadening the range of tasks the model can support.
A.4 EXPLORATION OF SIMPLIFIED MODEL VARIANT
During the development of our proposed UniUGG, we explored the early or simplified version of the model architecture. This helped us better understand the role of the core component in the model. Although not intended the final model, the simplified variant provides valuable insights for designing a unified framework for 3D understanding and generation.
As shown in Fig 14 (a), during training, LLM learns both spatial generation capability across views by predicting ViT tokens the novel view, and visual understanding capability via question answering As shown in Fig 14 (b), inference, the model performs either novel-view spatial generation or question answering using a single image as reference, conditioned on the input type.
Simplitied (Chm))
Simplitied (Cam)
Plshk w747
Fa
Taa Ach
Yau lal
You oahe
You RightS
Figure 15: Visualization of scenes generated by the simplified model variant. Although the simplified variant can imagine spatial structures under novel views, it lacks clear details.
22

---

Published as a conference paper at ICLR 2026
We present visualization generation results of the simplified variant in Fig. 13 While it outperforms the baseline CUT3R in terms of generating fine- ame-grained spatial structures, its textures remain notice- ably inferior to those of our full UniUGG. This performance gap is primarily attributed to the absence of the diffusion prior and Spatial-VAF, which play a crucial role in modeling high-frequency visual details and realistic textures. The simplified variant, lacking these core components, tends to generate over-smoothed surfaces.
23


Reviews:
[{"review_id": "2hBx6qCTx8", "reviewer": "Reviewer_D7LE", "summary": "The paper proposes a method to ingest images and can generate extended scenes based on text input or camera parameters. How this works is unclear to me. I've read the paper 3 times and I still don't fully understand how the different parts work together.", "questions": "- **Q.1:** Doesn't the use of a VAE in the second step defeat the purpose of semantically aligning the encoder in the first step?\n- **Q.2:** Why isn't the LLM trained directly to output tokens that can be decoded into visual representations by the VAE decoder, rather than predicting something that can be rendered through a diffusion process?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 1, "presentation": 1, "contribution": 4, "strengths": "- **S.1:** The qualitative results look great.", "weaknesses": "- **W.1:** I've worked with LLMs, I've worked on diffusion papers, and I know about the Mast3r paper. Yet, I have no clue what's going on in this work and why. I think the writing is a bit too dense to follow unless the reader is already intimately familiar with all the related works. And I also don't think the figures are doing a great job at illustrating the method. I can't say anything more about this work. Maybe an overarching, vastly simplified diagram would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6rEAoshzZw", "reviewer": "Reviewer_Qmfc", "summary": "The paper proposes an LLM-based 3D framework for spatial-VQA and 3D scene generation. It addresses the limitations of current vision-language models, which are largely restricted to 2D semantics and next-token objectives that fail to capture spatial and geometric structure. To this end, the paper proposes a geometric-semantic pretraining strategy for the image encoder using multiview supervision. A spatial-VAE module then compresses the geometric-semantic features into compact 4-dimensional latent tokens for geometry-conditioned image synthesis. The model combines a geometry-aware encoder, a spatial-VAE model and a language conditioned diffusion generator. Experiments are done on Feat2GS benchmark for novel-view synthesis and several vision-language reasoning benchmarks.", "questions": "1. There’s no discussion or evaluation of 3D output. Does the model learn explicit 3D geometry or only 3D features?\n2. How does the model perform if the full model is trained end-to-end?\n3. In the language-driven view transformation, how accurate is the angular or spatial accuracy?\n4. What is represented by the 4-dimensional spatial-VAE tokens?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The encoder is trained using geometric supervision which enables spatial reasoning compared to models trained on 2D images.\n2. A single latent representation is used for both 3D reasoning and 3D generation.\n3. The method allows geometric control through text or parametric view prompts.", "weaknesses": "1. There’s no discussion or visualization of the 3D output from the model.\n2. All the evaluations are done in 2D space and there is no 3D evaluation performed. It’s unclear if the model is actually learning the structure of the 3D world or simply rendering based on the distribution of the training data.\n3. The model is not trained end-to-end. The encoder and spatial VAE are trained separately and kept frozen during unified training (lines 301-302).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WQcu6oxqF6", "reviewer": "Reviewer_bejZ", "summary": "The authors propose a unified framework for 3D understanding and generation, introducing a geometric–semantic learning strategy for encoder training based on the MASt3R architecture. The method includes a Spatial-VAE that compresses visual features into latent tokens for efficient 3D decoding, and a joint training setup where a large language model (LLM) and a diffusion model predict new views. Additionally, the model is trained with visual question answering (VQA) tasks to maintain its 3D understanding capability. Quantitative and qualitative experiments are conducted on both 3D generation and 3D understanding tasks across various spatial reasoning benchmarks.", "questions": "1. Could the authors elaborate on the main limitations and share examples of failure cases?\n2. How sensitive is the model to large viewpoint changes in the 3D generation task?\n3. I am also curious on whether joint training on understanding and 3D generation introduce any performance trade-offs between the two tasks.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unifying 3D understanding and generation is an interesting and novel research direction.\n2. The proposed geometry–semantic encoder is well-designed; incorporating the MASt3R framework is an effective and interesting choice.\n3. The paper is well-written, and the experiments are thorough, covering a diverse set of novel view synthesis and vision–language spatial reasoning benchmarks.", "weaknesses": "1. The paper would benefit from a more detailed discussion of limitations and failure cases.\n2. The method’s robustness to more extreme view transformations is not clearly evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dHfmudD5Hx", "reviewer": "Reviewer_4To6", "summary": "This paper presents UniUGG, which is a unified model for both 3D understanding and generation. Specifically, both geometric and semantic information of input images is encoded with the visual encoder. Also, a spatial decoder is used for 3D generation. Experimental results show that the proposed unified 3D understanding and generation method achieves superior performance for the visual representation of the visual encoder, spatial understanding capability, and 3D generation capability.", "questions": "- How will the model perform if understanding and generation tasks are separately optimized? If separately optimizing them can achieve better performance for each of them, then there is limited justification for modeling them together. Otherwise, the cases (e.g., interactive multi-round scene editing like those in Uni3D-LLM [1]) that requires joint understanding and generation capabilities need to be shown.\n\n- How will the model perform in the more perceived 3D understanding / reasoning tasks, like SQA3D, ScanQA, ScanRefer that are more used in 3D VLMs?\n\n- What is the training cost of this work? The paper only mentions about the batch size and the number of optimization steps of the training the model, without mentioning what kind of GPU resources they are using, and how much time is needed for training the model, which makes it unclear how efficient this algorithm is.\n\n[1] Liu et al. Uni3D-LLM: Unifying Point Cloud Perception, Generation and Editing with Large Language Models. arXiv:2402.03327.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- Having a unified model for 3D understanding and generation is pioneering, which can probably serve as the basic work for the unified 3D model direction. \n\n- The idea of binding a MASt3R-powered spatial decoder with LLMs is innovative and reasonable to me, which enables the whole system to output something that is spatial-aware, like the point cloud in the generation part of this work.\n\n- The proposed method achieves decent experimental results on the benchmarks for visual representation, spatial understanding, and 3D generation, demonstrating that it is a well-rounded unified 3D model.", "weaknesses": "- It remains unanswered for the motivation of proposing such a unified 3D foundation model for both understanding and generation. In other words, there is lack of evidence that modeling them together in a single model can provide mutual benefit to each other. If there is no mutual benefit between these two streams of tasks, or even slightly harm each other, it will become unjustified for us to design such a delicate framework for unifying understanding and generation for 3D, because optimizing them separately may produce optimal performance.\n\n- I think it is a bit overclaiming to say the understanding task UniUGG solves are 3D understanding. Most of the tasks (like the ones in VSI-Bench, BLINK) are the vision-language tasks related to spatial reasoning, with limited relation with 3D understanding. I would say the benchmarks like SQA3D [1], ScanQA [2], ScanRefer [3] are more related to 3D understanding, as they are understanding and reasoning with 3D representations, instead of only reasoning on single (or sometimes multi-view) images for some spatial understanding tasks.\n\n- The writing of the methodology section (Sec. 3) makes me feel a bit difficult to understand how the proposed UniUGG framework looks like. It starts with introducing the visual encoder, which I find a bit lost because I even do not have a rough idea in mind about what modules the full UniUGG framework has. Therefore, I find confusing where is this visual encoder locates in the whole framework. The introduction of the following modules also share the same problem. Thus, I believe this section lacks a part that introduces the rough whole pipeline to audience at the beginning, for them to have a big picture in mind.\n\n- The pipeline figures (Figures 2-4) are also not clear enough. They are crowded with small text, and are hard to understand. Basically, I cannot find a clear figure showing what the inputs and outputs are for the model. The contents inside the figure are also confusing. For example, in Figure 3(b), the large language model seems to take in image 2 representation together with transformation queries and question & answer tokens. Is image 1 the target image that we want to generate, so it is not processed through large language models. And the encoding of image 1 and image 2 seems to be messed up with the representation of image 1 and image 2.\n\n[1] Ma et al. SQA3D: Situated Question Answering in 3D Scenes. ICLR 2023.\n\n[2] Azuma et al. ScanQA: 3D Question Answering for Spatial Scene Understanding. CVPR 2022.\n\n[3] Chen et al. ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language. ECCV 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)