UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis
Authors: Tung Le
Tuan Pham
Tung Nguyen
Deying Kong
Xiaohui Xie
Stephan Mandt
Venue: NeurIPS 2025 poster ()
Keywords: Novel View Synthesis
Autoregressive Models
Diffusion Models.
Primary area: deep_learning
TL;DR: We propose a method that unifies deterministic feed‑forward rendering with autoregressive diffusion to synthesize photorealistic novel views from sparse inputs in a single transformer framework.

Abstract: Novel view synthesis (NVS) seeks to render photorealistic, 3D‑consistent images of a scene from unseen camera poses given only a sparse set of posed views. Existing deterministic networks render observed regions quickly but blur unobserved areas, whereas stochastic diffusion‑based methods hallucinate plausible content yet incur heavy training‑ and inference‑time costs. In this paper, we propose a hybrid framework that unifies the strengths of both paradigms. A bidirectional transformer encodes multi‑view image tokens and Plücker‑ray embeddings, producing a shared latent representation. Two lightweight heads then act on this representation: (i) a feed‑forward regression head that renders pixels where geometry is well constrained, and (ii) a masked autoregressive diffusion head that completes occluded or unseen regions. The entire model is trained end‑to‑end with joint photometric and diffusion losses, without handcrafted 3D inductive biases, enabling scalability across diverse scenes. Experiments demonstrate that our method attains state‑of‑the‑art image quality while reducing rendering time by an order of magnitude compared with fully generative baselines.

UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis
Thanh-Tung Le
Tuan Pham"l
Tung Nguyen 2
Deying Kong Xiaohui Xie UCI UCLA Google
Stephan Mandt
x6
Single view
Three
Arbitrary views
views
Figure 1: UMAMI synthesizes photorealistic novel views from sparse inputs. Shown are single-view generation, three-view extrapolation, and six-view reconstruction. The hybrid model fuses deter- ministic rendering with diffusion-based completion for unseen regions, yielding fast and consistent results without explicit 3D priors.
Abstract
Novel view synthesis (NVS) seeks to render photorealistic, 3D-consistent images scene from unseen camera poses given only sparse set posed views. Exist- ing deterministic networks render observed regions quickly but blur unobserved areas, whereas stochastic diffusion based methods hallucinate plausible content yet incur heavy training- and inference-time costs. In this paper, we propose hybrid framework that unifies the strengths of both paradigms. A bidirectional transformer encodes multi-view image tokens and Plücker-ray embeddings, pro- ducing a shared latent representation. Two lightweight heads then act on this representation: (i) ferw-forward regression head that renders pixels where ge- ometry is well constrained, and (ii) a masked autoregressive diffusion head that completes occluded unseen regions. The entire model is trained end-to-end with joint photometric and diffusion losses, without handcrafted 3D inductive biases, enabling scalability across diverse scenes. Experiments demonstrate that our method attains state-of-the-an image quality while reducing rendering time by order of magnitude compared with fully generative baselines.
These authors contributed equally to this work. t denotes corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Novel view synthesis (NVS) has long been a central problem in computer vision and graphics, aiming to generate realistic, 3D-consistent images of scene from new camera viewpoints, using given set of input views with known poses. Traditional methods often require dense input views, treating NVS as sequential 3D reconstruction and rendering task [49 34]. Recently, modern deep network priors 998
21 have been proposed to address the sparse views reconstruction problems, and achieve realistic rendering results.
Two dominant strategies have emerged for sparse-view view NVS using deep networks: deterministic and generative- methods. Deterministic methods often build generalizable networks that predict novel views by incorporating explicit 3D inductive biases
X3 or by leveraging priors from 38 31 with minimum inductive bias. While these approaches
large-scale reconstruction models 27 can be effective and fast in rendering observed regions, they often struggle with uncertainty in unobserved areas, leading to blurry predictions. Conversely, generative NVS approaches [87 21 can generate plausible content for unseen regions, These methods typically employ pretrained input views and camera poses. However, despite their strong
diffusion models conditioned generative capabilities, they often require extensive training data and computational resources, and their iterative sampling process leads to slow rendering speeds.
In this work, we address the question: "Can we combine the rendering efficiency of deterministic models with the generation capabilities of generative models?" We aim to unify these disparate approaches into single, efficient framework. We observe that conventional diffusion models [102] 21. which iteratively generate full images using large UNet or Transformer backbones, can be inefficient if significant portions of the target view are already observable and could be rendered directly by feed-forward network.
To this end, we introduce UMAMI, Unifying Masked Autoregressive Models and DeterminIstic Rendering for View Synthesis, a novel hybrid framework for NVS from sparse inputs. Our approach integrates a masked autoregressive model, trained with a diffusion loss [40], alongside deterministic rendering head. Specifically, drawing inspiration from recent feed-forward NVS models [31,, we employ a transformer with bidirectional attention. The model encodes input multi-view images tokens and masked target image tokens, conditioned on Plücker ray embeddings for both input and target views represenation. The learned representation fulfills dual role: (1) conditions a lightweight MLP diffusion backbone that reconstructs unobserved regions through diffusion loss [40] and (2) it serves as input to another MLP that directly renders pixel intensities for observed regions, trained with photometric loss. Our method is designed to be purely data-driven, minimizing reliance on predefined inductive biases in its representation and rendering. This "inductive bias- free" design promotes scalability and generalizability, advantages empirically supported by prior work [21] [022 31. Ultimately, UMAMI aims to achieve accurate, training training-efficient and scalable novel view synthesis with photorealistic quality, enjoying both rapid rendering and robust generative completion.
We comprehensively evaluate our model through extensive experiments on RealEstate 10K [103] and DL3DV [T), demonstrating competitive performance across both interpolation and extrapolation settings, and under varying input-view configurations.
Our contributions as follows:
A hybrid framework for NVS: We propose UMAMI, a novel hybrid architecture that combines deterministic and diffusion-basse generation to effectively synthesize both visible and occluded regions from sparse views.
We demonstrate that UMAMI achieves state-of-the-ar performance across multiple bench- marks and input settings, while offering favorable tradc-off's between speed and quality.
2
Related works
Novel view synthesis (NVS) a rapidly advancing field. This section summarizes key prior works most relevant to our approach, with more exhaustive review provided in the Appendix.

---

2.1 Novel view synthesis (NVS)
Novel view synthesis (NVS) has traditionally relied on image-basee based rendering that blends reference views with proxy geometry [1 25 67]. light-field techniques that sample the plenoptic function from dense inputs [3]. and learning-ban
variants that predict blending weights or depth maps with CNNs [12. While multiview-stereo reconstructions enlarge the valid viewing volume 28 7 53 the breakthrough NeRF model introduced a differentiable volumetric representation whose photometric training signal became the new benchmark for NVS [49. Subsequent work has pushed NeRF toward higher fidelity 78 2 faster inference [4 58], and fewer input views [50] 84], or has hybridized it with explicit structures such dense sparse voxels 72
42 20] low-rank decompositions and
hashing 4
g [ or point/gaussian primitives 92 100 19 34 Despite significant progress in
rendering quality, these per-scene optimization methods often suffer from slow training times and limited generalization to novel scenes.
2.2 Deterministic NVS
To address the limitations of per-scene optimization, deterministic NVS methods train single network across multiple scenes for fast, feed-forward inference. Some approaches, such as PixelSplat [6]. MVSplat [T]. and NoPoSplat [96), learn to predict 3D Gaussian parameters directly. While efficient,
their reliance on specific 3D representations (e.g., NeRF 49), 3D Gaussians (44 can hinder scalability. Alternativeyy, data-driven methods like LVSM 31 and SRT [63] leverage Transformer- only backhones to map input images and target poses to novel views, demonstrating the potential to synthesize views without explicit 3D representations given sufficient data and careful network design. Although scalable and fast, the deterministic nature of these methods typically restricts view generation to regions observed in the input context. Our method, in contrast, aims to synthesize novel views even when parts of the scene are occluded or outside the context views.
2.3 Generative NVS
In addition to deterministic approaches, generative approaches have adapted powerful image and video diffusion models (DMs) B 69) for NVS [102 80 87 (36,, leveraging their strong generative priors. Early diffusion- based NVS models 666
48 often utilized image DMs conditioned on input images. Contemporary methods increasingly adopt video DMs [2] [02], conditioned camera poses, to achieve finer-grained control and generate high-quality views of unseen regions. However, training these large-scale generative models demands substantial data and computational resources, potentially impacting rendering performance.
3 Background
3.1 Novel View Synthesis
Deterministic approaches focus on learning a mapping fo π T that directly generates the target image Iss Here, 1st and _ represent context and target images, while л and ' denote their respective camera poses. This mapping fo may be realized through pure neural networks [31 71] or by integrating 3D inductive biases 198 6 [1 96]. Although generally efficient, fundamental limitation of deterministic methods is the inability to generate unseen region due to the deterministic nature.
Generative approaches learn sample 1st from a learned conditional distribution Iet (I -. , This distribution is often modeled using powerful generative frameworks such diffusion models (02 The advantage of such generative techniques lies in their ability to convincingly hallucinate regions absent in the input views. Nevertheless, this capability comes at significant computational cost for both training and inference, thereby posing challenges their widespread practical use in NVS applications.
3.2 Masked Autoregressive Image Generatation
Unlike diffusion models, autoregressive (AR) models 776. 51 22
approach the generation of an < < defining the order) by formulating
ordered token sequence {x² ,..., zn) (with

---

the problem as "next token prediction." This is mathematically expressed by factorizing the joint probability:
p(r',
p(a |a
(I)
where the conditional probability p(>
modeled by a neural network.
Departing from traditional AR methods 222 511. the Masked Autoregressive (MAR) model 40 presents an different approach that unifies random-order AR principles with masked generative modeling through the use of Diffusion Loss. In MAR, an autoregressive network produces a feature vector 2 f(-) Rº. This vector, alongside small MLP cØ(+), used model the conditional distribution p(ar 2) for token x E TRd The model is trained using the denoising criterion:
[zz, x) E,,, [lle c0
(2)
where c E IRd is Gaussian noise and t E IR is the timestep.
Compared to traditional diffusion models (59. 52,,
MAR sample an image by iteratively unmasking tokens using the MLP diffusion conditioned on learned latent from transformer. MAR demonstrates
computational efficiency while showcasing competitive performance with its counterparts.
Building upon MAR's efficient generative capabilities, work introduces novel hybrid method for the NVS task. Specifically, we leverage MAR's generative framework within a hybrid network that incorporates deterministic rendering. Furthermore, we propose a unique sampler specifically designed to efficiently generate novel views, thereby avoiding the iterative full-image generation This approach enables our generative solution to achieve rendering speeds an order of magnitude faster than previous generative NVS methods. We believe this be the first proposal hybrid method that successfully unifies generative model with deterministic head to tackle the NVS challenge.
typical of large backbone architectures 102 21.
4 Methods
In this section, we first outline our problem formulation (Section 4.1) and then details our hybrid model (Section 4.2). Subsequently, we present the training loss (Section 4.3) and conclude by proposing a novel hybrid sampler (Section
4.1 Problem Formulation
Given sparse input images with known camera poses { ")) our goal is to model the conditional distribution p retx, c I!! to synthesize realistic novel view !ts given its camera poses 18t
Data Representation To jointly encode image content and camera pose information, we follow the established convention of concatenating each image with its corresponding Plücker ray embed- dings [54] along the channel dimension. This concatenated representation subsequently processed through MLP-based tokenizer to produce discrete latent tokens. For simplicity, we forego VAE- based approach and directly tokenize each image into x 8 patches. We denote the resulting token sequence from context images and their poses {[1t TT
(c _e_e² ...,, .,,,.,)) and similarly define the target token sequence from {(I's, x(1))} as x = (x x² x MM) Ignoring the facts that
the target camera pose also embedded within x and treat them purely as image tokens, we write the target conditional distribution as p(x|c).
4.2 Hybrid Masked Autoregressive Models for Novel View Synthesis
As discussed in Section 3, deterministic-based NVS approaches [31] 63) model p(x(c) to be deterministic function F inputs: p(x|c) = 6(x F(c)). where ò is the Dirac delta function. While they have shown strong performance in generating high-fidelit outputs for regions covered by input views, they struggle handle unseen regions due their inability model inherent ambiguity. In contrast, generative models based on diffusion [02] 80 can generate plausible completions for unobserved regions, often incur significantly higher computational costs due to iterative sampling over the full image. This trade-off motivates our hybrid design in UMAMI, which is based on the factorization:
p(x c) 8(XD F(c)) p(xs|XD.>
(3)

---

Diffusion head
Input token
Target token
Input Plocker
MLP (+)
Masked token
Transformer
Transformer output token
MLP (Φ)
Target
Unpatchify
Placker
Deterministic head
RGB
Confidence
Figure 2: UMAMI synthesizes target images from their camera poses and context views (each paired with its Plücker pose). During training. randomly mask the target image, replace masked areas with learnable tokens, and concatenate these with the target's Plücker embedding. Input views also tokenized. A Transformer processes both tokenized inputs and the masked target representation to produce a latent 2. This code inputs to two MLP heads: a deterministic head (4)) outputs RGB and confidence, while diffusion head (ő) models the distribution of target tokens conditioned on z. The model is trained end-to-end using weighted loss combination (Section 4.3). At inference, the target image is initialized with learned masked tokens for our proposed hybrid sampling (Section 4.4).
where Xn and Xş are disjoint subsets of such that x xp Uxs. Intuitively, xp corresponds the tokens that are fully determined by the input context с (e.g., seen or deterministically visible regions) and can be computed directly as a function F(c). In contrast, x represents tokens uncertain or unseen regions, which require sampling from a complex conditional distribution pXxs|XD, c).
Model Architecture The architecture of UMAMI is illustrated in the Figure 2 UMAMI is a masked autoregressive model designed to support both efficient deterministic prediction and flexible stochastic generation by progressively unmasking target tokens. At the core of our model is transformer backbone 777 that extract the target latent representation from the partially masked x and context we adopt decoder-only, bi-directional transformer backbone.
c. Following previous works 21 31
To generate the target tokens, UMAMI uses specialized output heads. The deterministic head, parameterized by &p in Figure 2 calculates Fxx) using the extracted latents from the transformer backbone and reconstructs tokens x in single forward pass, leveraging regions of high confidence inferred from the context. In contrast, the diffusion head, parameterized by e in Figure 2 models the conditional distribution over X and performs iterative denoising to progressively generate plausible content in uncertain or unseen regions. Following MAR [40). both heads are small MLP networks with SiLU activation [18) that operate on each token latent individually, and the diffusion head takes an additional time embedding as input. This dual-head design enables UMAMI to adaptively combine the speed and accuracy of deterministic prediction with the generative capacity of diffusion models, effectively addressing both observed and novel view synthesis scenarios.
In practice, the separation between deterministic and uncertain regions is not known a priori. To address this, we introduce a pixel-wise confidence prediction that estimates an uncertainty score for each pixel. The confidence score of a patch is defined as the minimum confidence among its pixels. Given a threshold т, patches with confidence above that threshold are assigned to xp, while the remaining are treated as xs and handled via the stochastic generation process.
4.3 Training Losses
39 At each training We train UMAMI using masked autoregressive generative framework 44
step, a binary mask m is sampled uniformly to mask subset of the target image patches. Crucially, only the target image is masked (e.g. each selected patch replaced with learnable token) while the corresponding target camera pose embeddings are preserved. The model is then optimized to reconstruct the masked patches conditioned on the context and unmasked target information, using a combination of deterministic and diffusion losses:
5

---

Figure 3: Hybrid Masked Autore- gressive Sampler. Top: Conventional Masked Generative samplers 40 5 39 predict multiple tokens simultaneously using random ordering. Bottom (Ours); deterministic first pass for high confi- dence tokens, followed by simultaneous random-oree sampling for the remain- ing tokens, significantly boosts render- ing times for the NVS task.
Generative Samplera
Unmasked sakar Nedipredction sot
fiom dildion
Ours
Deeerikaiosly ctrainee tokana
from oillyiinn hoad
Nadt predicttor
The deterministic head produces token predictions & for masked patches, which yields reconstruction of the target images i To supervise this process, we employ standard photometric losses for novel view synthesis, defined as:
Deterministic reconstruction loss
Lrender MSE(I 1 + X Perceptual 1!*),
where Xp is the weight for balancing the perceptual loss
33 Importantly, this loss is computed over the full image rather than individual patches to encourage spatial consistency.
we augment the deterministic head to output pixel-level confidence map Sp, where each value indicates the model's confidence in its prediction. The confidence-aware loss balances the regression error with regularization term that penalizes overconfidence:
Confidence loss As discussed in Section 4.2
1 1 X log sp):
(5)
where X is hyper-ppare parameter controlling the regularization term [81]. The loss averaged over all masked parts. We compute a patch-wise confidence map by taking the minimum value of sp within each patch.
Diffusion loss To model the conditional distribution over uncertain tokens, we incorporate diffusion model following the formulation DDPM [26]. Specifically, we use linear noise schedule to corrupt the ground truth tokens and train the model to reverse this process. Given noisy token timestep and corresponding latent extracted from the transformer backbone, the diffusion head predicts the added noise é. The denoising objective is defined over all masked tokens:
Caif = E.,, [lle E(zelt, x)
(6)
where 10, I) is Gaussian noise, and t is sampled uniformly over the diffusion steps.
To better allocate learning effort, we emphasize uncertain regions during training by predicting a token-wise weighting scheme derived from the patch-level confidence map s. Specifically, we define the weight for each token as max(s, (s, Aa)/Ad, where da is a hyperparameter. This weighting encourages the model to focus more heavily on regions with lower confidence, enhancing generative quality areas with higher ambiguity.
Total loss
Our model is trained end-to-end using a weighted sum of the aforementioned losses.
4.4
Hybrid Masked Autoregressive Sampling
The overall sampling process illustrated in Figure 3
Given a masked target image and its corresponding camera pose, UMAMI performs hybrid inference by first identifying and reconstructing the set of deterministic tokens xp, and then generating the remaining uncertain tokens x through diffusion-bast process.
In the first stage, the model performs single forward pass through the deterministic head predict Xp, guided by the confidence map predicted from the transformer backbone. Tokens with confidence scores greater then a predefined threshold reconstructed deterministically.
In the second stage, the remaining masked tokens xs, are iteratively sampled using the diffusion head. We employ cosine unmasking schedule following the approach of [40], which gradually reveals

---

Table 1: Quantitative results RealEstate 10K across different validation splits. Best results highlighted in red, second- best in orange.
Re10K- 2iie2veeeetta
Re10K (el0K-2view-iiii 2view-inttrr
RetOK- 3view
Method
Purams (M)
PSNR 1 LPIPS 4 SSIM f
PSNR
LPIPS L SSIM 1
PSNR 1
LPIPS
SSIM
Deterministic
MVSplat 11]
12.0
23.30
0.160
0.830
26.39
0.128
0.869
25.64
0,142
0.857
DepthSplat 91
24.57
0.158
0.848
27.44
0.119
0.887
22.54
0.177
0.824
LVSM B1
171
28.51
0.117
0.882
29.67
0.098
0.906
30.04
0.090
0.936
Diffusion-based
ViewCrafter 99
21.42
0.203
0.710
0.164
0.830
SEVA T02
1300
24.00
0.100
0.797
25.66
0.061
0.847
27.57
0.073
0.892
UMAMI
271
28.95
0.107
0.897
28.85
0.101
0.899
31,06
0.084
0.946
more tokens at later iterations using T's unmasking steps. As xs] S |x]. especially in scenarios where target views significantly overlap with context views, we introduce a dynamic strategy to adjust the number of unmasking steps accordingly. Specifically, given maximum step budget T Taax for unmasking the entire token set x, the number of steps allocated for x is computed using a simple linear scaling rule: Tg |xs| / x maax] The hyperparameter Tmax is fixed across experiments, and T's is automatically determined by the number of tokens to be unmasked. Despite its simplicity, we find this strategy to be effective in practice and well-suited for varying levels of token uncertainty.
5 Experiments
Datasets We evaluate UMAMI on two scene-level novel view synthesis benchmarks: RealEstate 10K (CC-BY-4.0) (103) and DL3DV (CC-BY-4.0) [I]. RealEstate 10K consists of 80K indoor and outdoor video clips sourced from YouTube, while DL3DV features over 10K videos captured across a wide range of real-world locations. We train separate models for each dataset at a resolution of 256 x 256. For the RealEstate dataset, we adopt the evaluation split from PixelSplat [6]. which primarily features target views located between the 2 input views, making it suitable for assessing interpolation performance. We refer this split ability, we construct a complementary split by swapping the roles of the context and target views, which we denote as RelOK-2 View-Extra. Additionally, incorporate the 3-view validation split introduced in Reconfusion [89]. labeled as Re1OK-3View respectively. For the DL3DV dataset, we follow the validation setup from Zhou al. [102], using the 1-view, 3-view, and 6-view input configurations, which we name DL3DV- View, DL3DV-3View, and DL3DV-6View, respectively.
To evaluate extrapolation
Experiment Details Each model is trained for 100K iterations with a batch size of 32, using the Adam W optimizer 47 with a learning rate of 10-4 and a cosine decay schedule. Training takes approximately two days on 8x NVIDIA A100 GPUs. During training, we randomly sample 1 or 2 context views and select between 1 and 3 target views per training example. In the main experiments, we report results for predicting a single target view, while results for generating multiple target views accelerate convergence, we initialize our model using the pretrained
are included in the Appendix. transformer backbone from LVSM (31. We use a fixed threshold value of r 0.95 and a maximum sampling steps Tmax = 32 across our experiments, as we found those values balance well between generation quality and speed. For diffusion sampling, use 50 DDPM steps with CFG value of 2.0 and sampling temperature of 0.9. An details on other hyperparameters of our model are in the Appendix.
Baselines To the best of our knowledge, are the first method that perform a hybrid render on deterministic and generative method, thus we have no direct competitors. Therefore, we compare method different deterministic and generative baselines. For deterministic methods, compare UMAMI to MVSplat 11], LVSM 31.. For generative approaches, we compare UMAMI with
ViewCrafter 999 and SEVA [02
5.1 Experiment results
Tables and and present a comprehensive comparison of our method UMAMI against both deterministic and diffusion-based baselines on the RealEstate 10K and DL3DV datasets, respectively. On RealEsttae 10K, UMAMI consistently achieves top-tier performance across all splits. split, it matches LVSM closely, trailing by only 0.82 PSNR (28.85 vs.
Quantitative results
On the Re
7

---

Table 2: Quantitative results on DL3DV across 1-view, 3-view, and 6-view settings. Best results highlighted in red, second-best in orange.
DLaDV-IvieW
DI3DV-3view
DL3DV-6view
Method
PSNR
LPIPS
SSIM
PSNR
LPIPS J
SSIM 1
PSNR
LPIPS 1
SSIM
Deterministic
DepthSplat 91
9.63
0.580
0.349
12.52
0.405
0.452
15.72
0.481
0.513
Diffusion- based
ViewCrafter
8.97
0.616
0.323
11.50
0.576
0.400
0.558
0.469
SEVA 102
13.01
0.484
0.360
15.95
0.316
0.480
17.98
0.232
0.546
UMAMI
12.81
0.574
0.269
16.37
0.386
0.444
17.33
0.326
0.476
Input
Ours
LVSM
MVSplitt
SEVA
Figure 4: Qualitative results Re10K Evaluation of UMAMI the challenging 1110K--VVew--Etee extrapolation set, comparing it with LVSM (31), MVSplat [IT) and SEVA [102). UMAMI not only renders sharp details in observed regions but also generates plausible content for unseen areas. More results can be viewed in the Appendix.
29.67). On the Re10K-2vine 22ieww-extr split, obtains the highest PSNR (28.95) and SSIM (0.897), outperforming the second-best LVSM by 0.44 PSNR and 0.015 SSIM. while maintaining second- best LPIPS of O. 107; showing its capabilities of doing extrapolation. On the Re10K-3view setting, UMAMI surpasses all baselines with the best PSNR (31.06) and SSIM (0.946), while having the second best LPIPS of 0.084.
On the DL3DV benchmark, UMAMI delivers competitive performance across all input configurations. It achieves the second-best PSNR in both the 1-view (12.81) and 6-view (17.33) settings, and ranks second in LPIPS across all three input setups. Notably, in the 3-view setting, UMAMI achieves the highest PSNR among all methods. While SSIM lags behind DepthSplat, UMAMI outperforms ViewCrafter in both the 3-view and 6-view scenarios. These results underscore the robustness and adaptability our hybrid framework. demonstrating its effectiveness in handling diverse view configurations and maintaining a strong balance between reconstruction fidelity and perceptual quality,
we evaluate UMAMI on Rel0k test set with varying camera overlaps based on ratio of image overlap: small (0.05%-0.3%) 0..%%, medium (0.3% 0.55%), and large (0.55% 0.8%), determined using dense feature matching method, RoMA [17]. The results, shown in the Table 3 demonstrate that UMAMI outperforms NoPoSplat on all metrics and validation sets. As with NoPoSplat, we also observed that performance improves as the overlap ratio increases, which suggests less scene occlusion. Notably, our method remains robust even with small camera overlap, outperforming NoPoSplat across all metrics. This confirms UMAMI's s effectiveness across various datasets and overlap ratios.
Varying overlap ratios Following NoPoSplat 196]

---

Table 3: Quantitative comparison across overlaping ratios (Small, Medium, Large). Best results highlighted in red, best in orange.
Small SSIM 1
Medium SSIM
Large SSIM
Method
PSNR 1
LPIPS
PSNR
LPIPS 4
PSNR
I.PIPS
NoPoSplat
22.514
0.784
0.210
24.899
0.839
0,160
27.411
0.883
0.119
Ours
23,558
0.806
0.176
26.713
0.862
0.130
29.782
0.907
0.094
Qualitative results presented in Figure highlight the performance of our method against several methods. Firstly, MVSplat [1]], as deterministic method employing 3D Gaussians, is unable to generate content beyond the provided context images, resulting in black rendered areas in unobserved regions. Similarly, while LVSM (3 avoids such black areas by forgoing 3D inductive biases, its -on-generattve nature results in blurry predictions for unseen pixels. Our method overcomes these limitations of deterministic approaches, demonstrating the ability both accurately render observed regions and plausibly generate content in unobserved areas. Finally, in comparison to SEVA [102], considerably larger model with 1.3B parameters (versus our 271M parameters), our approach achieves comparable performance on visible regions and produces results with fewer artifacts.
5.2
Ablation study and Analysis
Table 4: Ablation study threshold Higher Table 5: Ablation study on the number of context improves image quality but increases transformer calls and runtime.
views. Increasing the number of context views (N₂) improves image synthesis quality by pro- viding more deterministic tokens and reducing the number of average transformer calls, due to higher confidence in a larger portion of the scene.
7
Time (s)
# Trans. Calls
LPIPS
0
0.02
1.00
0.398
0.5
2.71
12.31
0.394
No
Deter Tokens Trans. Calls LPIPS
0.8
4.30
18.99
0.389
0.9
4.62
20.30
0.387
1
119.24
29.55
0.574
0.95
4.77
21.14
0.386
3
394.68
21.14
0.386
1
7.63
33.00
0.377
6
527.25
16.98
0.326
Effect on threshold r during sampling We control the balance between UMAMI's deterministic and diffusion heads during sampling using threshold in Table 4 Setting 7 = 0 engages only the deterministic head, enabling UMAMI predict all tokens 0.02s with one transformer call. Conversely, setting = 1 relies exclusively on the diffusion head for sampling target views, which yields the optimal LPIPS score in our experiments. We observe that incrementally increasing from 0 to 1 enhances LPIPS performance, though at the cost of increased runtime due to more frequent transformer and diffusion sampling operations. Thus, by adjusting T, our dual-head model offers a flexible mechanism to trade off inference speed against generative quality.
Effect on the number of context views We conduct ablation study the effect of varying the number context views, as shown i5 Results indicate that increasing the number of input views leads significant boost in mean LPIPS. Furthermore, with more context available, the model exhibits higher confidence, resulting in a greater proportion of tokens being handled deterministically. This, in turn, reduces the number of transformer calls required during sampling, leading improved computational efficiency.
Run time analysis Unlike conventional methods with fixed rendering times, UMAMI offers opera- tional flexibility by adaptively engaging its deterministic and generative heads. For instance, UMAMI renders an image in approximately 5s when = 0.95 (details in Table This is considerably faster than generative counterparts like SEVA 102). which takes about minute to sample an image. While purely deterministic methods [11 achieve sub-second rendering, they sacrifice the ability to generate content for unobserved target regions. UMAMI thus provides a compelling trade-off: it achieves strong generative capabilities for modest increase in runtime compared to deterministic approaches, while remaining significantly more efficient than other generative models.
Backbone initialization mentioned in Section 5 we initialize our model using pretrained weight from LVSM (31]. To further demonstrate the strength of our method, we train the model from scratch

---

Table 6: Ablation results on DL3DV across 1-view, 3-view, and 6-view settings comparing pretrained and random initialization. Best results are highlighted in red, second-best in orange.
DL3DV-I LPIPS
DL3DV-3view
DL3DV-6view
Method
PSNR
SSIM
PSNR
LPIPS
SSIM
PSNR
LPIPS
SSIM
Pretrained
12.81
0.574
0.269
16.37
0.386
0.444
17.33
0.326
0.476
Random Init.
11.80
0.543
0.256
14.46
0.374
0.370
15.43
0.318
0.404
on DL3DV with randomly initialized weights, without relying LVSM pretrained on Re10K. This variant shares the same settings as the pretrained version, except for a larger batch size (512 vs. 32) to stabilize training. The results, presented in the Table 6 reveal that even without pretraining, performs
model slightly lower PSNR and SSIM but improved LPIPS. Remarkably, even with random initialization, our method consistently outperforms ViewCrafter and Sppth, underscoring that our performance stems from the strength of our hybrid deternminisiicgeneeaaiy design, rather than dependence LVSM initialization and Re 10K pretraining.
6 Discussion and Conclusion
While our method achieves competitive performance, it also has several limitations. First, because we operate directly in pixel space, each image is represented by large number of tokens (e.g., 32 x 32 1024), which increases memory and computational requirements. A promising direction for future work to adapt our framework to operate in the latent space of pretrained VAE 335] which would reduce the token count while preserving semantic content. Second, unlike recent diffusion- based NVS approaches [21 102 [9], our model does not make use of any pretrained text-to-image priors. Integrating such powerful generative priors [40] 16] could enhance the model's ability to hallucinate plausible unseen regions and improve visual fidelity in sparse- view settings. We also leave for future exploration techniques to further accelerate sampling and incorporate temporal consistency for vidco-based novel view synthesis. On the social impact side, this work could enable deepfake information, so users will be required to follow usage guidelines.
Limitations and Future Work
Conclusion We have presented UMAMI, hybrid framework for novel view synthesis that unifies deterministic and generative modeling to handle both seen and unseen regions effectively. By leveraging confidence-aware mechanism, our model adaptively allocates computation between fast deterministic head and diffusion based head, achieving a strong balance between efficiency and image quality. Extensive experiments on RealEstate 10K and DL3DV demonstrate that UMAMI is competitive with both deterministic and diffusion-oni baselines across various input configurations. Our results suggest promising direction for designing more efficient approaches to novel view synthesis.
Acknowledgements
Stephan Mandt acknowledges funding from the National Science Foundation (NSF) through an NSF CAREER Award IIS-2047488, IIS-2007719, the NSF LEAP Center, and the Hasso Plattner Research Center at UCI. Xiaohui Xie acknowledges funding from NIH IP0ICA288662-01A and Kay Family Foundation. Parts of this research were supported by the Intelligence Advanced Research Projects Activity (IARPA) via the Department of Interior/ Interior Business Center (DOI/IBC) contract number 40D0423C0075. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.
10

---

References
[1] J. T. Barron, B. Mildenhall, M. Tancik, Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5855-5866 2021.
[2] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 19697-19705, 2023.
[3] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311. 151277, 2023.
[4] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, J. Guibas, J. Tremblay, Khamis, et al. Efficient geometry-awarr 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133 133, 2022.
[5] H. Chang. H. Zhang, L. Jiang. Liu, and W. T. Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11315-11325, 2022.
[6] D. Charatan, S. L. Li, A. Tagliasacchi, and V. Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 19457-19467, 2024.
[7] G. Chaurasia, S. Duchene, O. Sorkine- Hornung, and G. Drettakis. Depth synthesis and local warps for plausible image-based navigation. ACM transactions on graphics (TOG), 1-1 2013.
[8] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In European conference computer vision, pages 333-350. Springer, 2022.
[9] A. Chen, Z. Xu, X. Wei, S. Tang, H. Su, and A. Geiger. Factor fields: A unified framework for neural fields and beyond. arXiv preprint arXiv:2302.011226 2023.
[10] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang. Yu, and H. Su. Mysnerf: Fast generaliz- able radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14124-14133, 2021.
[11] Y. Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T.-J. Cham, J.d Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pages 370-386. Springer, 2024.
[12] I. Choi, O. Gallo, A. Troccoli, M. H. Kim, and J. Kautz. Extreme view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7781- 7790, 2019.
[13] Davis, M. Levoy, and Durand. Unstructured light fields. In Computer Graphics Forum, volume 31, pages 305-314. Wiley Online Library, 2012.
[14] P. E. Debevec, C. J. Taylor, and Malik. Modeling and rendering architecture from pho- tographs: A hybrid geometry-and and image-based approach. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 465-474. 2023.
[15] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13142 13153, 2023.
[16] Deng, Pan, Diao, Z. Luo, Y. Cui, H. Lu, Shan, Y. Qi, and X. Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412. 2024.
11

---

[17] J. Edstedt, Q. Sun, G. Bökman, M. Wadenbäck, and M. Felsberg. Roma: Robust dense feature matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19790-19800, 2024.
[18] S. Elfwwng, E. Uchibe, and K. Doya. bigooidweighted linear units for neural network function approximation reinforcement learning. arxiv e-prints, art. arXiv preprint 1oXiv:1702.03115 2017.
[19] W. Feng, J. Li, H. Cai, X. Luo, and J. Zhang. Neural points:
Point cloud representation with neural fields for arbitrary upsampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18633-18642, 2022.
[20] S. Fridovich-Keil A. Yu, M. Tancik, Q Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5501-5510, 2022.
[21] R. Gao, A. Holynski, P. Henzler, A. Brussee, R. Martin-Brualla, P. Srinivasan, J. Barron, and B. Poole. Cat3d: Create anything 3d with multi-view diffusion models. Advances in Neural Information Processing Systems, 2024.
[22] K. Gregor, I. Danihelka, A. Mnih, C. Blundell, and D. Wierstra. Deep autoregressive networks. International Conference on Machine Learning, pages 1242-1250, 2014.
[23] He, Xu, Guo, G. Wetzstein, B. Dai, H. Li, and Yang. Cameractrl: Enabling camera control for text-to-video generation, 2024.
[24] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec. Baking neural radiance fields for real-time view synthesis. Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 5875-5884, 2021.
[25] B. Heigl, R. Koch, M. Pollefeys, J. Denzler, and L. Van Gool. Plenoptic modeling and rendering from image sequences taken by a hand-held camera. In Mustererkennung 1999: s1 DAGM Symposium, pages 94-101, 1999.
[26] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, 2020.
[27] Y. Hong, K. Zhang, J. Gu, S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, and H. Tan. LRM: Large reconstruction model for single image to 3d, 2024.
[28] M. Jancosek and T. Pajdla. Multi-view reconstruction preserving In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3 3121-3128, 128, 2011.
surfaces.
[29] H. Jiang, Z. Jiang. K. Grauman, and Y. Zhu. Few-view object reconstruction with unknown categories and camera poses. In Proceedings of the International Conference on Vision (3DV), pages 31-41, 2024.
[30] H. Jiang, Z. Jiang, Y. Zhao, and Q Huang. Leap: Liberate sparse-view 3d modeling from camera poses, 2023.
[31] H. Jin, H. Jiang, H. Tan, K. Zhang, S. Bi, T. Zhang, F. Luan, N. Snavely, and Z. Xu. LVSM: A large view synthesis model with minimal 3D inductive bias, 2024.
[32] M. M. Johari, Y. Lepoittevin, and F. Fleuret. Geonerf: Generalizing nerf with geometry priors, 2022.
[33] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Proceedings of the European Conference Computer Vision (ECCV), pages 694-711, 2016.
[34] B. Kerbl, G. Kopanas, T. Leimküühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[35] D. P. Kingma, M. Welling, et al. Auto-encoding variational bayes, 2013.
12

---

[36] X. Kong. S. Liu, X. Lyu, M. Taher, X. Qi, and A. J. Davison. Eschemet: A generative model for scalable view synthesis, 2024.
[37] J. Kulhánek, E. Derner, T. Sattler, and R. Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In European Conference on Computer Vision (ECCV), 2022.
[38] J. Li, H. Tan, K. Zhang, Z. Xu, F. Luan, Y. Xu, Y. Hong. K. Sunkavalli, G. Shakhnarovich, and S. Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model, 2023.
[39] T. Li, H. Chang, S. Mishra, H. Zhang, D. Katabi, and D. Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2142-2152, 2023.
[40] T. Li, Y. Tian, H. Li, M. Deng, and K. He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems,
2024.
[41] L. Ling, Y. Sheng, Z. Tu, W. Zhao, C.
Xin, K. Wan, L. Yu, Q. Guo, Z. Yu, Y. Lu, et al.
D13dv-108 10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22160-22169, 2024.
[42] L. Liu, J. Gu, K. Z. Lin, T. Chua, and C. Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurlPS), volume 33, pages 15651-15663, 2020.
[43] Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023.
[44] Y. Liu, C. Lin, Z. Zeng, X. Long, L. Liu, T. Komura, and W. Wang. Syncdreamer: Generating multiview-consisten images from single-view image, 2023.
[45] Y. Liu, S. Peng. L. Liu, Q. Wang, P. Wang, C. Theobalt, X. Zhou, and W. Wang. Neural rays for occlusion-awand image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2824-7833, 2022.
[46] Long, Guo, C. Lin, Liu, Z. Dou, L. Liu, Y. Ma, S. Zhang. M. Habermann, Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion, 2023.
[47] I. Loshchilov and F. Hutter. Decoupled weight decay regularization, arXiv preprint arXiv:1711.01101, 2017.
[48] A. Mercier, R. Nakhli, M. Reddy, R. Yasarla, H. Cai, F. Porikli, and G. Berger. HexaGen3ID: StableDiiffusion is just one step away from fast and diverse text-to-3D generation. arXiv 2024.
preprint aXX:::20002:
[49] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In European Conference Computer Vision, pages 405-421, 2020.
[50] M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5480-5490, 2022.
[51] A. D. Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756, 2016.
[52] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205. 2023.
[53] Penner and I.. Zhang. Soft 3d reconstruction for view synthesis. ACM Transactions on Graphics, 36(6):1-11, 2017.
13

---

[54] J. Plücker. On a new geometry of space. Philosophical Transactions of the Royal Society of London, pages 725-791, 1865.
[55] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3D using 2D diffusion. arXiv, 2022.
[56] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents, 2022.
[57] Reiser, Peng. Y. Liao, and Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14335-14345, 2021.
[58] C. Reiser, R. Szeliski, D. Verbin, P. Srinivasan, B. Mildenhall, A. Geiger, J. Barron, and P. Hedman. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics, 42 2023.
[59] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[60] R. Rombach, Esser, and B. Ommer. Geometry-free view synthesis: Transformers and no 3d priors. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 14356- 14366, 2021.
[61] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In Proceedings of SIGGRAPH, 2022.
[62] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(4):4713-4725 2022.
[63] M. S. Sajjadi, H. Meyer, E. Pot, U. Bergmann, K. Greff, N. Radwan, S. Vora, M. Lučić, D. Duckworth, A. Dosovitskiy, et al. Scene representation transformer: Geometry- novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6229-622222222 6238, 2022.
[64] K. Sargent, Z. Li, T. Shah, C. Herrmann, H.-X. Yu, Y. Zhang, E. R. Chan, D. Lagun, L. Fei-Fei, D. Sun, et al. ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310.
2023.
[65] Shi, H. Chen, Z. Zhang, M. Liu, C. Xu, Wei, L. Chen, C. Zeng, and H. Su. A single image to consistent multi-view diffusion base model, 2023.
[66] Y. Shi, P. Wang, J. Ye, L. Mai, K. Li, and X. Yang. MVDream: Multi-view diffusion for 3d generation, 2023.
[67] S. Sinha, D. Steedly, and R. Szeliski. Piecewise planar stereo for image-based rendering. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1881-1888, 2009.
[68] Sitzmann, S. Rezchikov, B. Freeman, J. Tenenbaum, and F. Durand. Light field net- works: Neural scene representations with single-evaluation rendering. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 19313-19325, 2021.
[69] Stability AI. Stable diffusion 3 (technical preview). Press release, 2024.
[70] M. Suhail, C. Esteves, L. Sigal, and A. Makadia. Generalizable patch based neural rendering. In European Conference on Computer Vision (ECCV), pages 156-174, 2022.
[71] M. Suhail, C. Esteves, L. Sigal, and A. Makadia. Light field neural rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8279-8279, 2022.
14

---

[72] C. Sun, M. Sun, and H. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2022.
[73] S. Szymanowicz. E. Insafutdinov, Zheng, D. Campbell, Henriques, C. Rupprecht, and A. Vedaldi. Flash3d: Feed-forward generalisable 3d scene reconstruction from single image, 2024.
[74] J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1-18. Springer, 2024.
[75] J. Tung, G. Chou, R. Cai, G. Yang, K. Zhang, G. Wetzstein, B. Hariharan, and N. Snavely. Megascenes: Scene-level view synthesis at scale. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
[76] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[77] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[78] D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P. Srinivasan. Ref- NeRF: Structured view-dependent appearance for neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[79] V. Voleti, C. Yao, M. Boss, A. Letts, D. Pankratz, D. Tochilkin, C. Laforte, R. Rombach, and V. Jampani. SV3D: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In Proceedings of the European Conference on Computer Vision (ECCV), pages 439-457, 2025.
[80] Voleti, C.-H. Yao, M. Boss, A. Letts, D. Pankratz, D. Tochilkin, C. Laforte, R. Rombach, and V. Jampani. SV3D: Novel multi-view synthesis and 3D generation from single image using latent video diffusion. In European Conference on Computer Vision, 2024.
[81] S. Wan, T.-Y. Wu, W. H. Wong, and C.-Y. Lee. Confnet: predict with confidence. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2921-2925. IEEE, 2018.
[82] P. Wang, H. Tan, S. Bi, Xu, F. Luan, K. Sunkavalli, W. Wang, Z. Xu, and K. Zhang. PF-LRM: Pose-free large reconstruction model for joint pose and shape prediction, 2023.
[83] Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4690-4699 2021.
[84] Z. Wang, Wu, W. Xie, M. Chen, and V. A. Prisacariu. Nerf--; Neural radiance fields without known camera parameters, 2021.
[85] Z. Wang, Z. Yuan, X. Wang, Y. Li, T. Chen, M. Xia, P. Luo, and Y. Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[86] D. Watson, W. Chan, R. Martin- J. Ho, A. Tagliasacchi, and M. Norouzi. Novel view synthesis with diffusion models, 2022.
[87] D. Watson, S. Saxena, L. Li, A. Tagliasacchi, and D. J. Fleet. Controlling space and time with diffusion models. In The Thirteenth International Conference on Learning Representations, 2024.
15

---

[88] X. Wei, K. Zhang. S. Bi, H. Tan, F. Luan, V. Deschaintre, K. Sunkavalli, H. Su, and Z. Xu. Meshlrm: Large reconstruction model for high-quality mesh, 2024.
[89] R. Wu, B. Mildenhall, P. Henzler, K. Park, R. Gao, D. Watson, P. P. Srinivasan, D. Verbin, T. Barron, B. Poole, et al. Reconfusion: 3D reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21551-21561, 2024.
[90] D. Xie, S. Bi, Z. Shu, K. Zhang, Z. Xu, Y. Zhou, S. Pirk, A. Kaufman, X. Sun, and H. Tan. Lrm-zero: Training large reconstruction models with synthesized data, 2024.
[91] H. Xu, S. Peng, F. Wang, H. Blum, D. Barath, A. Geiger, and M. Pollefeys. Depthsplat: Connecting gaussian splatting and depth. arXiv preprint arXiv:2410.1386. 2024.
[92] Q. Xu, Xu, J. Philip, S. Bi, Z. Shu, Sunkavalli, and Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition ( (CVPR), pages 5438 5448, 2022.
[93] Y. Xu, H. Tan, F. Luan, S. Bi, P. Wang. J. Li, Z. Shi, K. Sunkavalli, G. Wetzstein, Z. Xu, and K. Zhang. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model, 2023.
[94] Y. Yan, Z. Xu, H. Lin, H. Jin, H. Guo, Y. Wang, K. Zhan, X. Lang, H. Bao, X. Zhou, and
S. Peng. Streetcrafter: Street view synthesis with controllable video diffusion models, 2024.
[95] J. Yang. Z. Cheng, Y. Duan, P. Ji, and H. Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion, 2023.
[96] Ye, S. Liu, Xu, X. Li, M. Pollefeys, M.-H. Yang, and Peng. No pose, no prob- lem: Surprisingly simple 3d gaussian splats from sparse unposed images. arXiv preprint arXiv:2410.24207 2024.
[97] J. Ye, P. Wang, K. Li, Y. Shi, and H. Wang. Consiteent--tt-3 Consistent image to 3d view synthesis via geometry- aware diffusion models, 2023.
[98] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[99] W. Yu, J. Xing, L. Yuan, W. Hu, X. Li, Z. Huang. X. Gao, T. Wong, Y. Shan, and Y. Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis, 2024.
[100] Q. Zhang. S. Baek, Rusinkiewicz, and Heide. Differentiable point- based radiance fields for efficient view synthesis. In SIGGRAPH Asia Conference Papers, pages 1-12, 2022.
[101] C. Zheng and A. Vedaldi. Free3d: Consistent novel view synthesis without 3d representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9720-9731 9730099971111 2024.
[102] J. J. Zhou, H. Gao, V. Voleti, A. Vasishta, C.-H. Yao, M. Boss, P. Torr, C. Rupprecht, and V. Jampani. Stable virtual camera: Generative view synthesis with diffusion models. arXiv preprint wxXiv:2503.1448) 2025.
[103] T. Zhou, R. Tucker, J. Flynn. G. Fyffe, and N. Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
16

---

Supplement to "UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis"
A
Failure cases
Input
Target
Target
Target
Target
Pred
Pred
Pred
Pred
Input
Target HR
Target
Pred
Pred
Pred
Figure 5: Failure cases. Our method may produce noticeable artifacts when target camera poses too distant from the input view. Increasing the scale of training data and model parameters could improve the robustness of UMAMI.
B
Related works
B.1 Feed-forward deterministic NVS methods
Early generalizable methods for Novel View Synthesis (NVS) demonstrated the potential of neural networks, trained across various scenes, to enable fast inference of novel views underlying 3D rep- resentations a feed-forward manner. Prominent examples include PixelNeRF (8). MVSNeRF [10], and IBRNet [83] which typically predict volumetric 3D representations by incorporating 3D-specific priors like epipolar geometry or plane sweep cost volumes. Subsequent research has extended these capabilities, improving performance particularly under challenging conditions such sparse input 30 and adapting these techniques for emerging representations like 3D Gaussian
views 445
32
29
Splatting (3DGS) 6
73
74)
Recently, 3D Large Reconstruction Models (LRMs) have emerged 27
38 82
88 90 leveraging
the power of scalable transformer architectures 777 trained on extensive datasets to learn generic 3D priors. While these methods successfully avoid explicit architectural reliance on epipolar projection or cost volumes, they still typically depend pre-defined 3D representations such as tri-plane NeRFs, meshes, or 3DGS, along with their corresponding rendering equations. This reliance can limit their flexibility and overall potential.
An alternative line of work attempts to directly learn a geometry-fres rendering function 70 63 68 60 37). However, these approaches often face limitations in model capacity and scalability, which can hinder their ability to capture high-frequency details. Notably, Scene Representation Transformers (SRT) (33 aimed to avoid explicit, handcrafted 3D representations by learning latent
17

---

scene representation via a transformer, an objective shared by our encoder-decodee architecture. Despite this similarity, certain design choices SRT, such as its CNN-based token extractor and the use of cross-attention in the decoder, have been shown to lead to less effective performance. leveraging
To address the issue, LVSM [1 proposes method that fully transformer-basee bidirectional self-attention for enhanced representational power. Furthermore, they introduce a novel and more scalable decoder-only architecture that directly learns the NVS function with minimal 3D intermediate latent representation.
inductive bias and without relying
Our proposed method adopts the versatile and scalable decoder-only transformer backbone from LVSM, which has demonstrated its efficacy in NVS tasks by leveraging data-driven approach with minimal handcrafted 3D inductive bias. However, a crucial distinction lies in the nature of our approach: unlike the deterministic LVSM, our method is generative. We aim to address the inherent limitations of deterministic methods by harnessing the generative capabilities of masked autoregressive diffusion models in an efficient manner.
B.2 Generative-based NVS methods
The pursuit of generative-based (NVS) has recently seen significant advancements through the integration of diffusion models, drawing inspiration from successes in broader NVS 68 63 and
generative image-to-imane tasks 61 56 62]
An early exploration in this domain was 3DIM 86), which trained image-to-image diffusion models for object-level multi-view rendering without explicit 3D representations. However, by training from scratch on limited 3D data, 3DiM's applicability was restricted to category-specific scenarios and lacked zero-shot generalization capabilities. Building on this, Zero-1-to-3 [ adopted similar geometry -reee pipeline but significantly improved generalization and output quality by fine-tuning a pretrained 2D diffusion model on a larger 3D object dataset [15]. Despite these improvements, a key challenge for Zero-I-to-3 and other early image-based diffusion models for NVS (c.g., for distant viewpoints 665 was multi-view inconsistency, as they typically generated each target view independently and probabilistically. leading to jitter or inconsistencies when rendering a camera trajectory.
To address this multi-view inconsistency, subsequent research diverged into several directions. One line of work focused on integrating explicit 3D inductive biases-such 3D representations or epipolar attention-into the diffusion denoising process. Examples include SyncDreamer [44), ConsistNet [95]. Consistent-I-ton of increased computation. Another set of approaches, including Instant3D [38), MVDream [66]. and Wonder3D [46), aimed to predict a single grid of multiple, specific views simultaneously. While this improved consistency across those fixed views, it sacrificed the ability for fine-grained camera control. Works like MVDream 666. SyncDreamer 44 and more recently HexGen3D (8). generate multiple fixed views from a conditional image but do not support arbitrary viewpoint selection. To achieve consistent 3D object geometry from these image-based models, further steps like NeRF distillation, using techniques such as Score Distillation Sampling (SDS) [55] 64 or direct optimization sampled images 89 21,, are often necessary. However, distillation techniques such as SDS can introduce substantial computational overhead due to test-time optimization.
(97). and MegaScenes (75). though these often came the cost
More recently, promising trend has emerged with models that jointly predict multiple target views while maintaining accurate camera control and ensuring view consistency, often through mechanisms
like cross-view attention. This category includes methods such as FD [OT. EscherNet (36],
CAT3D (21), and SV3D [79.. Several video model- approaches 85 23
99 94 102 also fall
into this paradigm, increasing NVS performance. Despite these advancements, achieving high-quality generation with these recent models often necessitates substantial computational resources and extensive training data. Furthermore, their reliance on full-image iterative sampling typically results in slow inference times, limiting practical applicability. Our proposed method, UMAMI, addresses this critical issue by enabling photorealistic novel view rendering while maintaining efficient inference times.
18

---

Table 7: Hyperparameters for training UMAMI. We use the same set hyperparameters for both RealEstate 10K and DL3DV experiments.
Component
Parameter
Value
Image size
256
Image Tokenizer
Patch size
Channels
9 (3 RGB + 6 for Plücker)
Layers
24
Hidden dim
768
Transformer
Head dim
64
QK Norm
True
Batch size / GPU
Num GPUS
Learning rate
0.0002
Optimizer (3,, 82)
(0.9, 0.95)
Training
Grad clip norm
3.0
Mixed precision
True
Weight decay
0.02
Train steps
100k
Warmup steps
1000
Input Target views
to 2/ to
Data Setup
Center Crop
True
L2 loss
1.0
LPIPS loss
0.0
Loss Weights
Perceptual loss
0.5
Diffusion loss
Confidence loss
C Implementation details
C.1
Hyperparamters
We report the hyperparameters used in Table 7
C.2 Algorithm
We describe the sampling process of UMAMI in Algorithm
Algorithm Hybrid Inference in UMAMI
Require: Trained model, context views {(exx ), target pose Teg. threshold 7, max unmasking
steps Tmax
Tokenize context views into c, initialize target tokens x with masked tokens
2: Encode (c, x) with Transformer to obtain latent z
3: Predict confidence map Sp and patch-level scores
4: Partition target tokens:
Deterministic tokens: XD {z; s; 2 T}
Stochastic tokens: xs < {2; 8; <
5: Predict x in one pass using deterministic head: &D = (((2))
6: Compute sampling steps: Ts [|>s|/|>]] Tmax]
7: for I to 1 do
8: Sample random unmasked set C xs following a cosine scheduler.
9: Update x, by DDPM sampling using yp head.
10: end for
11: Merge &D and Es into full target image fg.
19

---

D Additional quantitative results
D.I
Multiple images generation
Table 8: Multi-view generation results on RealEsttate I0K.
Dataset
# gen views
PSNR
LPIPS +
SSIM f
1
28.95
0.107
0.897
Re10K-2views-Exxraa
28.65
0.109
0.892
1 28.85
0.101
0.899
Re10K-2views-Interp
3
28.52
0.105
0.894
As shown in Table 7 our model is trained to predict up to three target views simultaneously. This joint prediction encourages consistency across generated images. In Table 8 we report results for generating one and three views. The generation quality is comparable across both settings. Notably, we use fixed number unmasking steps maaxx = 32) for all cases, which means generating multiple views in parallel can improve inference efficiency without sacrificing quality.
E
Additional qualitative results
Input
Target
Ours
DepthSplat
PARK PRIME STEAKHOKSH
IME STEAKHOUSE
MMES STRA MHOOSE
Figure 6: Additional qualitative comparisons on DL3DV dataset.
Additional qualitative evaluations are presented on the DL3DV dataset [4]] where our method is
compared against DepthSplat [91) under three-view input configuration (Figure 6 As depicted UMAMI demonstrates notably sharp rendering, particularly in unobserved regions. This is achieved by leveraging its generative capabilities to synthesize plausible details unobserved region of input images.
Furthermore, to investigate the impact of the diffusion threshold hyperparameter, т, on UMAMI'S performance, value was systematically varied, with findings illustrated in Figure 77 An initial setting of 5 0, corresponding fully deterministic operation of UMAMI, achieved rapid inference.
20

---

Input
Prreets/Peddcctions
t= 0
t == 0.5
0.95
Figure 7: Impact of the diffusion threshold hyperparameter rendering outcomes. The top row shows the single input view alongside four corresponding target views. The subsequent rows (2-4) illustrate the results 7 is incrementally increased. While lower promotes deterministie behavior and faster inference, higher values of lead to notably sharper image rendering quality.
However, this configuration resulted in image blurring, an artifact attributable unobserved regions in the input view. Progressively increasing 7 to 0.5 and subsequently to 0.95 yielded a significant enhancement rendering quality. This improvement, however, was accompanied by an increase in running time. Finally, to demonstrate the complete sampling dynamics of our method, the unmasking processes for r 0.95 and for full unmasking diffusion process (7 = 1) are presented in the supplementary video.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the abstract, we mainly claimed that our method can achieve state-of-the-al aceuracy and achieve faster speed compared to generative methods. There supported with experimental evidence our main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We have provided a dedicated subsection that discusses the limitations of our approach.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
This work does not include theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The "Experiments" section details the experimental setup, including GPU configurations, learning rates, optimizers, and the datasets used. Additionally, the code will be made publicly available upon acceptance.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The code currently proprietary. We intend to open-source upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/publiccguu..ee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The settings are specified in the "Experiment" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The experiment works with large datasets, and the resources needed are such that running it multiple times for statistical significance would present a significant cost challenge.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included the information in the "Experiments" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have pointed out this work could used to generate deepfake information.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As only train model using standardized NVS datasets, this does not apply to our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly stated the licenses for both datasets used and included references to their associated research papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification: manuscript.
No new datasets code are being submitted in conjunction with this
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
27

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper does involve LLMs any important, original, non-standard components
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to LLM policy for what should or should not be described.
28


Reviews:
[{"review_id": "r9mGmYu5DH", "reviewer": "Reviewer_bgjp", "summary": "This paper introduces UMAMI, a novel hybrid framework that integrates deterministic rendering and masked autoregressive diffusion for the task of novel view synthesis from sparse inputs. The method uses a bidirectional Transformer to encode image tokens and Plücker-ray embeddings into a latent representation, which is then processed by two heads: A deterministic regression head for well-constrained (visible) regions, and a masked autoregressive diffusion head for hallucinating occluded/unseen areas. By combining both approaches and introducing a confidence-aware mechanism to split regions between the two heads, UMAMI achieves a strong balance of quality and efficiency. It shows state-of-the-art performance on RealEstate10K and DL3DV across interpolation and extrapolation settings, while maintaining significantly lower inference times than diffusion-only baselines.", "questions": "Questions and Suggestions\n- How does confidence prediction generalize? Could the authors clarify whether the confidence predictor generalizes well across scenes, or if it's overfitting to typical visibility patterns in training data?\n- Could pretrained priors be integrated? Would combining UMAMI with pretrained image/video priors (e.g., Imagen, VideoCrafter, or Cosmos) boost generalization, especially for hallucinating semantically plausible content in unobserved regions?\n- Is it possible to use a buffer to \"remember\" synthesized images and re-use them as input known regions for faster speed when a certain region is revisited in the same session? How can the method ensure 3D consistency within the single scene being generated?\n- An interactive 3D visualizer would help in understanding the approach better as well as how the hybrid sampler performs.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and includes comprehensive ablations and discussions, such as runtime analyses and number of context views.\n- The combination of deterministic (feed-forward) and diffusion-based rendering into a unified model is elegant and well-motivated, with the aim of tackling the weaknesses of both paradigms. The use of a predicted confidence map to dynamically assign image regions to deterministic or stochastic pathways is a clever strategy to optimize performance and speed.\n- Strong experimental results on both interpolation and extrapolation tasks across two large-scale datasets. The method outperforms both deterministic and diffusion-based baselines in many settings. It provides flexible inference-time control over quality versus speed via the threshold parameter tau.\n\nWeaknesses\n- While the presented hybrid approach is empirically justified, it is not clear whether the model will fail in any of the intermediate steps and hence leading to undesired artifacts. For example, what if the confidence mask is not accurately predicted? What happens if the camera goes far away beyond context images? How can the model ensure that the deterministically regressed regions can consistently fuse with the generated part? It would be interesting to visualize these results, and present some in-depth theoretical analysis of the hybrid sampling mechanism.\n- Unlike some recent diffusion-based NVS methods, UMAMI does not leverage large-scale pretrained image/video priors, potentially limiting performance in extremely sparse-view, ambiguous, or out-of-domain cases. It would be nice to have some discussions about how large foundation models can be leveraged under these settings.\n- 3D Consistency might be an issue especially for the generative part of the model. There seems to be no mechanism to ensure that the rendered images look similarly with similar camera view points. While it makes sense to utilize an image model which is more flexible, it would be interesting to explore directions where synthesized views can be added to the context images for better 3D consistency.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time to review our paper and provide valuable feedback. We would like to address each of your concerns separately below:\n\n**1. While the presented hybrid approach is empirically justified, it is not clear whether the model will fail in any of the intermediate steps and hence leading to undesired artifacts. For example, what if the confidence mask is not accurately predicted? What happens if the camera goes far away beyond context images? How can the model ensure that the deterministically regressed regions can consistently fuse with the generated part? It would be interesting to visualize these results, and present some in-depth theoretical analysis of the hybrid sampling mechanism.**\n\n**Q: How does confidence prediction generalize? Could the authors clarify whether the confidence predictor generalizes well across scenes, or if it's overfitting to typical visibility patterns in training data?**\n\n*Note: Due to the recent notice from NeurIPS 2025, we apologize that we are not able to provide a visual illustrations for this concern. We will include visual result in our supplemental and project page in our final revision.*\n\na) Confidence Prediction and Generalization:\nWe have manually inspected the predicted confidence maps in many cases. We observed a consistent qualitative pattern: the confidence is predictably higher in areas seen by the context images and lower in unseen regions. This aligns with our core assumption that the model is more certain about what it has directly observed. Regarding the question “what if the confidence mask is not accurately predicted”, we want to point out that, with different values of tau, we are effectively changing the predicted masks, from an empty set to the whole image area. We did not notice obvious artifacts. Also, the test set of the Re10k dataset contains lots of different scenes, and the predicted confidence maps can be generalized very well on this test set, and it’s not overfitting to specific patterns in the training data.\n\nb) Consistency in Hybrid Sampling:\nDespite the fact that we are using two heads, i.e., a deterministic head and a generative diffusion head, we see no visible seams in the generated images in the target views, as shown in Figure 1, 4, and 6. This is because although being two heads, they are not separate and independent. The two heads are carefully linked by the hybrid masked autoregressive sampler, as illustrated in Figure 3. Firstly, a hyper-parameter tau is designed to balance their contributions. Secondly, the generative head is designed to be autoregressive, which generates new regions by unmasking tokens while explicitly conditioning on the tokens produced by the deterministic head. This mechanism ensures that the generated content naturally extends from the regressed regions, guaranteeing a smooth and consistent fusion. \n\nc) Handling Out-of-Context Cameras:\nWhen the target camera goes far away beyond the context views, the confidence map would show a low confidence score and the generated images are performed entirely by generative heads. Some failure cases in this category have been shown in Figure 5, Supplement A. In these scenarios, the model is essentially performing a more challenging generative task without much deterministic guidance. If the camera is extremely distant, our current model (271M parameters) may produce noticeable artifacts. We hypothesize that increasing the model size and training data would significantly improve robustness in these challenging, far-reaching camera positions. The confidence maps for these cases are very low, which suggests that the core mechanism of our proposed method works as expected. We’ll also add the visualization of the confidence maps to Figure 5, Supplement A.\n\n**2. Unlike some recent diffusion-based NVS methods, UMAMI does not leverage large-scale pretrained image/video priors, potentially limiting performance in extremely sparse-view, ambiguous, or out-of-domain cases. It would be nice to have some discussions about how large foundation models can be leveraged under these settings.**\n\n**Q: Could pretrained priors be integrated? Would combining UMAMI with pretrained image/video priors (e.g., Imagen, VideoCrafter, or Cosmos) boost generalization, especially for hallucinating semantically plausible content in unobserved regions?**\n\nWe thank the reviewer for these excellent suggestions. Though it is tempting to adopt large-scale pretrained image/video generative models, e.g., Imagen, VideoCrafter, or Cosmos,  to our framework, our method is fundamentally different to conventional pretrained diffusion models. While common diffusion models generate images by iteratively removing noise of the whole image, our generative scheme, which is based on MAR [1], renders images by autoregressively unmasking tokens in random raster order. Our diffusion process is performed at token-level, instead of image-level like in common pretrained diffusion models. Thus, MAR [1] and its extension, Fluid [2] can be used as the pretrained prior for our NVS model. We acknowledge that it is a promising avenue to explore and leave this direction for future consideration. \n\n**3. 3D Consistency might be an issue especially for the generative part of the model. There seems to be no mechanism to ensure that the rendered images look similarly with similar camera view points. While it makes sense to utilize an image model which is more flexible, it would be interesting to explore directions where synthesized views can be added to the context images for better 3D consistency.**\n\n**Q: Is it possible to use a buffer to \"remember\" synthesized images and re-use them as input known regions for faster speed when a certain region is revisited in the same session? How can the method ensure 3D consistency within the single scene being generated?**\n\n*Note: Due to the recent notice from NeurIPS 2025, we apologize that we are not able to provide a video illustration for this concern. We will include video result in our supplemental and project page in our final revision.*\n\nIt is worth noting that our method originally aims for set NVS [3,4], which considers target views as an arbitrary order. To enforce 3D consistency, one can employ 3D scene optimization such as NeRF and 3DGS as suggested in [3,4]. In our experiments, we observe that generating images on the fly might cause some flickering. Furthermore, due to resource constraint, during training we only sample 3 target views, compared to 21 frames in SEVA, which can limit the consistency of extrapolation results. That said, the idea of exploring directions which auto-regressively generate new view based on generated views are promising and worth exploring in our future work. \n\n[1] Autoregressive Image Generation without Vector Quantization.\n\n[2] Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens.\n\n[3] ReconFusion: 3D Reconstruction with Diffusion Priors.\n\n[4] CAT3D: Create Anything in 3D with Multi-View Diffusion Models."}, {"review_id": "bAcDM8N1Ez", "reviewer": "Reviewer_G1uS", "summary": "The paper introduces UMAMI, a hybrid framework that unifies deterministic rendering and masked autoregressive diffusion models for novel view synthesis (NVS) from sparse input views. UMAMI leverages a bidirectional transformer to encode both context images (paired with Plücker ray embeddings) and a masked target view. It then uses two separate lightweight MLP heads: one for feed-forward pixel regression in observed regions, and another for diffusion-based generation in uncertain/unseen areas, guided by a learned confidence map. The paper demonstrates competitive performance across interpolation, extrapolation, and varying numbers of input views on RealEstate10K and DL3DV benchmarks, along with significant inference speedups over purely generative models.", "questions": "Please refer to the weaknesses part.", "limitations": "Please refer to the weaknesses part.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The proposed method combines deterministic regression and masked autoregressive diffusion for NVS, with a clear architectural motivation and evidence of improved trade-offs between speed and quality.\n- There is a comprehensive quantitative comparison with strong baselines (Table 1, Table 2) on established large-scale benchmarks (Re10k, DL3DV), covering both deterministic and generative NVS methods. UMAMI often matches or exceeds the best baselines, while reducing inference time compared to diffusion-only models.\n- The paper is overall clear and easy to follow.\n\n## Weaknesses\n- The LVSM model was not trained on the DL3DV dataset, only on Re10k, and the work in this paper builds upon further training of the pre-trained LVSM model. However, the performance metrics on DL3DV are notably poor. Does this suggest that the model's performance advantage primarily derives from LVSM? It would be informative to evaluate the model’s performance without the pre-trained LVSM.\n- The reported improvements in metrics compared to LVSM are not sufficiently significant, and some metrics even show a decline.\n- The performance comparison of DL3DV against Flare [1] indicates notably poor results on DL3DV. Given that purely implicit NVS tasks should not be heavily influenced by scale, why does DL3DV exhibit such suboptimal performance on these metrics?\n\n[1] FLARE: Feed-forward Geometry, Appearance and Camera Estimation from  Uncalibrated Sparse Views. arXiv 2502.12138 (CVPR 2025)", "was_revised": true, "final_justification": "The authors’ rebuttal has addressed most of the concerns raised in my previous review. Due to OpenReview’s limitations preventing the upload of additional figures for qualitative evaluation, I recommend that the authors include these qualitative results in the camera-ready version of the paper.", "rebuttal": "We thank the reviewer for your time to review our paper and provide valuable feedback. We would like to address each of your concerns separately below:\n\n**1. The LVSM model was not trained on the DL3DV dataset, only on Re10k, and the work in this paper builds upon further training of the pre-trained LVSM model. However, the performance metrics on DL3DV are notably poor. Does this suggest that the model's performance advantage primarily derives from LVSM? It would be informative to evaluate the model’s performance without the pre-trained LVSM.**\n\nOur validation split for the DL3DV dataset follows SEVA. Importantly, these splits include scenes where the target views have minimal overlap with the input views. As shown in Table 2, our model achieves performance comparable to SEVA, despite being only one-fifth of its size, and significantly surpasses ViewCrafter and DepthSplat in most cases. To accelerate training and improve convergence, we initialize our model using LVSM pretrained weights, a common practice in the deep learning community. That said, to further demonstrate the strength of our method, we also train the model from scratch on DL3DV with randomly initialized weights, without relying on LVSM pretrained on Re10K. This variant shares the same settings as the pretrained version, except for a larger batch size (512 vs. 32) to stabilize training. The results, presented in the table below, reveal that even without pretraining, our model performs comparably—showing slightly lower PSNR and SSIM but improved LPIPS. Remarkably, even with random initialization, our method consistently outperforms ViewCrafter and DepthSplat, underscoring that our performance stems from the strength of our hybrid deterministic-generative design, rather than dependence on LVSM initialization and Re10K pretraining. We will include this result in our final revision. \n\n|              | 1view PSNR | 1view LPIPS | 1view SSIM | 3view PSNR | 3view LPIPS | 3view SSIM | 6view PSNR | 6view LPIPS | 6view SSIM |\n|:-------------|:----------:|:-----------:|:----------:|:----------:|:-----------:|:----------:|:----------:|:-----------:|:----------:|\n| **Pretrained** | 12.81      | 0.574       | 0.269      | 16.37      | 0.386       | 0.444      | 17.33      | 0.326       | 0.476      |\n| **Random** | 11.80      | 0.543       | 0.256      | 14.46      | 0.374       | 0.370      | 15.43      | 0.318       | 0.404      |\n\n**2. The reported improvements in metrics compared to LVSM are not sufficiently significant, and some metrics even show a decline.**\n\nOur metrics indeed show a slight decline on interpolation tasks compared to LVSM, and this is an expected and deliberate outcome of our model's design. Our model is explicitly designed to excel in scenarios with significant occlusion content, such as challenging extrapolation tasks where unseen regions must be plausibly generated. To achieve this, we train the model with a joint objective that includes diffusion loss. This design choice creates a deliberate trade-off: on challenging extrapolation tasks, i.e. high occlusion content, our method significantly outperforms deterministic baselines by generating high-quality, coherent content for unseen regions, a benefit that is demonstrated qualitatively (Fig. 4) than with pixel-wise metrics. Meanwhile, on simple interpolation tasks (lacking occlusion content), the inclusion of the diffusion loss can lead to a marginal decrease in PSNR compared to a purely deterministic model like LVSM, which is optimized solely for reconstruction accuracy. That said, the inferior result on the interpolation set is not an unexpected failure, but rather a direct consequence of optimizing for a more difficult and general problem. Furthurmore, as also mentioned by other reviewers, using metric like PSNR to evaluate generative NVS models might not fully capture our method's benefits. We believe sacrificing a small amount of reconstruction fidelity on the simple regression task is a very favorable trade-off for gaining robust generative capabilities. We will clarify this point explicitly in the experimental section of our revised paper.\n\n**3. The performance comparison of DL3DV against Flare [1] indicates notably poor results on DL3DV. Given that purely implicit NVS tasks should not be heavily influenced by scale, why does DL3DV exhibit such suboptimal performance on these metrics?**\n\nWe would like to highlight that the view selection strategy used in FLARE's testing setup differs from ours on the DL3DV dataset. Specifically, our test set chooses the target views that have minimal overlap with the context views, making the task more challenging. In contrast, FLARE only evaluates view interpolation where the target regions are very close to the context views, thus making the FLARE validation set to be much easier. To ensure a fair comparison, we evaluated our model using FLARE's view split. The results show that our model matches (SSIM) or outperforms (PSNR, LPIPS) FLARE on DL3DV across all three evaluation metrics: PSNR, LPIPS, and SSIM.\n\n| 8-view         | PSNR  | LPIPS | SSIM  |\n|:----------------|:------|:------|:------|\n| Ours (tau=0.95) | 24.17 | 0.167 | 0.710 |\n| Flare           | 23.33 | 0.237 | 0.746 |"}, {"review_id": "z9t91SedmL", "reviewer": "Reviewer_Ydyv", "summary": "The paper introduces UMAMI, a hybrid novel view synthesis (NVS) framework that combines deterministic rendering and generative masked autoregressive modeling within a unified model. The approach leverages a bidirectional transformer backbone to process multi-view image tokens and Plücker ray embeddings. Two prediction heads are designed: a deterministic head for directly regressing RGB colors and confidence scores in well-constrained regions, and a masked autoregressive diffusion head for progressively generating content in uncertain or unseen regions. A hybrid sampling strategy is employed, starting with deterministic predictions and using the generative head to refine low-confidence areas. The method demonstrates promising results on RealEstate10K and DL3DV datasets, achieving competitive quality with faster rendering than fully generative baselines, and providing plausible synthesis for both observed and unseen regions.", "questions": "Overall, the idea and general framework of the paper look interesting and promising, and I lean positive. However, I have main concerns regarding video consistency, as outlined in the weaknesses section. While the method is able to generate some promising individual images as shown in the paper, I am not yet convinced it can produce a continuous, consistent novel view trajectory, which is essential for many NVS applications.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe high-level idea of unifying deterministic and generative models for NVS is novel and reasonable. It addresses the trade-offs between speed and generative capability. To this end, the paper presents several thoughtful design choices, such as confidence estimation, confidence-based token sampling, and efficient hybrid inference, combining deterministic and stochastic predictions effectively. The results look promising with plausible novel view renderings in unobserved regions.\n\nWeaknesses: \n1. The quantitative results in Table 1 show limited improvement over baselines such as LVSM. Even for the extrapolation case, the PSNR gains are modest, and on interpolation tasks, UMAMI’s PSNR is actually lower than LVSM, which is somewhat surprising given that UMAMI uses LVSM pretrained weights (as stated in L248). This raises questions about whether introducing generative capacity may inadvertently sacrifice the accuracy of deterministic estimation.\n\n    That said, I also feel that Table 1 may understate the generative benefits of the proposed method. For example, the qualitative results in Figure 4 clearly illustrate improvements in rendering unseen regions, but these benefits are not captured effectively by traditional metrics like PSNR, SSIM, or LPIPS. I recognize that these metrics are the standard in the community, but they are arguably not ideal for evaluating the perceptual or generative quality of novel view synthesis models, especially for challenging extrapolation scenarios.\n\n2. My biggest concern with this paper is the lack of video rendering results, which I believe is critical for novel view synthesis (NVS) papers, especially those incorporating generative components. Without video demonstrations, it is impossible to properly evaluate how consistent the method is when generating images across different viewpoints.\n\n    If I understand correctly, the entire method operates on a per-novel-view basis, where modules such as deterministic estimation, confidence filtering and the masked autoregressive (MAR) regression process are applied independently for each target view. This per-view independence may introduce view inconsistencies, particularly in generating the uncertain, unobserved regions across continuous camera trajectories. Including video results—and ideally, some comparisons—would be essential to assess this aspect and provide stronger evidence of the model's practical applicability.", "was_revised": true, "final_justification": "Overall, I do not have strong complaints about the paper, and I think the idea is interesting, so I will keep my positive rating. I am not ready to give a higher score due to the lack of video results, which makes it difficult to fully assess the method’s behavior about across-view consistency in practice. It is unfortunate that NeurIPS has the no-video constraint in the rebuttal period; I hope the community can be more open to this in the future.", "rebuttal": "We thank the reviewer for your time to review our paper and provide valuable feedback. We would like to address each of your concerns separately below:\n\n**1. The quantitative results in Table 1 show limited improvement over baselines such as LVSM. Even for the extrapolation case, the PSNR gains are modest, and on interpolation tasks, UMAMI’s PSNR is actually lower than LVSM, which is somewhat surprising given that UMAMI uses LVSM pretrained weights (as stated in L248). This raises questions about whether introducing generative capacity may inadvertently sacrifice the accuracy of deterministic estimation.**\n\nWe agree with the reviewer that the quantitative metrics in Table 1, while standard for the field, do not fully capture the primary contribution of our work. Metrics like PSNR excel at measuring the pixel-wise fidelity of reconstructed regions but are fundamentally limited in their ability to evaluate the perceptual quality and plausibility of generated content in unseen areas. The modest drop in PSNR on interpolation tasks is an expected consequence of our hybrid training objective. By incorporating a powerful diffusion loss to enable generative capabilities, the model makes a slight trade-off in reconstruction accuracy on well-constrained views to gain a significant leap in performance on challenging extrapolation scenarios. We argue this is a highly favorable trade-off. The qualitative results in Figure 4 provide a much clearer picture of our method's strength. UMAMI's core novelty is its ability to plausibly and coherently complete large, unobserved regions—a capability that purely deterministic models like LVSM lack. We believe this generative power is a significant step forward for building efficient generative NVS. We will revise the paper to include a more explicit discussion on the limitations of these metrics for generative NVS and to better frame our results in this context.\n\n**2. My biggest concern with this paper is the lack of video rendering results, which I believe is critical for novel view synthesis (NVS) papers, especially those incorporating generative components. Without video demonstrations, it is impossible to properly evaluate how consistent the method is when generating images across different viewpoints.**\n\n*Note: Due to the recent notice from NeurIPS 2025, we apologize that we are not able to provide a video illustration for this concern. We will include video result in our supplemental and project page in our final revision.*\n\nIt is worth noting that our method originally aims for Set-NVS [1,2], which considers target views as an arbitrary order. To enforce 3D consistency, one can subsequently employ 3D scene optimization such as NeRF and 3DGS as suggested in [1,2]. That said, we have generated video rendering results on DL3DV and Re10K on various settings and observe that our rendering results are reasonably 3D consistent, though there still exist some minor flickers, which are inevitable in diffusion NVS. We will include these results in our revision project.\n\n[1] ReconFusion: 3D Reconstruction with Diffusion Priors.\n\n[2] CAT3D: Create Anything in 3D with Multi-View Diffusion Models."}, {"review_id": "jYdMzcsiY2", "reviewer": "Reviewer_fjdZ", "summary": "This paper proposes a novel generalized method for novel view synthesis. The proposed method combines the advantages of the two worlds: the fast speed of the deterministic NVS method and the ability to synthesize high-quality occluded regions of the generative-based NVS method. To achieve this, the paper proposes a novel network architecture, where the unoccluded regions are predicted by a deterministic network while the occluded regions are predicted by a generative-based method. The detection of occluded and unoccluded regions is achieved by a confidence learning framework.", "questions": "* Can the author explain the difference between this paper and LVSM? I find the network architecture of this paper and LVSM to be quite similar. If we discard the diffusion head while only using the deterministic head, the whole method seems identical to the decoder-only version of LVSM. If so, I suggest that the author discuss LVSM more in the related work sections.\n* Following the previous question. Can the author explain why the proposed method obtains inferior results on the Re10K-2view-interp setup? Is there a trade-off between the deterministic NVS method and the generative-based NVS method, maybe something like: the deterministic method can do well (i.e., obtain better metrics) on scenarios lacking occlusion content, while the generative-based method on scenarios with significant occlusion content? Can the author provide some insights on this point? \n  * Or just because the PSNR metric is questionable? If so, I suggest that the author clarify this point in the experiment section.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* Overall, the paper is well-written and easy to follow. The motivation is clear, and the proposed technique is neat.\n* The combination of MAR for NVS is novel and effective.\n* The paper works on an important problem.\n\n### Weaknesses\n* The most significant improvements in speed over the prior generative-based method stem from the usage of MAR. MAR is faster than the Diffusion Model. From Table 3, if we use the MAR to generate all of the content, only 7.63s is required, which is significantly faster than diffusion-based methods like SEVA (1 min is required). Consider that the margin is relatively small (4.77s vs. 7.63s), in what scenarios do we prefer the tau=0.95 version to improve the speed while at the cost of quality? To me, this point is important because the provided trade-off is one of the core contributions of this paper. I guess maybe a small tau leads to better 3D consistency? If the author can address this problem, I will consider raising my score.", "was_revised": true, "final_justification": "This paper provides a strong rebuttal to address my concerns. I do not raise the score due to the lack of video results. I suggest the author add video results for better evaluation of the view consistency in the final version.", "rebuttal": "We thank the reviewer for your time to review our paper and provide valuable feedback. We would like to address each of your concerns separately below:\n\n**1. The most significant improvements in speed over the prior generative-based method stem from the usage of MAR. MAR is faster than the Diffusion Model. From Table 3, if we use the MAR to generate all of the content, only 7.63s is required, which is significantly faster than diffusion-based methods like SEVA (1 min is required). Considering that the margin is relatively small (4.77s vs. 7.63s), in what scenarios do we prefer the tau=0.95 version to improve the speed while at the cost of quality? To me, this point is important because the provided trade-off is one of the core contributions of this paper. I guess maybe a small tau leads to better 3D consistency? If the author can address this problem, I will consider raising my score.**\n\n*Note: Due to the recent notice from NeurIPS 2025, we apologize that we are not able to provide a video illustration for this concern. We will include video result in our supplemental and project page in our final revision.*\n\nThank you for raising this important question! Indeed, with smaller tau=0.95 instead of tau=1.0, we can achieve better 3D consistency. As illustrated in Figure 3, the autoregressive sampler depends on the tokens from the deterministic head, which are absent when tau=1.0. These tokens are crucial due to their high 3D consistency. With the availability and the help of these tokens, the diffusion head can generate content that is more 3D-consistent.\nTo prove our argument, we have generated two sets of videos, with tau=0.95 and tau=1.0, respectively. It has been observed that videos generated with tau=0.95 consistently exhibit better 3D consistency compared to those generated with tau=1.0. We will include these comparative videos in the final version of our paper.\n\n**2. Can the author explain the difference between this paper and LVSM? I find the network architecture of this paper and LVSM to be quite similar. If we discard the diffusion head while only using the deterministic head, the whole method seems identical to the decoder-only version of LVSM. If so, I suggest that the author discuss LVSM more in the related work sections.**\n\nOur method is different from LVSM in various aspects:\n\nGenerative capability: It is worth noting that LVSM is a deterministic rendering method, which means the rendered target views are observed in the context views. In the unobserved regions, LVSM, and other deterministic methods, are not able to generate realistic images, and often produce blurry or black pixels as shown in Figure 4 of our paper. Unlike LVSM, our method possesses generative capability that can efficiently synthesize target scenes even in both visible and occluded regions from context views. \n\nA unified hybrid architecture: Unlike LVSM's purely deterministic approach, UMAMI introduces a novel architecture that integrates both a regression head and a diffusion head within a single, end-to-end trained network. A critical component of this is our deterministic head, which not only regresses pixel values but also predicts a confidence map. This map is essential for dynamically guiding the inference process. \n\nA novel hybrid inference strategy: In LVSM, an image is rendered by regressing pixels from trained models. UMAMI, on the other hand, performs hybrid inference by first identifying and reconstructing a set of deterministic tokens, and then generating the remaining uncertain tokens via a diffusion-based process. In the first pass, UMAMI performs a single forward pass through the deterministic head, guided by the confidence map predicted from the transformer backbone. Tokens with confidence scores greater than a predefined threshold tau are reconstructed deterministically. In the second pass, the remaining masked tokens are iteratively sampled using the diffusion head. This multi-pass, confidence-guided sampling is a core contribution of our work. \n\nWhile we build upon a similar backbone, our contribution is the novel integration of generative and deterministic pathways, the confidence-guided mechanism to arbitrate between them, and the efficient hybrid sampling strategy that this enables. We will revise the related work section to make this distinction and our specific contributions clearer, and we thank the reviewer for this valuable suggestion.\n\n**3. Following the previous question. Can the author explain why the proposed method obtains inferior results on the Re10K-2view-interp setup? Is there a trade-off between the deterministic NVS method and the generative-based NVS method, maybe something like: the deterministic method can do well (i.e., obtain better metrics) on scenarios lacking occlusion content, while the generative-based method on scenarios with significant occlusion content? Can the author provide some insights on this point? \nOr just because the PSNR metric is questionable? If so, I suggest that the author clarify this point in the experiment section.**\n\nWe agree with the reviewer's hypothesis, and it highlights the core trade-off of our proposed method. Our model is explicitly designed to excel in scenarios with significant occlusion content, such as challenging extrapolation tasks where unseen regions must be plausibly generated. To achieve this, we train the model with a joint objective that includes diffusion loss. This design choice creates a deliberate trade-off: on challenging extrapolation tasks, i.e. high occlusion content, our method significantly outperforms deterministic baselines by generating high-quality, coherent content for unseen regions, a benefit that is demonstrated qualitatively (Fig. 4) than with pixel-wise metrics. Meanwhile, on simple interpolation tasks (lacking occlusion content), the inclusion of the diffusion loss can lead to a marginal decrease in PSNR compared to a purely deterministic model like LVSM, which is optimized solely for reconstruction accuracy. That said, the inferior result on the interpolation set is not an unexpected failure, but rather a direct consequence of optimizing for a more difficult and general problem. We agree with the reviewer that this underscores the limitations of using PSNR to evaluate generative NVS models. We believe sacrificing a small amount of reconstruction fidelity on the simple regression task is a very favorable trade-off for gaining robust generative capabilities. We will clarify this point explicitly in the experimental section of our revised paper."}]

Author rebuttal:

Meta-review: The final recommendations of the expert reviewers are 4x Borderline accept. These recommendations are made after addressing the authors' feedback and through a thorough discussion. This paper introduces UMAMI, a novel framework that integrates a feed-forward rendering with masked autoregressive diffusion for the task of novel view synthesis from sparse inputs. Specifically, the authors utilize a feed-forward method for well-constrained regions and a masked autoregressive diffusion for unseen or occluded regions. Reviewers agree that this presents a compelling and principled framework for generative novel view synthesis.

While the proposed method demonstrates promising performance compared to state-of-the-art approaches, the primary concern raised by reviewers is the limited scope of the experimental evaluation. Notably, the paper lacks results on an established benchmark such as the DL3DV dataset, which is widely used for evaluating novel view synthesis in complex real-world scenes.

However, this concern was partially addressed by the authors during the rebuttal, where they provided the requested experimental results. More importantly, the existing results are technically sound, validated by rigorous ablations, and sufficient to demonstrate the merit of the proposed approach. The technical novelty, architectural insight, and clear contribution of the work are substantial and outweigh the limitation in benchmark coverage. Given the overall strength of the method and the thoughtful response to reviewer feedback, the consensus supports acceptance.
Decision: Accept (poster)